CT-AI ISTQBCertified Tester AI Testing Exam Questions and Answers

According to the syllabus, pre-trained models often inherit biases and limitations from the data and processes used in their original training, which may not align with the new use case. Specifically, the syllabus states:

"When using a pre-trained model, the training data and process cannot be fully controlled or known by the user of the model. As a result, the model can inherit biases or inaccuracies that were part of its original development and training process."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 1.8.3)

A. Black box attacks based on adversarial examples create an exact duplicate model of the original.

Black box attacks do not create an exact duplicate model. Instead, they exploit the model by querying it and using the outputs to craft adversarial examples without knowledge of the internal workings.

B. These attack examples cause a model to predict the correct class with slightly less accuracy even though they look like the original image.

Adversarial examples typically cause the model to predict the incorrect class rather than just reducing accuracy. These examples are designed to be visually indistinguishable from the original image but lead to incorrect classifications.

C. These attacks can't be prevented by retraining the model with these examples augmented to the training data.

This statement is incorrect because retraining the model with adversarial examples included in the training data can help the model learn to resist such attacks, a technique known as adversarial training.

D. These examples are model specific and are not likely to cause another model trained on the same task to fail.

Adversarial examples are often model-specific, meaning that they exploit the specific weaknesses of a particular model. While some adversarial examples might transfer between models, many are tailored to the specific model they were generated for and may not affect other models trained on the same task.

Therefore, the correct answer isDbecause adversarial examples are typically model-specific and may not cause another model trained on the same task to fail​​.

The syllabus explicitly mentions challenges of handling personal data and ensuring privacy when testing AI-based systems:

"The management of personal data and sensitive data is often a concern during testing, as testing typically requires realistic data and it is difficult to fully anonymize data."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 7.3, page 52 of 99)

When written requirements are given in text documents, the best way to generate test cases is by using Natural Language Processing (NLP). Here's why:

Natural Language Processing (NLP): NLP can analyze and understand human language. It can be used to process textual requirements to extract relevant information and generate test cases. This method is efficient in handling large volumes of textual data and identifying key elements necessary for testing.

Why Not Other Options:

Analyzing source code for generating test cases: This is more suitable for white-box testing where the code is available, but it doesn't apply to text-based requirements.

Machine learning on logs of execution: This approach is used for dynamic analysis based on system behavior during execution rather than static textual requirements.

GUI analysis by computer vision: This is used for testing graphical user interfaces and is not applicable to text-based requirements.

The syllabus states that non-determinism is one of the key challenges for ensuring safety in AI-based systems:

"The characteristics of AI-based systems that make it more difficult to ensure they are safe... include: complexity, non-determinism, probabilistic nature, self-learning, lack of transparency, interpretability and explainability, and lack of robustness."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 2.8, page 25 of 99)

Recognizing digits from a scan of handwritten numbers using machine learning is an example of classification. Here’s a breakdown:

Classification: This type of machine learning involves categorizing input data into predefined classes. In this scenario, the input data (handwritten digits) are classified into one of the 10 digit classes (0-9).

Why Not Other Options:

Reinforcement Learning: This involves learning by interacting with an environment to achieve a goal, which does not fit the problem of recognizing digits.

Regression: This is used for predicting continuous values, not discrete categories like digit recognition.

Clustering: This involves grouping similar data points together without predefined classes, which is not the case here.

The syllabus definesexplainabilityas the ability to understand how the AI-based system comes up with a particular result:

"Explainability is considered to be the ease with which users can determine how the AI-based system comes up with a particular result."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 2.7)

In a neural network, the activation value of a neuron is determined by a combination of inputs from the previous layer, the weights of the connections, and the bias at the neuron level. Here’s a detailed breakdown:

Inputs for Activation Value:

Activation Values of Neurons in the Previous Layer:These are the outputs from neurons in the preceding layer that serve as inputs to the current neuron.

Weights Assigned to the Connections:Each connection between neurons has an associated weight, which determines the strength and direction of the input signal.

Individual Bias at the Neuron Level:Each neuron has a bias value that adjusts the input sum, allowing the activation function to be shifted.

Calculation:

The activation value is computed by summing the weighted inputs from the previous layer and adding the bias.

Formula: z=∑(wi⋅ai)+bz = \sum (w_i \cdot a_i) + bz=∑(wi​⋅ai​)+b, where wiw_iwi​ are the weights, aia_iai​ are the activation values from the previous layer, and bbb is the bias.

The activation function (e.g., sigmoid, ReLU) is then applied to this sum to get the final activation value.

Why Option A is Correct:

Option A correctly identifies all components involved in computing the activation value: the individual bias, the activation values of the previous layer, and the weights of the connections.

Eliminating Other Options:

B. Activation values of neurons in the previous layer, and weights assigned to the connections between the neurons: This option misses the bias, which is crucial.

C. Individual bias at the neuron level, and weights assigned to the connections between the neurons: This option misses the activation values from the previous layer.

D. Individual bias at the neuron level, and activation values of neurons in the previous layer: This option misses the weights, which are essential.

Pairwise testing is used to handle the large number of combinations of parameters that can arise in complex systems like self-driving cars. The question asks which of the given options isleast likelyto be a reason for the explosion in the number of parameters.

Different Road Types (A): Self-driving cars must operate on various road types, such as highways, city streets, rural roads, etc. Each road type can have different characteristics, requiring the car's system to adapt and handle different scenarios. Thus, this is a significant factor contributing to the growth of parameters.

Different Weather Conditions (B): Weather conditions such as rain, snow, fog, and bright sunlight significantly affect the performance of self-driving cars. The car's sensors and algorithms must adapt to these varying conditions, which adds to the number of parameters that need to be considered.

ML Model Metrics to Evaluate Functional Performance (C): While evaluating machine learning (ML) model performance is crucial, it does not directly contribute to the explosion of parameter combinations in the same way that road types, weather conditions, and car features do. Metrics are used to measure and assess performance but are not themselves variable conditions that the system must handle.

Different Features like ADAS, Lane Change Assistance, etc. (D): Advanced Driver Assistance Systems (ADAS) and other features add complexity to self-driving cars. Each feature can have multiple settings and operational modes, contributing to the overall number of parameters.

Hence, theleast likelyreason for the incredible growth in the number of parameters isC. ML model metrics to evaluate the functional performance.

The question asks which characteristic is least likely to cause safety-related issues for an AI system. Let's evaluate each option:

Non-determinism (A): Non-deterministic systems can produce different outcomes even with the same inputs, which can lead to unpredictable behavior and potential safety issues.

Robustness (B): Robustness refers to the ability of the system to handle errors, anomalies, and unexpected inputs gracefully. A robust system is less likely to cause safety issues because it can maintain functionality under varied conditions.

High complexity (C): High complexity in AI systems can lead to difficulties in understanding, predicting, and managing the system's behavior, which can cause safety-related issues.

Self-learning (D): Self-learning systems adapt based on new data, which can lead to unexpected changes in behavior. If not properly monitored and controlled, this can result in safety issues.

The syllabus clearly explains the structure of a deep neural network (DNN):

"A deep neural network comprises three types of layers. The input layer receives inputs... Between the input and output layers are hidden layers made up of artificial neurons, which are also known as nodes."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 6.1, page 45 of 99)

Setting model and algorithm hyperparameters is an essential step in the machine learning workflow, primarily occurring during the tuning phase.

Evaluating the model (A): This stage involves assessing the model's performance using metrics and does not typically include the setting of hyperparameters.

Deploying the model (B): Deployment is the stage where the model is put into production and used in real-world applications. Hyperparameters should already be set before this stage.

Tuning the model (C): This is the correct stage where hyperparameters are set. Tuning involves adjusting the hyperparameters to optimize the model's performance.

Data testing (D): Data testing involves ensuring the quality and integrity of the data used for training and testing the model. It does not include setting hyperparameters.

Hence, the most appropriate stage of the ML workflow to set model and algorithm hyperparameters isC. Tuning the model.

The syllabus discusses using AI-based tools to reduce GUI test brittleness:

"AI can be used to reduce the brittleness of this approach, by employing AI-based tools to identify the correct objects using various criteria (e.g., XPath, label, id, class, X/Y coordinates), and to choose the historically most stable identification criteria."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 11.6.1)

The syllabus explains that supervised learning requires correctly labeled data so the algorithm can learn the relationship between input features and output labels:

"In supervised learning, the algorithm creates the ML model from labeled data during the training phase. The labeled data is used to infer the relationship between the input data and output labels."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 3.1.1)

 AI-Based Autonomous Functions:An AI-based autonomous system is one that can respond to its environment without human intervention. The other options either involve human decisions or do not use AI at all.

 Reference:ISTQB_CT-AI_Syllabus_v1.0, Sections on Autonomy and Testing Autonomous AI-Based Systems​​​​.

The question asks which test is least likely to be performed during the ML model testing phase. Let's consider each option:

Testing the accuracy of the classification model (A): Accuracy testing is a fundamental part of the ML model testing phase. It ensures that the model correctly classifies the data as intended and meets the required performance metrics.

Testing the API of the service powered by the ML model (B): Testing the API is crucial, especially if the ML model is deployed as part of a service. This ensures that the service integrates well with other systems and that the API performs as expected.

Testing the speed of the training of the model (C): This is least likely to be part of the ML model testing phase. The speed of training is more relevant during the development phase when optimizing and tuning the model. During testing, the focus is more on the model's performance and behavior rather than how quickly it was trained.

Testing the speed of the prediction by the model (D): Testing the speed of prediction is important to ensure that the model meets performance requirements in a production environment, especially for real-time applications.

The syllabus describes metamorphic testing as:

“Testing involves defining metamorphic relations and then applying those relations to check that the transformations result in expected outcomes, even when the expected output of the system is unknown or not well-defined.”

In this scenario, applying the metamorphic relation (speed differences) and checking the transformed outcome (arrival times) fits the definition of metamorphic testing.

(Reference: ISTQB CT-AI Syllabus v1.0, Section 9.5, page 69 of 99)

Decision trees are a foundational algorithm used in supervised machine learning. The syllabus describes:

"A decision tree is a tree-like ML model whose nodes represent decisions and whose branches represent possible outcomes."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 3.4)

In the syllabus, theevolutioncharacteristic for AI-based systems means the ability of the system to evolve and adapt its behavior in response to changes in the environment or in its own performance:

"Evolution is the system’s ability to change itself to adapt to new situations, different hardware, or a changing operational environment."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 2.3)

Detecting biases in the ML pipeline involves various tests to ensure fairness and accuracy throughout the ML process.

Testing the distribution shift in the training data for inappropriate bias (A): This involves checking if there is any shift in the data distribution that could lead to bias in the model. It is an important test but not the most direct method for detecting biases.

Test the model during model evaluation for data bias (B): This is a critical stage where the model is evaluated to detect any biases in the data it was trained on. It directly addresses potential data biases in the model.

Testing the data pipeline for any sources for algorithmic bias (C): This test is crucial as it helps identify biases that may originate from the data processing and transformation stages within the pipeline. Detecting sources of algorithmic bias ensures that the model does not inherit biases from these processes.

Check the input test data for potential sample bias (D): While this is an important step, it focuses more on the input data and less on the overall data pipeline.

Hence, the most likely useful test to help detect different kinds of biases in the ML pipeline isB. Test the model during model evaluation for data bias.

A. This is an example of expert system bias.

Expert system bias refers to bias introduced by the rules or logic defined by experts in the system, not by the data distribution.

B. This is an example of sample bias.

Sample bias occurs when the training data is not representative of the overall population that the model will encounter in practice. In this case, the over-representation of ethnicity A (70%) compared to B, C, and D (30%) creates a sample bias, as the model may become biased towards better performance on ethnicity A.

C. This is an example of hyperparameter bias.

Hyperparameter bias relates to the settings and configurations used during the training process, not the data distribution itself.

D. This is an example of algorithmic bias.

Algorithmic bias refers to biases introduced by the algorithmic processes and decision-making rules, not directly by the distribution of training data.

Based on the provided information, optionB(sample bias) best describes the situation because the training data is skewed towards ethnicity A, potentially leading to biased model performance​​.