iSQI CT-AI_(v1.0)_World ISTQB Certified Tester AI Testing (v 1.0) Exam Practice Test

For AI-enabled medical devices, the most required aspects for certification are safety, transparency, and side effects. Here’s why:

Safety (Aspect III): Critical for ensuring that the AI system does not cause harm to patients.

Transparency (Aspect IV): Important for understanding and verifying the decisions made by the AI system.

Side Effects (Aspect V): Necessary to identify and mitigate any unintended consequences of the AI system.

Why Not Other Options:

Autonomy and Maintainability (Aspects I and II): While important, they are secondary to the immediate concerns of safety, transparency, and managing side effects in life-critical processes.

References:This explanation is aligned with the critical quality characteristics for AI-based systems as mentioned in the ISTQB CT-AI syllabus, focusing on the certification of medical devices​​.

When written requirements are given in text documents, the best way to generate test cases is by using Natural Language Processing (NLP). Here's why:

Natural Language Processing (NLP): NLP can analyze and understand human language. It can be used to process textual requirements to extract relevant information and generate test cases. This method is efficient in handling large volumes of textual data and identifying key elements necessary for testing.

Why Not Other Options:

Analyzing source code for generating test cases: This is more suitable for white-box testing where the code is available, but it doesn't apply to text-based requirements.

Machine learning on logs of execution: This approach is used for dynamic analysis based on system behavior during execution rather than static textual requirements.

GUI analysis by computer vision: This is used for testing graphical user interfaces and is not applicable to text-based requirements.

References:This aligns with the methodology discussed in the syllabus under the section on using AI for generating test cases from textual requirements​​.

In a neural network, the activation value of a neuron is determined by a combination of inputs from the previous layer, the weights of the connections, and the bias at the neuron level. Here’s a detailed breakdown:

Inputs for Activation Value:

Activation Values of Neurons in the Previous Layer:These are the outputs from neurons in the preceding layer that serve as inputs to the current neuron.

Weights Assigned to the Connections:Each connection between neurons has an associated weight, which determines the strength and direction of the input signal.

Individual Bias at the Neuron Level:Each neuron has a bias value that adjusts the input sum, allowing the activation function to be shifted.

Calculation:

The activation value is computed by summing the weighted inputs from the previous layer and adding the bias.

Formula: z=∑(wi⋅ai)+bz = \sum (w_i \cdot a_i) + bz=∑(wi​⋅ai​)+b, where wiw_iwi​ are the weights, aia_iai​ are the activation values from the previous layer, and bbb is the bias.

The activation function (e.g., sigmoid, ReLU) is then applied to this sum to get the final activation value.

Why Option A is Correct:

Option A correctly identifies all components involved in computing the activation value: the individual bias, the activation values of the previous layer, and the weights of the connections.

Eliminating Other Options:

B. Activation values of neurons in the previous layer, and weights assigned to the connections between the neurons: This option misses the bias, which is crucial.

C. Individual bias at the neuron level, and weights assigned to the connections between the neurons: This option misses the activation values from the previous layer.

D. Individual bias at the neuron level, and activation values of neurons in the previous layer: This option misses the weights, which are essential.

References:

ISTQB CT-AI Syllabus, Section 6.1, Neural Networks, discusses the components and functioning of neurons in a neural network.

"Neural Network Activation Functions" (ISTQB CT-AI Syllabus, Section 6.1.1).

A. Using a pixel comparison of the GUI before and after the change to check the differences.

Pixel comparison is a traditional method and does not involve AI. It compares images at the pixel level, which can be effective but is not an intelligent approach. It is not considered an AI usage and is the least likely usage of AI for detecting GUI changes.

B. Using computer vision to compare the GUI before and after the test object changes.

Computer vision involves using AI techniques to interpret and process images. It is a likely usage of AI for detecting changes in the GUI.

C. Using vision-based detection of the GUI layout changes before and after test object changes.

Vision-based detection is another AI technique where the layout and structure of the GUI are analyzed to detect changes. This is a typical application of AI.

D. Using a ML-based classifier to flag if changes in GUI are to be flagged for humans.

An ML-based classifier can intelligently determine significant changes and decide if they need human review, which is a sophisticated AI application.

Back-to-back testing is a method where the same set of tests are run on multiple implementations of the system to compare their outputs. This type of testing is typically used to ensure consistency and correctness by comparing the outputs of different implementations under identical conditions. Let's analyze the options given:

A. Comparison of the results of a current neural network model ML model implemented in platform A (for example Pytorch) with a similar neural network model ML model implemented in platform B (for example Tensorflow), for the same data.

This option describes a scenario where two different implementations of the same type of model are being compared using the same dataset. This is a typical back-to-back testing situation.

B. Comparison of the results of a home-grown neural network model ML model with results in a neural network model implemented in a standard implementation (for example Pytorch) for the same data.

This option involves comparing a custom implementation with a standard implementation, which is also a typical back-to-back testing scenario to validate the custom model against a known benchmark.

C. Comparison of the results of a neural network ML model with a current decision tree ML model for the same data.

This option involves comparing two different types of models (a neural network and a decision tree). This is not a typical scenario for back-to-back testing because the models are inherently different and would not be expected to produce identical results even on the same data.

D. Comparison of the results of the current neural network ML model on the current data set with a slightly modified data set.

This option involves comparing the outputs of the same model on slightly different datasets. This could be seen as a form of robustness testing or sensitivity analysis, but not typical back-to-back testing as it doesn’t involve comparing multiple implementations.

Based on this analysis, optionCis the one that describes a situation of back-to-back testing the least because it compares two fundamentally different models, which is not the intent of back-to-back testing.

Prevalence Rate and Model Performance:

The cancer detection system being developed by "Splendid Healthcare" needs to account for the fact that the type of cancer has a 2% prevalence rate in the population. This indicates that the dataset is highly imbalanced with far fewer positive (cancer) cases compared to negative (normal) cases.

Importance of Recall:

Recall, also known as sensitivity or true positive rate, measures the proportion of actual positive cases that are correctly identified by the model. In medical diagnosis, especially cancer detection, recall is critical because missing a positive case (false negative) could have severe consequences for the patient. Therefore, maximizing recall ensures that most, if not all, cancer cases are detected.

Importance of Precision:

Precision measures the proportion of predicted positive cases that are actually positive. High precision reduces the number of false positives, meaning fewer people will be incorrectly diagnosed with cancer. This is also important to avoid unnecessary anxiety and further invasive testing for those who do not have the disease.

Balancing Recall and Precision:

In scenarios where both false negatives and false positives have significant consequences, it is crucial to balance recall and precision. This balance ensures that the model is not only good at detecting positive cases but also accurate in its predictions, reducing both types of errors.

Accuracy and Specificity:

While accuracy (the proportion of total correct predictions) is important, it can be misleading in imbalanced datasets. In this case, high accuracy could simply result from the model predicting the majority class (normal) correctly. Specificity (true negative rate) is also important, but for a cancer detection system, recall and precision take precedence to ensure positive cases are correctly and accurately identified.

Conclusion:

Therefore, for a cancer detection system with a low prevalence rate, maximizing both recall and precision is crucial to ensure effective and accurate detection of cancer cases.

Recognizing digits from a scan of handwritten numbers using machine learning is an example of classification. Here’s a breakdown:

Classification: This type of machine learning involves categorizing input data into predefined classes. In this scenario, the input data (handwritten digits) are classified into one of the 10 digit classes (0-9).

Why Not Other Options:

Reinforcement Learning: This involves learning by interacting with an environment to achieve a goal, which does not fit the problem of recognizing digits.

Regression: This is used for predicting continuous values, not discrete categories like digit recognition.

Clustering: This involves grouping similar data points together without predefined classes, which is not the case here.

References:The explanation is based on the definitions of different machine learning types as outlined in the ISTQB CT-AI syllabus, specifically under supervised learning and classification​​.

The process of developing a machine learning model typically involves the use of three types of datasets:

Training Data:This is used to train the model, i.e., to learn the patterns and relationships in the data.

Validation Data:This is used to tune the model’s hyperparameters and to prevent overfitting during the training process.

Test Data:This is used to evaluate the final model’s performance and to estimate how it will perform on unseen data.

Let's analyze each option:

A. Training data - validation data - test data

This option correctly includes all three types of datasets used in the process of creating and validating a model. The training data is used for learning, validation data for tuning, and test data for final evaluation.

B. Training data - validation data

This option misses the test data, which is crucial for evaluating the model’s performance on unseen data after the training and validation phases.

C. Training data - test data

This option misses the validation data, which is important for tuning the model and preventing overfitting during training.

D. Validation data - test data

This option misses the training data, which is essential for the initial learning phase of the model.

Therefore, the correct answer isAbecause it includes all necessary datasets used during the process of learning and creating the model: training, validation, and test data​​.

Factors Affecting ML Functional Performance:The data pipeline, quality of the labeling, and biased data are all factors that significantly affect the performance of machine learning models. The number of classes, while relevant for the model structure, is not a direct factor affecting the performance metrics such as accuracy or bias.

Reference:ISTQB_CT-AI_Syllabus_v1.0, Sections on Data Quality and its Effect on the ML Model and ML Functional Performance Metrics​​​​.

To calculate the accuracy, recall, and specificity from the confusion matrix provided, we use the following formulas:

Confusion Matrix:

Actually Rotten: 45 (True Positive), 8 (False Positive)

Actually Fresh: 5 (False Negative), 42 (True Negative)

Accuracy:

Accuracy is the proportion of true results (both true positives and true negatives) in the total population.

Formula: Accuracy=TP+TNTP+TN+FP+FN\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}Accuracy=TP+TN+FP+FNTP+TN​

Calculation: Accuracy=45+4245+42+8+5=87100=0.87\text{Accuracy} = \frac{45 + 42}{45 + 42 + 8 + 5} = \frac{87}{100} = 0.87Accuracy=45+42+8+545+42​=10087​=0.87

Recall (Sensitivity):

Recall is the proportion of true positive results in the total actual positives.

Formula: Recall=TPTP+FN\text{Recall} = \frac{TP}{TP + FN}Recall=TP+FNTP​

Calculation: Recall=4545+5=4550=0.9\text{Recall} = \frac{45}{45 + 5} = \frac{45}{50} = 0.9Recall=45+545​=5045​=0.9

Specificity:

Specificity is the proportion of true negative results in the total actual negatives.

Formula: Specificity=TNTN+FP\text{Specificity} = \frac{TN}{TN + FP}Specificity=TN+FPTN​

Calculation: Specificity=4242+8=4250=0.84\text{Specificity} = \frac{42}{42 + 8} = \frac{42}{50} = 0.84Specificity=42+842​=5042​=0.84

Therefore, the correct combinations of accuracy, recall, and specificity are 0.87, 0.9, and 0.84 respectively.

References:

ISTQB CT-AI Syllabus, Section 5.1, Confusion Matrix, provides detailed formulas and explanations for calculating various metrics including accuracy, recall, and specificity.

"ML Functional Performance Metrics" (ISTQB CT-AI Syllabus, Section 5).

A. Black box attacks based on adversarial examples create an exact duplicate model of the original.

Black box attacks do not create an exact duplicate model. Instead, they exploit the model by querying it and using the outputs to craft adversarial examples without knowledge of the internal workings.

B. These attack examples cause a model to predict the correct class with slightly less accuracy even though they look like the original image.

Adversarial examples typically cause the model to predict the incorrect class rather than just reducing accuracy. These examples are designed to be visually indistinguishable from the original image but lead to incorrect classifications.

C. These attacks can't be prevented by retraining the model with these examples augmented to the training data.

This statement is incorrect because retraining the model with adversarial examples included in the training data can help the model learn to resist such attacks, a technique known as adversarial training.

D. These examples are model specific and are not likely to cause another model trained on the same task to fail.

Adversarial examples are often model-specific, meaning that they exploit the specific weaknesses of a particular model. While some adversarial examples might transfer between models, many are tailored to the specific model they were generated for and may not affect other models trained on the same task.

Therefore, the correct answer isDbecause adversarial examples are typically model-specific and may not cause another model trained on the same task to fail​​.

Coverage for Neuron Activation Values:Sign change coverage is used to ensure that test cases cause each neuron to achieve both positive and negative activation values. This type of coverage ensures that the neurons are thoroughly tested under different activation states.

Reference:ISTQB_CT-AI_Syllabus_v1.0, Section 6.2 Coverage Measures for Neural Networks, which details different types of coverage measures, including sign change coverage​​.