CT-GenAI ISTQB Certified Tester Testing with Generative AI (CT-GenAI) v1.0 Questions and Answers

Classical Machine Learning(which includes algorithms like Random Forests, Support Vector Machines, and Linear Regression) is characterized by its reliance onFeature Engineering. This is the process where human experts manually select, extract, and transform raw data into a set of "features" or variables that the algorithm can process. For instance, in a classical ML model predicting software defects, a tester might have to manually define features like "lines of code changed" or "number of previous bugs." In contrast,Deep Learningand its subset,Generative AI(Options B and D), utilize "Representation Learning." This means the multi-layered neural networks automatically identify and extract the relevant features from raw, often unstructured data (like text or images) without explicit human instruction.Symbolic AI(Option A) is based on hard-coded logical rules rather than data-driven learning. Understanding this distinction is fundamental for testers, as it determines the level of data preparation required: Classical ML requires high human effort in data structuring, while GenAI requires high effort in prompt engineering and grounding.

In the evaluation of GenAI outputs for testing, metrics must align with the specific nature of the task. Foranomaly identification, the goal is to correctly identify true issues without an overwhelming number of false positives; therefore,Precisionis the critical metric (the ratio of true anomalies to the total flagged). Conversely, forpartition testing(identifying valid and invalid input classes), the goal is thoroughness and ensuring no significant category is missed.Recallis the most appropriate metric here, as it measures the model's ability to "call back" or cover all possible relevant partitions from the requirement set. As highlighted in the CT-GenAI syllabus, evaluating AI effectiveness often requires a combination of these model-performance metrics. While "Accuracy" (Option D) provides a general view, it is often misleading in imbalanced testing scenarios (like anomaly detection where anomalies are rare). By using Precision and Recall together, a test organization can quantitatively assess if the AI is both trustworthy in its alerts and comprehensive in its test design coverage.

This scenario describes a failure in the model's ability to apply logical weight to specific domain concepts, specifically in the context of Risk-Based Testing (RBT). When an LLM ranks a low-impact UI element (a tooltip) higher than a critical functional failure (payment processing), it demonstrates a "Reasoning error in risk calculation logic." While LLMs can follow formulas like $Risk = Likelihood \times Impact$, they may lack the deep semantic understanding of "Impact" within a specific business domain unless explicitly guided. This is not necessarily a hallucination (Option C), as the model isn't necessarily inventing facts, but rather misapplying the logic of prioritization. It is also distinct from dataset bias (Option D), which would involve a systematic skewing across all outputs. In professional testing, this type of error highlights the necessity of "human-in-the-loop" verification. Testers must review AI-generated prioritizations to ensure that the logical deductions align with the actual business risk and technical criticality of the features being tested.

It is important to distinguish between a general-purposeChatbotand a specializedLLM applicationwithin a testing framework. A chatbot is primarily designed for multi-turn, conversational interactions where the user can ask questions and receive answers in a natural language format. While useful for general queries, it often lacks the specialized workflow integration needed for rigorous testing. Conversely,LLM applications(or "LLM-powered tools") are built with a specific "scope" in mind, such as automated test generation, code analysis, or requirement mapping. These applications often use the LLM as an underlying engine but surround it with specific UI components, data connectors (like RAG), and fixed task-oriented prompts to achieve a defined testing outcome. While chatbots are "free-form," LLM apps are "capability-driven." This distinction is key for organizations defining a GenAI strategy; simply providing a chatbot to testers is rarely sufficient. Instead, organizations should develop or adopt LLM applications that integrate directly into the CI/CD pipeline and provide structured, actionable test artifacts that support defined quality engineering tasks.

The primary differentiator between an LLM-powered test infrastructure and a traditional chatbot is the move from "deterministic" to "probabilistic" logic. Traditional chatbots (Option D) rely on "if-then" logic, decision trees, and predefined scripts. They can only respond to queries that match specific keywords or patterns mapped in their database. In contrast, an LLM-powered infrastructure utilizes the generative capabilities of Large Language Models to synthesize and create new content based on context. This allows it todynamically generate test insights(Option A)—such as predicting potential regression risks based on unstructured code diffs or drafting test cases for a brand-new feature described in natural language. While traditional bots provide fixed, scripted responses (Option B), LLMs can "reason" through multi-step testing problems and provide nuanced explanations. This contextual awareness is powered by the model's training on vast amounts of technical documentation, enabling it to assist in exploratory testing and complex analysis that traditional, rule-based systems simply cannot handle.

Generative AI is exceptionally capable of creating structured and unstructured data, but its role is limited to "generation" and "transformation," not infrastructure management or direct database administration. Creating production database backups (Option A) is a physical data management task involving the copying of actual stateful data from a server to storage; this is handled by database management systems (DBMS) and DevOps pipelines, not LLMs. Conversely, LLMs excel at the logic-based tasks listed in the other options. They can analyze requirements to identify and set boundary values (Option B) for input validation. They are also highly effective at creating combinatorial data (Option C), such as pairwise or all-combinations tables, by understanding the relationships between variables. Finally, one of the most powerful uses of GenAI in testing is generating synthetic datasets (Option D)—creating "fake" but realistically structured data that mimics production patterns without exposing Sensitive Personally Identifiable Information (SPII), thereby supporting privacy-compliant testing.

The effectiveness of an LLM is heavily dependent on the specificity of itsOutput Format. While role definition (Option C) and technical instructions (Option D) are helpful, the most significant "value add" for a test lead is receiving information that is directlyactionable. By expanding the output format to include structuredrisk predictions, severity levels, and recommended actions(Option B), the tester is forcing the LLM to perform a deeper level of analysis. Instead of just "flagging trends," the model must now synthesize the data to determinewhya trend is a risk andwhatthe team should do about it. This aligns with the "Advanced Prompting" section of the CT-GenAI syllabus, which emphasizes using AI for decision support. A structured report that includes a "timeline for intervention" allows the human tester to quickly validate the AI's logic and make informed decisions, transforming the LLM from a simple data summarizer into a strategic predictive tool that actively supports the maintenance of release quality and schedule adherence.

Differentiating between prompting techniques is essential for a tester to select the right tool for the task.Few-shot promptingis characterized by providing the model with a few examples of inputs and desired outputs, allowing it to learn the pattern and format.Prompt Chaininginvolves breaking a complex task into a sequence of smaller, interconnected prompts, where the output of one step becomes the input for the next (e.g., first extract requirements, then generate test cases from those requirements).Meta-promptingis a more advanced technique where the user asks the LLM to help design, write, or refine the prompt itself—essentially using the AI as a "prompt engineer" to optimize the instructions. Option D correctly identifies these core characteristics. Options A, B, and C contain fundamental mischaracterizations: for instance, Few-shotrequiresexamples (contradicting A), and Chaining is theoppositeof a single prompt (contradicting A). Mastering these distinctions allows testers to move from simple "chatting" to sophisticated AI orchestration that can handle complex, multi-stage testing workflows with high reliability.