TMMi-P_Syll2020 TMMi - Test Maturity Model Integration Professional Questions and Answers

In a staged model like TMMi, an organisation follows a predefined improvement path that includes a set of process areas organised by maturity levels. Each level builds upon the previous one, and the organisation progresses in a step-by-step manner through these maturity levels to improve its processes. The staged model is well-structured and prescriptive, offering a clear improvement roadmap for the organisation.

An indirect benefit of a test improvement programme is improved staff motivation. While direct benefits, such as shortened lead time or higher defect detection rates, are focused on measurable improvements, indirect benefits include enhanced morale and engagement of the staff involved. A well-structured test improvement programme can lead to better recognition of the testing function within the organisation, leading to increased satisfaction and motivation among testing professionals.

The process area Test Lifecycle and Integration at TMMi Level 3 focuses on establishing a coherent test process across multiple test levels and integrating the test lifecycle with the development lifecycle. This process area addresses the creation of a master test plan , which is essential for ensuring consistency and avoiding redundancy or omissions across different test levels.

The specific goal and practice for establishing a master test plan are well documented within the Test Lifecycle and Integration process area. Practices include performing a product risk assessment, establishing the test approach, and developing the master test plan itself​​.

Advanced Reviews is a TMMi Level 4 process area. At TMMi Level 4 (Measured) , organizations focus on making testing a quantifiable process , using metrics and measurements to evaluate and improve product quality and the testing process. Advanced Reviews involve more structured and formalized review processes that are integrated with the dynamic testing approach to measure and improve product quality earlier in the lifecycle.

The Test Policy and Strategy is part of Level 2 .

Defect Prevention is part of Level 5 .

Test Lifecycle and Integration is part of Level 3 .

TMMi References :

TMMi defines Advanced Reviews as a Level 4 process area aimed at improving product quality through more rigorous and measurable review processes​​.

The Testing quadrants model is an Agile practice that helps establish a test strategy, as it provides a framework for categorising different types of tests (e.g., business-facing or technology-facing) and helps in balancing test activities. Within the TMMi Level 2 process area for Test Policy and Strategy, testing quadrants offer an Agile way to define and implement a test strategy that ensures comprehensive testing across various levels and types of testing activities.

Informal assessments, as described in the TMMi framework, are often quicker and cheaper as they do not require the extensive corroboration of evidence that formal assessments do. One key characteristic of informal assessments is that they may rely solely on interviews, without using other forms of corroboration such as documentation reviews or observations. These informal assessments provide an indicative view of the organization's maturity level but do not result in formal certification​

Formal assessments, in contrast to informal ones, are conducted with more rigor, and the results can lead to an official certification of maturity level. A formal assessment report includes a validated maturity level rating, making it distinct from informal assessments, which are used only for indicative purposes and do not result in an official rating. Thus, only formal assessments provide an assessment report with a valid maturity level rating​​.

Both Test Organization and Test Training Program are process areas associated with TMMi Level 3 (Defined). At this level, the organisation establishes a formal test organisation and ensures that there is a dedicated test training program. This helps to institutionalise testing as a profession and ensures the systematic development of skills needed for testing activities across the organisation.

The specific goals of the Advanced Reviews process area include:

Coordinate the peer review approach with the dynamic test approach : Ensuring that static testing, such as peer reviews, is aligned with the dynamic testing approach to optimize testing efforts.

Adjust the test approach based on review results early in the life cycle : This goal aims to improve the test approach by using early feedback from reviews to make adjustments that can enhance testing later in the lifecycle.

The other options (2, 4, and 5) refer to different process areas or goals. For example, Perform peer reviews is part of the Peer Reviews process area at TMMi Level 3, not Advanced Reviews.

TMMi References :

The Advanced Reviews process area at TMMi Level 4 focuses on aligning peer reviews with the dynamic test approach and adjusting the test approach based on review results​​.

In the TMMi framework, specific goals can indeed have different specific practices across maturity levels. The practices associated with a specific goal evolve as an organization matures, reflecting the need for more sophisticated approaches at higher levels of maturity. For example, test design practices at Level 2 are more basic compared to the more comprehensive practices at Level 3, which include integration with other lifecycle processes.

The other statements are incorrect. For instance, sub-practices are informative and not mandatory, and while generic goals have the same set of practices across process areas, specific goals vary by maturity level.

TMMi References :

TMMi process areas include specific goals that may have different practices depending on the maturity level, as the process areas evolve with higher maturity​​.

One of the key differences between informal and formal assessments is that informal assessments are not as rigorous. Informal assessments are more flexible, quicker, and less costly; however, they do not provide the same level of detail or accuracy as formal assessments. The statement that "informal assessments provide a result just as accurate as with a formal assessment" is therefore false. Formal assessments involve multiple types of evidence and a more thorough evaluation, while informal assessments typically rely on fewer types of evidence (often only interviews)​​.

CMMI version 2 recognises the importance of testing in software development and includes two dedicated practice areas: Verification and Validation . These areas specifically address the processes required to ensure that a system or component meets its specified requirements and that the final product is fit for its intended use. While CMMI does not provide a testing-specific maturity model, these practice areas contribute to the overall quality management framework, and organisations adopting CMMI can leverage these processes to improve their testing practices. The TMMi framework complements CMMI by providing more detailed guidelines for testing.

The specific practice "Define a coordinated approach that coordinates both static and dynamic testing" is part of the Advanced Reviews process area at TMMi Level 4. The goal of this process area is to integrate static and dynamic testing, particularly through peer reviews, which helps optimise both defect detection and product quality management throughout the lifecycle​​​.

In the TMMi framework, a Specific Goal is unique to a particular process area and represents the essential outcomes that must be achieved for that process area. In contrast, a Generic Goal applies across multiple process areas and ensures that the processes are institutionalised, managed, and aligned with broader organisational objectives. Generic Goals help maintain consistency and effectiveness of processes across the organisation, while Specific Goals focus on the particular needs of an individual process area.

A. Required component (II) : Must be achieved. These are specific and generic goals that are mandatory for reaching a particular maturity level.

B. Expected component (III) : Will typically be implemented. Expected components include specific and generic practices, which provide guidance on how to achieve required goals.

C. Informative component (I) : Supports the understanding. Informative components provide details that help organisations understand and implement the required and expected components effectively, such as sub-practices and example work products.

In the scenario described, where an organisation suffers from poor test estimations despite many practices being in place, the process area that requires attention is Test Planning. Specifically, SG 3 (Establish Test Estimates) focuses on developing well-founded estimates for testing effort and cost. This includes creating a top-level work breakdown structure, defining the test lifecycle, and determining estimates for the test effort and costs associated with different test activities. By addressing these practices, the organisation can improve its ability to generate accurate test estimations, leading to better project planning and resource allocation.

In the Learning phase of the IDEAL model, one of the critical activities is "Analyse and Validate" . This involves reviewing the entire improvement process to determine whether the intended goals were achieved and identifying what worked well and what could be improved. The objective is to learn from the experience to enhance future improvement initiatives and continually refine the ability to implement change​​.

The Acting phase of a test improvement cycle focuses on executing the planned improvements. Key activities during this phase include creating solutions to address identified problems, planning actions to implement those solutions, and piloting solutions before full-scale deployment. These steps ensure that the improvements are well-defined, tested in a controlled environment, and refined before being fully implemented across the organisation.

Regular code reviews and architectural reviews are part of the Advanced Reviews process area, which belongs to TMMi Level 4 . This process area emphasizes conducting static reviews (e.g., code and design reviews) early in the lifecycle to measure product quality and adjust the testing strategy based on the insights gained from these reviews.

TMMi References :

The Advanced Reviews process area in TMMi Level 4 aims to ensure that static testing activities such as code and architectural reviews are used to measure product quality early in the lifecycle​​.

The absence of a master test plan spanning multiple test levels points to a weakness in the Test Lifecycle and Integration process area, which is a TMMi Level 3 process area. This process area focuses on ensuring that testing is integrated throughout the development lifecycle and includes the development of comprehensive test plans (including master test plans) that cover multiple test levels and align with the overall testing strategy.

TMMi References :

The Test Lifecycle and Integration process area at TMMi Level 3 addresses the need for comprehensive and integrated test planning across all test levels​​.

The specific goals of the Test Process Optimization process area at TMMi Level 5 include:

Select test process improvements .

Evaluate new testing technologies to determine their impact on the testing process .

Deploy test process improvements .

However, "Determine, plan and implement test process improvements" (2) and "Monitor product quality against plan and expectations" (5) are not part of the Test Process Optimization goals. Instead, these activities belong to other process areas, such as Test Measurement or earlier maturity levels.

TMMi References :

The goals for Test Process Optimization at TMMi Level 5 focus on selecting and deploying process improvements and evaluating new technologies​​.

Each statement about TMMi level 2 in an Agile context requires careful consideration within the framework:

(a) : False. While Agile projects indeed use metrics like Velocity, TMMi requires tracking end-results such as escaped defects. Both team performance and outcomes, including defect-related metrics, are significant. Test performance indicators at TMMi level 2 in an Agile context should not solely focus on team performance metrics .

(b) : True. The product risk assessment process in Agile projects typically adopts a more lightweight approach compared to traditional sequential lifecycle models. Agile teams often conduct risk assessments in a more collaborative and iterative manner .

(c) : False. In Agile, monitoring and controlling processes are still critical to ensure goals and quality targets are met, even if the methodology allows for flexibility and adaptation .

(d) : False. Traceability is still relevant in Agile, although it may be implemented differently. Agile methodologies such as Scrum do not eliminate the need for traceability between requirements, test conditions, and tests, which is an essential part of TMMi process areas .

(e) : True. Agile allows for flexibility in defining the test environment closer to the execution phase, enabling changes that reflect evolving requirements and design decisions .

Thus, the correct combination is A : Statements (a) and (e) are true, while statements (b), (c), and (d) are false​ .

The Diagnosing phase in the IDEAL test improvement cycle is focused on understanding the current state of the organization's testing processes and identifying the areas for improvement. One of the key activities during this phase is Develop Recommendations , which involves analysing the current process against the goals and proposing areas where improvements can be made to achieve the desired state. Other activities in this phase include characterizing the current and desired states .

For safety-critical systems, reliability testing is a key concern. The identified shortcoming relates to the lack of formal techniques and approaches for reliability testing. The Non-Functional Testing process area, at TMMi Level 3, addresses testing non-functional aspects like reliability, performance, and usability. This process area would need specific attention as it provides a structured approach to assess non-functional risks, define testing techniques, and execute tests to verify the reliability of the product, which is critical for medical systems.

A typical business reason for starting a test improvement program is to achieve higher productivity . Test process improvements can lead to better efficiency in test execution, reduction of defects, and optimised resource usage, all of which contribute to the overall productivity of the organisation. By improving testing processes, organisations can reduce time-to-market and minimise costs associated with fixing defects post-production, which in turn boosts productivity.

The TMMi process area Test Planning is indeed supported by several CMMI process areas , including Project Planning , Requirements Management , and Risk Management . These CMMI areas provide foundational support for ensuring that the test planning process incorporates adequate project management, requirement traceability, and risk identification, all of which are critical for effective testing.

Specific Goal 2 of the CMMI Validation process area relates to validation activities, not peer reviews, and therefore does not support the TMMi Peer Reviews process area.

Causal Analysis and Resolution is linked to Defect Prevention in TMMi, not Quality Control.

Performance Management does not directly support the Non-Functional Testing process area in TMMi.

TMMi References :

The TMMi Test Planning process area is supported by relevant CMMI process areas, such as Project Planning , which helps in aligning project goals with test planning activities​​.

With a 40% achievement in the Test Planning process area, the rating would be "Partly achieved." According to TMMi, a process area is rated "Partly achieved" when the percentage of achievement is between 15% and 50%. This indicates that some practices have been implemented, but significant improvement is still required to meet the full requirements of the process area.

The Debugging-oriented phase from Gelperin and Hetzel's evolutionary testing model is associated with Maturity Level 1 "Initial" of TMMi. At this maturity level, testing is not yet a distinct and well-defined process but is often interleaved with debugging. Organizations at this level have chaotic and ad-hoc testing activities, with no separation between testing and debugging.

Maturity Level 1 organizations lack formal processes, and testing is often reactive, performed alongside fixing bugs in the code. This is in line with the debugging-oriented phase, where testing and debugging are seen as part of the same activity.

TMMi References :

The TMMi framework draws directly from the evolutionary model of Gelperin and Hetzel, associating the Debugging-oriented phase with Level 1, which describes a chaotic, unmanaged testing environment​​.

In the context of TMMi, the focus of test process improvement is usually driven by business needs that aim to improve the efficiency, effectiveness, and predictability of the testing process. The typical business reasons for test improvement include:

Increase predictability of testing (Option B): This refers to making the testing process more reliable and consistent, allowing better planning and forecasting of testing activities.

Reduce the costs of failure by improving effectiveness of testing (Option C): This is a direct goal of test process improvement, as it helps catch defects earlier, reducing costs associated with defects in later stages of the lifecycle.

Reduce time to market by increasing efficiency of testing activities (Option D): By making testing more efficient, organizations can release products faster while maintaining or improving quality.

Option A, Introduce a new process improvement framework , is NOT a typical business reason for test improvement. This refers to the introduction of a framework, which could be part of a process improvement strategy but is not a direct business reason for improving testing. Test improvement efforts are generally focused on achieving tangible business benefits, such as cost reduction, risk mitigation, or faster delivery, rather than the implementation of a specific framework for its own sake.

TMMi References :

TMMi emphasizes business-driven goals for process improvement that align with reducing risks, enhancing quality, and improving testing efficiency and effectiveness​.

The specific goal "Project Goals for Product Quality and their Priorities are Established" belongs to the Product Quality Evaluation process area, which is part of TMMi Level 4. This process area focuses on defining measurable and prioritised project goals for product quality and managing actual progress towards those goals throughout the lifecycle. Product quality goals typically include both functional and non-functional aspects of the system.