The Forgotten Elements of Systems Engineering: Part 1 Test and Evaluation

Published by on April 6, 2015 at 9:00 am.

Part 1: Test and Evaluation

V Diagram

Complexity is in systems engineering’s very nature. As systems engineers we create solutions to complex problems. So, of course from time to time, we forget important elements of the system’s lifecycle. We might even believe we are remembering them, but we do not give these elements the time and attention they desperately need. I have dedicated a three part series to “The Forgotten Elements of Systems Engineering.”

The first part in our three part series on “The Forgotten Elements of Systems Engineering,” shines light on the fact that many systems engineers fail to remember Test and Evaluation (T&E) planning during the design process. T&E is a part of the Verification and Validation in a system lifecycle. It is the plan to execute test cases, test plans, test procedures, modeling and simulation and report the issues found. Unfortunately, many develop the T&E late in the lifecycle, often starting half-way through acquisition. Stakeholders, such as the testers do not interact in time to understand the system needs. They make the perfunctory Test and Evaluation Master Plan (TEMP) early in the project. The TEMP contains countless TBDs, but the plan does not evolve with the system’s changes. When the time comes to test the testers often have to start with little or nothing in the way of a real T&E plan and requirements.

NDIA suggests “incorporating T&E expertise early in the acquisition cycle (ITEA Journal)” Not creating a plan for the T&E early in the lifecycle causes lots of problems. The main problems are schedule delays and increased costs. If test requirements (detailed metrics used to determine performance and operational suitability) and plans (which include facilities, equipment, integrated schedules and costs) are developed early, testers and operational personnel can validate the system’s capabilities. If it passes these criteria, it will meet their operational needs. Showing the users how the system is validated provides confidence that testing has been sufficient to take the risk of introducing a new system to the operational environment. Bringing in a new system that does not work costs not only money and time, but in many cases it can even cause loss of life.

It’s important for stakeholders, especially the testers to interact early on in the system’s lifecycle to better understand all the test needs. The testers, at a company I used to work with, would always complain to me that systems come in for testing without any notice. The test developers had no understanding of the system, sometimes the system wouldn’t even come with requirements, and the testers were expected to get straight to testing and evaluating this foreign system. This is unacceptable. The testers should be refining the TEMP throughout the lifecycle.

If you look above at the V-model you can see that each step going down the left side of the V-model corresponds to a step on the right. The entire right side is devoted to testing. The V-model encourages T&E to receive output in each phase of the lifecycle. Due to improved modeling languages and model-based systems engineering (MBSE) tools, T&E is much easier to perform earlier and simultaneously with other steps in the lifecycle. LML was designed to support the complete lifecycle, so aspects of that are considered in the language. Most of the entities are common to other phases (i.e., Assets, Actions, etc.). Perhaps the greatest influence T&E had on LML was in the Measure entity attributes. If you look closely at them you will see information that is extremely useful for T&E. The “Value” can be the measured value, while the other values (Projected, Threshold, and Objective) are often estimated values. LML also included Improvement Direction and Tolerance, as information you would want to measure. The Measure entities relate to Time and Location as well. A number of the Labels are also T&E related (COI, MOE, MOP, MOS, TPM, etc.). The depth of LML, is one of the many reasons Innoslate uses LML for its main modeling language, along with SysML.

Testing and Evaluating in a real time simulator

So by having the language entities needed to capture test results, as well as the processes, procedures, equipment, etc. should improve the inclusion of T&E in the early phases of design, as well as capture the data. These more defined modeling languages allow for modern MBSE tools that can help you create a test plan, test reports, and test procedures. Tools, like Innoslate have built in automated modeling and simulation. Simulators such as the Monte-Carlo simulator and the Discrete Event simulator enable you assist in testing physical aspects of the system.

T&E is one of the most forgotten elements in systems engineering. Forgetting T&E can seriously delay a system’s project and dramatically increase the costs and risks. Remember these three rules to maintain proper T&E processes: 1) Use modern modeling languages and software tools to start planning early in the system’s lifecycle. 2) Involve all the stakeholders, especially the testers throughout the lifecycle. 3) Evolve the T&E plan with the system’s changes. If you can remember these rules, you will have a successful system project.