Considerations

Systems Engineering is a creative activity that requires us to explore problems with imperfect knowledge. We can use analytical and design techniques to help us think problems through, but the results are often imperfect. What is more, at the end of analysis and design we need to do more work - some implementation - before we understand the imperfections of our thinking.

In contrast, we use experiments to help us choose between 2 or more alternatives in a way that is objective and fast. Experimentation is objective because a key aim is to obtain data that helps us to make decisions. Experimentation Is fast because we focus on trying a single option at a time with a carefully defined and controlled scope.

Using experimentation, we support our bias to action by avoiding extended periods of thought, analysis and design. We prefer to try stuff out. We prefer to gather data. We prefer to make data-led decisions based on what we have learned.

Every experiment is based on a hypothesis - the concept we want to test. The result of an experiment may either prove the concept, or disprove it. Neither of these outcomes is a failure because we have obtained data that will help us to make a decision. We might demonstrate that the concept we are testing does not work, but that is an objective result.

An experiment is only a failure when we fail to gather the data we need to make a decision.

Minimum Viable Product

When creating new solutions or substantial new features in an existing solution we will often use the approach of a Minimum Viable Product (MVP) to test the value of the solution or feature. The use of an MVP is a form of experiment. The hypothesis is that our MVP is a valuable starting point. We test the hypothesis by receiving customer feedback and through the use of volumetric and performance data where appropriate.

Levels

Green

Effective Experimentation Is Routine

Experiments are routinely used as the basis for making decisions. Experiments are performed using a level of formality appropriate to the scale, complexity and impact of the experiment. Key experiments are reflected in the Roadmap.

Every experiment has measurable success criteria so the outcomes can be objectively assessed. Data is measured, recorded and analysed to enable objective decision making.

Lessons learned by performing experiments are shared routinely.

Amber

Inconsistent Use Of Experimentation

Experiments are sometimes used as the basis for making decisions. Experiments may not be well defined and may lack objective measures and criteria by which the outcome of the experiment can be judged.

It is hard to use the outcomes of experiments to make objective decisions.

Lessons learned by performing experiments may not be shared. It is hard to improve the way experiments are performed because of the lack of shared experience.

Red

Little Effective Use Of Experimentation

Experiments are very rarely used to support decision makIng, or are not used at all. A lack of data means it is hard to evaluate the success of any experiment.

Outcomes of ill-defined experiments cannot support subsequent decision making because of the lack of data.

There is no effective learning from performing experiments.