Once you have an explicit hypothesis, you must decide what variables can effect its truth, and how much control you have over each variable. A state variable or independent variable is a variable that can be manipulated and will characterize your project and influence your evaluation results. The outcome, in turn, is evidenced by values of the dependent variables, which will change with changes in the independent variables.In formal experiments you sample over the independent variables, while in a case study you select a value of the variable that is typical for your application environment.
One key motivation for using a formal experiment rather than a case study or a survey is that the results of an experiment are usually more generalizable.
In order to make your investigation meaningful you should know how each of the three techniques helps in answering different research questions.
- Confirming theories and "conventional wisdom". Case studies and surveys can be used to confirm the truth of these claims in a single organization, while formal experiments can investigate the situations in which the claims are generally true.
- Exploring relationships. The relationship can be suggested by a case study or survey. A formal experiment can be designed to test the degree to which the relationship holds.
- Evaluating the accuracy of models. Formal experiments can confirm or refute the accuracy of models. Models present a particular difficult problem when designing an experiment or case study, because their predictions often affect the outcome. That is, the predictions become goals, and the developers strive to meet the goals, intentionally or not.