Estimating cost of labor for a controlled experiment
- by Lorin Hochstein
Let's say you are a software engineering researcher and you are designing a controlled experiment to compare two software technologies or techniques (e.g., TDD vs. non-TDD, Python vs. Go) with respect to some qualities of interest (e.g., quality of resulting code, programmer productivity).
According to your study design, participants will work alone to implement a non-trivial software system. You estimate it should take about six months for a single programmer to complete the task. You also estimate via power analysis that you will need around sixty study participants to obtain statistically significant results, assuming the technologies actually do yield different outcomes.
To maximize external validity, you want to use professional programmers as study participants. Unfortunately, it isn't possible to find professional programmers who can volunteer for several months to work full-time on implementing a software system. You decide to go the simplest route and contract with a large IT consulting firm to obtain access to programmers to participate in the study.
What is a reasonable estimate of the cost range, per person-month, for the programming labor?
Assume you are constrained to work with a U.S.-based firm, but it doesn't matter where in the U.S. the firm itself or the programmers or located.
Note: I'm looking for a reasonable order-of-magnitude range suitable for back-of-the-envelope calculations so that when people say "Why doesn't somebody just do a study to measure X", I can say, "Because running that study properly would cost $Y", and have a reasonable argument for the value of $Y.
