Psychology of InstrumentationPosted: February 4, 2009
Cary Millsap reread Knuth and found a fascinating quote:
“I’ve become convinced that all compilers written from now on should be designed to provide all programmers with feedback indicating what parts of their programs are costing the most; indeed, this feedback should be supplied automatically unless it has been specifically turned off.”
Knuth wrote this in 1974. Its been 35 years. Compilers (and runtime environments such as JVM) still do not provide this feedback automatically. I believe that the need for this did not diminish.
Since compilers do not provide performance feedback, there is a very large market for instrumentation vendors. The sell you a bit of code that you can integrate in your programs, or install on your J2EE application servers or on your database, and it will give you exact information about where your program is spending its time.
I’ve worked for an instrumentation vendor, and I also talked to many of them as part of my production DBA role. Whenever an instrumentation vendor talks to a prospective customer, the first question is always: “What is the overhead?”. Not “How it can help me?”, “How much I can expect to improve my performance?” or “Is it easy to use?”. The answer to the first question is always the same, by the way, 5-10%.
The reason that this is always the first question is that people in general are always more concerned about losing something currently they have than about gaining something new. This tendency is called “endowment effect”, and it is quite powerful. Salespeople will tell you that supply is limited when you hesitate about buying something, because they know that fear of losing (even if it is just losing the option of buying this) is a more powerful motivator than any attractive features of the product.
In our case, developers and DBAs are more worried about losing something they have (Performance benefits of running non-instrumented code) than they are interested in the benefits of knowing where their product is spending time.
Suppose that all compilers and runtime environments had instrumentation enabled by default, with the option to turn it off. The same endowment effect would now work in our favor! No one would turn this off, because they’ll be more concerned about losing information they currently have (and became used to having) than they’ll care about the performance benefits of turning this instrumentation off.
The pervasive lack of instrumentation in software products is more a result of psychological bias than real technical concerns. Software vendors can work around these psychological issues by building instrumentation as a default into tools involved in the development and deployment process. Just as Knuth said 35 years ago.
In the mean while, I hope that by being aware of this irrational psychological bias, developers, DBAs and production application owners can overcome it and make sure they have the data they need to monitor and improve their applications.