Thanks to Tom Raggett for this link from Joel on Software about the pitfalls of measurement when applied to knowledge workers:
It seems like any time you try to measure the performance of knowledge workers, things rapidly disintegrate, and you get what Robert D. Austin calls measurement dysfunction. His book Measuring and Managing Performance in Organizations is an excellent and thorough survey of the subject. Managers like to implement measurement systems, and they like to tie compensation to performance based on these measurement systems. But in the absence of 100% supervision, workers have an incentive to "work to the measurement," concerning themselves solely with the measurement and not with the actual value or quality of their work.
Is a worker who starts to "work to the measurement" ever likely to have concerned themselves with the "actual value or quality of their work" in the first place? I would say: only one who is easily jaded. And if they find themselves jaded, why aren't they moving on? My concern about such measurement is not in the dysfunction it engenders in workers, but in the dysfunction it suggests in the managers. Surely a manager has no need of such a system if they are truly in touch with the work they are supposed to be managing. Measurement systems and their kin are brought into play because something is inherently wrong. Managers who have lost faith in their ability to justify their own intuitive assessment of performance (ie who have become jaded themselves) are the ones who will grasp at measurement systems. If they stick to these, then no-one can accuse their own intuitive decisions of being sexist / racist or in any way skewed. A manager capable of being literate about his/her own intuitive knowledge of a situation, and who can defend their decisions, is one to be kept hold of. Managers that blindly accept performance measurement as just another quality system to be implemented, are jaded, should probably have moved on, but sadly, probably never will.