Rating Programmers

The Programmer Competency Matrix is an insightful tool to aid in objectively rating programmers in terms of ability when you have no history of results, like in an interview.

In computer science, however, they are missing Boolean algebra and discrete mathematics which are prerequisites for successfully working with data structures and algorithms. Also, in Software Engineering, I’d like to see levels of ability in risk management. Those two nits aside, this matrix is a handy tool to objectively rate interviewees or ranking software engineers by ability within an organization (e.g. associate, senior, principal, architect, fellow, etc.). Too often people are rated subjectively on the wrong criteria, for example: easy to work with rather than the results they produce and the abilities they possess. A tool like this can help you get over objective rating.

In my experience, titles do a have a function; they communicate to people outside your organization what you do and the level in which you contribute. Markup to the titles like senior in senior software developer, has little value. The meaning of associate, senior or architect is so subjective, they tell you nothing. When I look at a business card and I see the title senior software developer, I ignore the prefix (e.g. senior) and only assume they work with software. This is a side-effect of pervasive subjective ratings and a lack of tools to objectively measure ability.

When selecting a method to rate or rank, there’s a temptation to make it so clear, you don’t have to think. The idea is the perfect solution will arrive at the same conclusion every time. However, the best solution frequently arrives at the right conclusion much of the time. Implementing a black-or-white rating scheme invites people to “game it,” or put another way, algorithmic rating invites sub-optimization. A great example of inviting sub-optimization is in a BusinessWeek article, “Data Mining Moves to Human Resources” (via Slahsdot). The articles describes how Cataphora uses algorithms to rate employees, using behavior and communications. Though Cataphora provides objective criteria, it’s rigidly applied. Today, an algorithm can’t holistically measure effectiveness of two different programmers in a meaningful way. Once people know the metrics, and the metrics will be figured out, people will sub-optimize their work, prioritizing conformance to the metrics over achieving business goals.

Imagine if programmers were rated on clarity and volume of their communications. Suddenly you would have programmers spending significant time writing email and documentation. While that’s not entirely bad, email and documentation don’t directly contribute to revenue as code does. Since communications are overhead — necessary, but not directly contributing to revenue — you want the smallest amount possible that gets the job done right. Motivating workers to get a check on an HR form is dangerous, especially with knowledge-intensive workers, as they will reverse-engineer any system to figure out how to do a good job.

My belief is managers can be objective in rating employees without losing a holistic view that is tied to company objectives. While I believe HR can create a framework to capture individual manager’s rankings and ratings on employees and candidates, no one in the organization can successfully resort to using a single set of metrics to effectively rank everyone. Individuals are too individual to be measured in a rigid manner. Being objective takes thought, but when done right, so does every other part of your job. When I hear about a process that removes critical thought, like in the BusinessWeek article, I see trouble. Use objective criteria that make sense, but apply them thoughtfully.



Comments are closed.