Posts Tagged ‘interview’

Rating Programmers

Saturday, March 14th, 2009

The Programmer Competency Matrix is an insightful tool to aid in objectively rating programmers in terms of ability when you have no history of results, like in an interview.

In computer science, however, they are missing Boolean algebra and discrete mathematics which are prerequisites for successfully working with data structures and algorithms. Also, in Software Engineering, I’d like to see levels of ability in risk management. Those two nits aside, this matrix is a handy tool to objectively rate interviewees or ranking software engineers by ability within an organization (e.g. associate, senior, principal, architect, fellow, etc.). Too often people are rated subjectively on the wrong criteria, for example: easy to work with rather than the results they produce and the abilities they possess. A tool like this can help you get over objective rating.

In my experience, titles do a have a function; they communicate to people outside your organization what you do and the level in which you contribute. Markup to the titles like senior in senior software developer, has little value. The meaning of associate, senior or architect is so subjective, they tell you nothing. When I look at a business card and I see the title senior software developer, I ignore the prefix (e.g. senior) and only assume they work with software. This is a side-effect of pervasive subjective ratings and a lack of tools to objectively measure ability.

When selecting a method to rate or rank, there’s a temptation to make it so clear, you don’t have to think. The idea is the perfect solution will arrive at the same conclusion every time. However, the best solution frequently arrives at the right conclusion much of the time. Implementing a black-or-white rating scheme invites people to “game it,” or put another way, algorithmic rating invites sub-optimization. A great example of inviting sub-optimization is in a BusinessWeek article, “Data Mining Moves to Human Resources” (via Slahsdot). The articles describes how Cataphora uses algorithms to rate employees, using behavior and communications. Though Cataphora provides objective criteria, it’s rigidly applied. Today, an algorithm can’t holistically measure effectiveness of two different programmers in a meaningful way. Once people know the metrics, and the metrics will be figured out, people will sub-optimize their work, prioritizing conformance to the metrics over achieving business goals.

Imagine if programmers were rated on clarity and volume of their communications. Suddenly you would have programmers spending significant time writing email and documentation. While that’s not entirely bad, email and documentation don’t directly contribute to revenue as code does. Since communications are overhead — necessary, but not directly contributing to revenue — you want the smallest amount possible that gets the job done right. Motivating workers to get a check on an HR form is dangerous, especially with knowledge-intensive workers, as they will reverse-engineer any system to figure out how to do a good job.

My belief is managers can be objective in rating employees without losing a holistic view that is tied to company objectives. While I believe HR can create a framework to capture individual manager’s rankings and ratings on employees and candidates, no one in the organization can successfully resort to using a single set of metrics to effectively rank everyone. Individuals are too individual to be measured in a rigid manner. Being objective takes thought, but when done right, so does every other part of your job. When I hear about a process that removes critical thought, like in the BusinessWeek article, I see trouble. Use objective criteria that make sense, but apply them thoughtfully.


Interviewing Programmers

Sunday, January 25th, 2009

After my former CEO said, “We’re shutting the company down,” I had to interview. Which is not much fun when the interviewers aren’t good at it. Bad interviewers, about half of those whom I met, did one or more of the following mistakes:

  • Assumed candidates are perfect fits, until proven otherwise
  • Relied on random quiz questions to establish technical competence
  • Looked for a grocery list of skills rather than intelligence or ability
  • Forgot the candidate is also interviewing them

Start at 0 and Work Up

In most interviews, I walk in the door as a “100%-fit” in the eyes of the interviewer and lost percentage points for every question I missed. For knowledge-intensive workers, it’s the wrong approach. At one interview, I aced it after correctly answering all their questions. The job wasn’t a good fit for me, but because I had the right answers, I was a fit. However, the job required skills I didn’t have — they just hadn’t found a reason to not hire me. Failing to find reasons to not hire is not the same, or as good as, finding reasons to hire. Find reasons to hire candidates rather than assuming they will work until they miss a question. This removes luck – some can sail through interviews and still not be a match.

Quiz Questions

The technical interview usually unfolds as a quiz of technical questions, that are often obscure. The logic is, if the candidate knows the obscure parts of a technology, they must know the common parts. At one interview, I aced the quiz and managed to get bonus points by answering the question, “How do you share file descriptors between Unix processes?” Coincidentally, the week before, I read the solution in W. Richard Stevens‘s Advanced Programming in the Unix Environment. They over-rated my ability based on trivia questions I happened to get right. Joel Spolsky wrote a significant article for hiring technical workers, The Guerrilla Guide to Interviewing and in it, said the following about interviewers employing quiz questions:

This is the kind of person who thinks that smart means “knows a lot of facts.” […] There is no possible, imaginable correlation between people that know that particular piece of trivia and people that you want to hire.

He’s right. However, candidates need to enter with a minimum of knowledge. Phone screens are good for determining if the requisite knowledge is present. For C/C++ programmers, I ask basic knowledge-based questions on fundamental aspects of the language. What is the stack and the heap? What is the time complexity for an algorithm that traverses a binary-tree? Explain inheritance and polymorphism. This is not obscure trivia, or a quiz, its a sampling of the basics you need to know to be effective in C++. That and some descriptions of how they contributed in recent projects will fill out phone screens nicely.

Match Ability Before Skills

Looking for specific skills or experiences can exclude great candidates. Sometimes, you can spot a match for another group or, if you have open mind, you can spot abilities that can overcome skill gaps. Imagine your open position requires a C++ network programmer, though the candidate knows networking programming, it’s with Python and Perl. The person demonstrates amazing knowledge of algorithms, fantastic problem-solving skills and a great attitude that would fit in with your team. Moreover, the candidate is self-taught in both Python and Perl, becoming adept in each in weeks. Given a choice between someone with high intelligence and gaps in skill and another with average intelligence and no gaps, with all else being equal, I’ll select the more intelligent candidate.

Forgot the Interview Goes Both Ways

In one interview, I was made to wait 30 minutes as the hiring manager “had an important meeting.” It’s hard not to read into the manager’s priorities. Worse, when my prospective manager showed up, it was clear he had not read my resume. When he became bored with my answers, he would interrupt to start with the next question. An interview is like a first date where everyone is on their best behavior. If the prospective employer is rude during the interview, it’s hard for candidates to imagine the workplace values workers and their contributions. Maybe the manager had a legitimate emergency and was “off their game.” The best thing to do would have been to apologize and re-schedule the interview. Though that’s also poor, it is better than overwhelming potential employees with unprofessional behavior. Even if the interviewees are poor fits, it’s best to treat them with the same respect as everyone else because it’s the right thing to do.


  1. Screen the prospect, ensure they have the minimum knowledge, skill set and experience.
  2. Figure out what the candidate can do.
  3. Avoid quiz questions, focus on relevant problem-solving.
  4. Don’t just match positions with skills, look at the candidate’s abilities.
  5. Remember, the candidate in interviewing you, too.

Favorite Technology Trap

Saturday, January 24th, 2009

Many programmers have a favorite language — usually the one they know best. To me, it’s a fault if the developer sees the language as a hammer and every problem a nail. If hiring a plumber I’d ask, “What is your favorite tool?” The winning answer would be, “Whichever tool suits best suits the problem in front of me.” Experience with a technology should have no bearing on a technical solution, but it often does.

One likable thing about interviewing is meeting others passionate about technology. Recently, after my last company closed, I interviewed. Small, early-stage startups seem to be the most chaotic in technology choices. At early stages, it’s a small group of developers with only one goal: prove the idea works. When I see a baffling choice, like a high-level scripting language, doing CPU-intensive tasks, I ask, “Why?” The more defensive the answer, the more frightened I gets.

Favorite technologies create selective vision for the afflicted. Problems in other technologies are immediately obvious while the favored tool remains The Chosen Answer. Everyone who disagrees is a heretic. Additionally, when you have a favorite technology, and that technology hits its limit, it is somehow acceptable to change the problem.

“Why did you use Java to stream multi-gigabyte files to create SHA-1 hashes?”

“Java has excellent support for databases and creating hash keys.”

“The SHA-1 key generation could not keep up with the stream, missed bytes, and produced the wrong hash key.”

“You’re right – we need to figure out another way to create the hash keys.”

That’s the problem with a favorite tool, you rationalize your way into using it, even when it’s not appropriate. Why would smart people do this?

Their favorite technology is where many invest significant time. Most people want to be in, not out, of their comfort zone. Sticking with a well-known tool is a way to stay comfortable. This is a huge issue with change management. Getting people to learn new skills often meets huge resistance as they fear devaluation while they come up to speed with the new technology.

A way to avoid this trap is to hire engineers who like what they can do with technologies more than any one technology. Ask open-ended questions in interviews, e.g. what languages or platforms they like or don’t like or what kinds of problems they like to solve. Don’t lead the candidate or broadcast your intent, just listen. If you take some time to get to know the candidate rather than asking obscure technology quiz questions, you may be able to figure out if they are already in the favorite technology trap.