On-line interviewee testing

published: Tue, 28-Jun-2005   |   updated: Thu, 27-Oct-2005

At work, we have several openings for talented C# and C++ developers, with various areas of expertise (ASP.NET, multi-tier architectures, web services, compact framework, etc).

Of course, as with any company that advertises job openings, we get a lot of résumés and part of the job is to remove the wheat from the chaff. HR acknowledges that reviewing résumés takes time, doing the initial phone interview takes even longer, and the full three hour in-person interview takes, well, three hours.

So, they're looking into the usefulness of on-line testing services to wean out the coding horrors from the gurus. Now, I'm not a great fan of this way of judging a developer based on his answers to yes/no type questions, or his ability to retain arcane knowledge. I'd rather guide the interview towards discussions of problem solving; how the interviewee would approach class design; investigate how the interviewee responds to questions that don't have simple, or even right, answers. (I'm reminded in this endeavor of talking to our nieces on the phone: they're at the age where, if you don't cast your conversation in a certain way, they'll reply in monosyllables: yes, no, fine, yesterday, at the vet's, etc.)

Since I'm known around here as knowing a little about C#, they asked me to take one of the tests from one of these Internet testing sites to see what it was like and what kind of results they'd get from a known quantity (that would be me). I fired up Firefox and started the test.

First, the good news: I was rated as a "total match" for Configuresoft. W00t! Everyone heaves a sigh of relief, etc, and I get to keep my cube, chair and PC, and am not escorted off the premises.

Second, well, not exactly bad news, but somewhat bizarre news. I was caught off-guard a little because the test wasn't just about C#. It also had sections with personality-type questions, abstract thinking questions, math questions, questions on the proper use of English, and, horror, questions on SQL.

I really dislike personality or behavioral questions of the form "I'm not above using other people to get my way if I believe I am right" with a 5/6 point sliding scale from "Totally Agree" to "Totally Disagree". I really hate them. Why? Because the question seems to always be phrased in a black/white manner but I can see situations where it might apply and others when it won't. To me this particular question evokes "the ends justify the means" paradox.

To be honest, I do not answer these types of questions truthfully in all cases. "Mmm, what would the average employer prefer to hear here? Let's answer accordingly."

A huge laugh this: I was marked down on perfectionism partly because “Julian Bucknall describes himself as […] seldom expecting to be 100% accurate”. Correct, damn right. I never expect to be 100% accurate: I err, I’m human. So this presumably means that I don’t consider myself to be a perfectionist. Which is worse: a perfectionist (who couldn’t/wouldn’t admit fault) or a pragmatist (who not only admits fault but knows what to do about it, and still aspires to perfectionism)? Gaak. It makes you want to argue about the meaning of the various questions when applied to computer programming.

One of the things the personality questions didn't investigate is my sometimes penchant for scanning big blocks of text and missing something. So, most of the sections had a big page before the multiple-choice questions describing the section, what would happen, etc. The first one of these I skipped reading entirely, anxious to get into the math questions. The first question (of 11) showed up on the screen, with a timer counting down from 180 seconds. I looked at it: just 180 seconds to answer all questions in the section? Geez! I was really under the gun. The first question was a piece of cake: 10 seconds with my calculator and I was done. The next question was one of those awful logic questions. Shit. I read the text, and glanced at the clock: 165 seconds left. I thought about it, drew some Venn diagrams on a piece of paper, was getting nowhere. 125 seconds left. I guessed the answer, just to get off this question and onto the next. A simpler calculation type question, another 10 seconds with the calculator. 165 seconds left.

Huh? 165? I then realized that I had 180 seconds per question. Boy, did I feel like an idiot. Unfortunately, the test did not allow you to go back to questions you;d already answered. So, I got one question wrong in that section ("Julian Bucknall scored 91%, which was higher than 88% of previous test takers").

After the English grammar and usage section, finally, up came the C# section. Hah. Here we go. Multiple choice again, and 180 seconds per question. We're off! The first question read, and I kid you not, "Which of the following categories of variables are classified as initially unassigned?" and then gave a list of five, Static variables, Instance variables of class instances, Local variables, Array elements, and Value parameters. WTF? I'm going to let you ponder this for a couple of seconds to see whether you know this off the top of your head (I freely admit I didn't).

Are you back? Yep, it was Local variables, of course. Ha! Section 5.3 of The C# Programming Language by Hejlsberg et al discusses this at depth (and given the terminology used in the question, it was copied directly from the book), and talks about what the compiler has to do. All fascinating stuff to be sure, but I've managed to get through 3 plus years of C# without having visited this particular backwater.

The majority of the rest of the C# questions were, in my view, horribly edge case. They required some very subtle reading and knowledge of the C# language specification. These were not questions about C# that people would see in 95% or even 98% of their programming. Some of them were very poorly written to boot (I just didn't understand some of the questions on the use of asynchronous invocation -- it used terminology I wasn't familiar with; oh, and, yes, four out of 12 questions were about asynchronous invocation: it's a terribly important major-use-case use of C# and we must make sure that everyone we test knows about it). The display of the code for each question was horrible as well (no indentation): sometimes I had to copy the code into the IDE to be able to better visualize it.

I remember one question involving some code that no one would ever write, period, since it involved a (quite frankly) perverse way of returning a single item of data from a method and then asking a question about behavior that would never arise ‘cos you’d never write the method like that. Are you ready? Here's the method signature:

void A(out int x) {...}

Yep, I guessed it: you've never written an abomination like that, have you? Oh, and sorry you were reading this post while drinking coffee. I'm sure it'll wipe right off.

Anyway, I scored 75% on this section, believe it or not. Wow, one question wrong out of every four? Who is this guy? Thinks he knows C#, eh? We'd better not hire him. But get this: I scored higher than 96% of previous test takers. Yep, according to this test I'm in the top 4% of C# developers, old chum. Take that.

Next section was "Abstract thinking". Ooh, boy. Lots of questions about spatial reasoning here. In fact, I was commended in the final report for my spatial reasoning skills. It's sure going to come in handy in a programming career.

Then the biggie: the section where I had to write some C# code. As usual I skipped the intro, anxious to get into the question. Now, you're not going to believe this, but the page came up with the usual timer area, but it was blank. I looked at this, and it didn't change. It didn't show any counting down. OK, I thought, I have as much time as I like to do this question. I read the question. A pretty easy assignment (no, I won't repeat the question in case you're going to do the same test). It was a string parsing question essentially. I thought of two ways of doing it (using regex, or String.Split()), but vanity took over: could I devise a way to do it that was more unusual and perhaps elegant, full of flair and originality.

I hacked around a bit in VS2005, trying various ideas and came up with a neato recursive solution. I wrote it, tested it, fixed the bug, and the copied the code and pasted... Uh, the test had advanced to the next page with a "you ran out of time" message. WTF? I couldn't go back. Bastards.

So I flunked the "write some C# code" section. Bloody hell.

Right after that was the "test your knowledge of SQL" section. I tried puzzling out the first few questions (the questions were posed as, here are some tables with example data, here is a SQL statement, what are the resulting rows? Or what is significant about the data that is returned?), but after a while I got bored with it and basically tried to answer each question by considering the SQL and the answers from a "higher-level". OK, I just eliminated the obviously wrong answers and guessed from the rest.

Score: 75%, which was better than 94% of previous test takers. Ye gods, not only am I in the top 4% of C# developers, I'm in the top 6% of SQL developers too! W00t! Where's the red carpet to my cube? Where's the personal valet to bring me drinks? Watch out world, here I come.

Well, to summarize, I didn't think this particular test was all that relevant. The testing process does produce a detailed report (I've been quoting from it, but normally the interviewee doesn't get to see it) with some interesting personality/behavior categorizations and some example questions to explore this traits further in a normal interview. To repeat, though, I did find the C# questions very edge-case and not that relevant to the vast majority of C# programming, and there just wasn't enough of them.

We have a couple more free tests, so we're going to get a couple of other people to test themselves (one on C++ development and one on SQL development) and see what they feel about it. I'll let you know.