There are some thoughts on this at
Joel on Software that are pretty good. In particular I agree with what he says about people who don't "get" pointers, and identifying those who do.
A lot of Java programmers seem to get by on builder tools alone, and can't really write code freehand. I don't want those people. I would ask lots of "write me a little snippet to do the following" kinds of questions to test their mastery of the language, especially things like exception handling.
I find that a lot of Java programmers, including self-proclaimed senior architects, are very weak on basic computer science. If I am hiring for a decently senior position, I'll send them up to the whiteboard and ask things like this:
- Show me a linked list (if they don't ask what kind, have them show you both single and double)
- Show me a tree
- If I asked you to write code to traverse the tree, what question would you ask me before beginning?
- Show me a directed acyclic graph (if they are a wiseass (which is OK with me) they will just point at the tree; then I'd ask them to alter the tree so it's a DAG but not a tree)
- Show me a generalized graph that is not a DAG.
I also find that a lot of Java programmers are helpless in SQL - they know how to use ORM tools to build database objects for them, but they can't write a query themselves. That's important to me, though it may not be so important to you. So I diagram a mini-database (manufacturer, product, invoice, invoice-line-item) and ask them to write me a query that lists all the manufacturers whose items appeared on invoices last week. Someone with SQL skills can rattle this off, but a surprising number take a very long time or can't do it at all. (Bonus brownie points to those who have the added insight that it probably should be a SELECT DISTINCT query.)
I really want to see a code sample. In my experience, 90% of good programming is good taste, so I want to see that their code is thoughtfully designed, has clearly and intelligently designed interfaces to the other code that it works with, that it's readable and shows some sort of consistency around naming conventions, and so forth. Code that isn't like that is not stuff I like putting into production and having to support.
I like people who can function pretty independently, so near the end I usually ask a question along these lines: "it's your first day, you arrive here and there's an email from me saying I'm out and unreachable and asking you to investigate, right away, the following type of bug that's been reported by the customer. Walk me through what you're going to do next." Poor answers are vague: "I guess I'd start trying to figure out the code." The best answers are very proactive and specific: "well, today I'm interviewing with 3 other people in your group, so by the time I come in I know 3 people here who understand the application, and I probably have an idea which one to start with for this particular problem; I'll talk to her. I'll also ask how the customer complaints come in to our group, so I can go through that channel and find out all the details of exactly what went wrong so I can reproduce it locally. Once I understand the problem and reproduce it, I'll..." and so forth. In my experience people who ace THAT test are rare, and are outstanding contributors.