I've had all these same thoughts, I love hand tools. There's no substitute for making lots of holes lots of ways, hard to put into words what works and what doesn't, especially when we find a way to make whatever tool is at hand do the job.
Nevertheless, there's nothing like drilling a well-clamped object in a decent drill press.
I once made a 4' horizontal hand-operated drill press, for drilling pen blanks from tricky hardwoods. Ok, I'm a bit nuts; I think I'm in good company here. Maintaining rigidity as one works is the important variable, not hand versus power tools. Think impedance?
The hand drill I have comes apart, one could attach it to a very long dowel, and make a very rigid drill press from it. Then one could probably do anything with it, given enough patience and time...