Looks like I was a little too optimistic about that buzz/hum, I checked all my soldering, made sure nothing was shorting, checked and rechecked voltages, made sure everything was properly grounded, etc etc.
The buzz/hum is only in the the right ear and is independent of the volume, if I put my headphones on and turn the amp on then as soon as it warms up the buzz shows up, audio plays clearly along with the buzz. Also the buzz never gets louder as the volume is turned up either, likewise for all the way down. It is loud enough that I can't ignore it, it imposes on music.
There is a second, more normal volume related hum as the volume gets louder I begin to hear, it goes away if I touch the inner enclosure where there is no paint, is there a more permanent solution to that? The case itself is a ground according to my multimeter.
I'm quite proud of how it's turned out, regardless of the problems. If I could get rid of that buzz then I'm home free. The sound on this thing is pretty amazing though, I wasn't sure what to expect, but it's more than just "louder" music, a simple MP3 took on a whole new life.
I was going to attach pictures but I keep getting an error message, I'll try again later.