Lots and lots of places. The usual suspects like Mouser, Digikey, Allied, Handmade Audio, Redco, Antique Electronic Supply, Michael Percy Audio, Radio Daze, and probably a few others I've forgotten. I'd post links, but am on the iPhone right now; a quick Google should get you to these.
For transformers, I like Electra-Print, Lundahl (search for K&K Audio), Hammond (widely available), One Electron, Plitron, and Sowter.
However, the best source for parts is in your back yard. Comb the phone book for local electronics and industrial/electronics surplus stores and pay them a visit. You can find oddball industrial leftovers with great transformers inside for much less than a new one. Sometimes they will have grab bags of assorted components. And you can repurpose old cabinets for new projects. Cabinets and transformers are the most expensive parts usually, so if you save on those and get the rest from Mouser you'll save a bundle and get cool cabinets no one else has. Also, get to know the shop owner and tell him what you like or want. You'll start getting phone calls when those come in.
Another great source is the Hamfest. Check with the ARRL site and see when they are in your area. You'll find lots of tubes and lots of bargains. These are also a perfect place to ask people where they find stuff. Oh, and there will be a mess of test equipment, too.
Finally, scour eBay. I got some nifty paper-in-oil caps the other day and have picked off some power transformers on the cheap.