Hello folks. I went into the past and bought an Archos 605 Wi-Fi 4GB from ebay. Only spent $24.50 on it, to boot. Buttons feel really solid and are shaped differently so you can feel your way around the player. It has microSDHC support, so the limited space isn't a big issue (I will probably carry 20GB of music around at most). It's also a great video player for something with buttons, having a 4.3", 800x480, 24-bit color screen, specifications which are still fairly standard today. This player seemed quite ahead of its time.
The things I don't like are that it only plays MP3 and WAV. You could purchase an optional pack in the past to play AAC, but the store is no longer supported. Audio quality is meh compared to modern players, it seems. I'm going to play around with the EQ and see if I can get it any better.
Thanks for all of the advice given in this thread. My next player will probably be a Sansa, though.