I figured I start with the experts. If anyone uses a portable amp (like a total airhead) and an MP3 player (I have the Rio Kharma) and has a carry bag solution that can be used while walking about town, please let me know. I will be using the bag for exercise walking as well so I would like to find something that does not bounce around alot.
I use SFBags iPod Gear Pouch to carry my SR-71 and iPod and some other things (e.g., additional earbuds, PDA). I hold the pouch in my hand as I walk around.
I would think you should be able to fit a portable amp and player in a conventional fannypack. If you're concerned about the equipment jostling around in the bag, people have used rubberbands or velcro straps to tie the amp and player together.
I've also read that some people use a small camera bag (the ones for compact cameras) to carry their gear. But most likely, such a bag utilizes a belt loop if not a shoulder strap.
A regular fannypack is probably your most convenient and cost-effective option for exercise purposes.