I just got into vinyl and finally have my rig set up took me about 2 months.
The major cost is the turntable and cartridge. I'd would recommended stalking craigslist and buy locally. I took a chance and purchased my TT over ebay, it came pretty banged up and missing the cartridge. Seller never responded to my emails.
You can probably score a decent vintage turntable and cart for under $300, I've heard stories of people picking up nice late 70's Japanese direct drive units for pocket change. The phono pre-amp you can get new from http://www.phonopreamps.com/
for around $70 or else go used.
So far my rig is a Sony PS-X7 TT from the early 80's
Ortofon 2M red cart
Vintage Marantz 2265B receiver circa late 1970's (I've always wanted one and it's has a pretty nice sounding phono amp)
Swan M1.2 bookshelf speakers (awesome deal over audiogon)
Small Linux computer running SqueezeSlave connected via USB to a DIY AMB Gamma2 for streaming FLAC's.
I'm still undecided about vinyl. It is a bit of a pain cleaning used records and new records can be pricey.
The first new record I bought was a new copy of Depeche Mode's Violator, and I have to say it sounds incredible as does my near mint copy of Steely Dan's Aja. My thrift store and other used LP's vary, some sound good while some are noisey. I don't mind the occasional snap, crackle and pop over speakers but they are distracting over headphones.