Let's see how well I can handle this:
You'll need a turntable, first off. That's really just a bunch of heavy parts (plinth), a motor, and a heavy circle that you place records on (platter). Attached to that turntable should be a tonearm, and attached to that tonearm should be a cartridge.
There are many varieties of cartridges in the world, but the only one that matters to a beginner is MM, or moving magnet. These have the highest output, and are generally the least expensive.
You turntable will likely have RCA cables attached to it, but don't be fooled. You can't simply plug it in to any old thing. In order to make records a reasonable size, they are EQ'ed very strangely. You need a phono preamp to restore the EQ to something listenable, to provide the right resistance for your cartridge, and to give you an output level that you can actually hear. Some oft-recommended budget options are the Bellari VP129, and the Cambridge 640p. Some phono stages have a headphone amp, some do not. Lots of older integrated amps and many good newer integrated amps have a phono stage built in. If you have one like that, you don't need a phono preamp.
The two most common options recommended for beginners are the Technics SL1200 (usually modified by KAB) and the Rega P2, P3, or P3-24. What is most important for beginners is that you buy locally, and preferably from someone who can set everything up for you. Setup is daunting, and uses a whole host of tools you do not yet own. It all will get easier, but vinyl is a crazy beast right at first.