Vinyl setup

I've never owned one, but if i remember correctly, rega turntables have the arm set up at such an height that only rega branded cartridges, which are slighly shorter than average, are a drop-in replacement. If you want to install something else on a P3 you'll have to buy some shims , otherwise the VTA could turn out to be wrong.

Decent carts include the Audio Technica 440Mla which has superb tracking accuracy with fine line stylus, the Ortofon 2M line is pretty good as well, i've tried the red with good results, check out also entry level Grados.

With such pickups you'll just need a Moving Magnet preamp, which are fairly cheap, the Tcc-750 sells for about 50$ and offers great  bang for the buck, if you want to spend a little more there are more flexible units compatible with Moving Coil cartridges, like the Cambridge 640p or the Project PhonoBox II SE. I have the latter and it's pretty solid, compatc, and performs well even with an oldschool low output cart like my Denon dl-103.

As a turntable, take a look at the Project lineup, the Debut or Expression III could be interesting and cost much less than a Rega.

Rega carts are good on Rega arms, but you are not limited to them.  You can adjust the VTA, as well.  If you want aftermarket parts for adjusting VTA, J.A. Michell makes them as well as several other companies.  You'll find that there's an entire cottage industry dedicated to tweaks and other parts for Rega tables.  You can change just about anything you want with a Rega.  One aftermarket mod I think is terrific is the Michell counterweight.  It's a lot easier to deal with than the Rega counterweight.

There are lots and lots of phono stages out there.  How much are you looking to spend?  I started off with a Planar 3 and a NAD PP2.  I thought it sounded pretty good, but sold it when I went to using all vacuum tubes.  One option you might not have considered is buying a preamp that has a phono stage inside.  I found that the least expensive way to get a tubed phono stage was to pick up a Conrad-Johnson PV2 for about $300.  It's entirely tubed and gives you two tape loops out, so you can connect two headphone amps, as well as a power amp for speakers and several sources.  If preamps interest you, take a look at the used ones.  You can often get a good phono stage and a lot more.

Cartridges are all over the map, too.  A couple of inexpensive ones I've tried and enjoyed were the Grado Gold and the Shure M97xE.  Both are affordable and good performers.  But do a lot of reading - each cartridge has its own sound.  This isn't like digital, either.  Each part really makes a big difference.

Finally, if you haven't read the Vinyl Anachronist articles yet, check them out.  You'll learn a lot.

Musical Fidelity V-LPS could be an inexpensive yet worthy phono..