So if you don't want a DAC just 1/8'' jack out of the pc will carry your music out via a Y cable with analogue on the other end into your headphone amp. It will work, sound will suffer though as compared to using a DAC. Technically you are still using a DAC, as the music resides on the PC digitally, you are also using the PC to convert that into an analogue signal to send out to your amp.
The PC actually has a bona fide DAC in it, but there's this huge disjoint between Digital To Analog Converters as a class of commodity ICs, and fancy aluminum boxed magic components that cost thousands of dollars and are built around those COTS parts, at least in the world of audio.
It isn't a Y cable either (that's a splitter) - you'd want a TRS mini (3.5mm or 1/8") to RCA stereo adapter, this may exist as a cable.
Well yeah, I'm sorry. It isn't going to sound terrible by any means, but within the context of an external DAC there is a lot of improvement to be had.
I think "lot" should be qualified here. A lot of people expect to hear this life-altering night and day difference between a PC soundcard and mega-buck external devices, and the reality is, that different just isn't there.
The biggest potential issue for onboard/integrated audio is grounding faults which will produce hum/buzz/etc on the output. A discrete soundcard will get you around this, as long as your power supply doesn't ripple like a hurricane (and if this is a cheaply built OEM machine, especially a white-box, it almost certainly does - cheap power supplies are like a plague).
Whats colinhardings method? Also what are some inexpensive USB-only DAC's?
See jazzerdave's reply. In addition to what he said, I'd echo that if you are using a desktop PC, get a soundcard instead - you'll save $100. At least. You should also look at the USB based audio interfaces from Creative and others, likely will save money too (spending a fortune for features limited "audiophile" parts has never made sense to me).
Also if your PC has a digital audio output (a lot of newer machines do), you could hook up to a S/PDIF DtoA. This assumes that you're having issues with the onboard output; if you just want to spend some money, there's a lot of better things you could spend it on (like new cans, new music, etc).
You can take a 1/8" to RCA cable and plug it straight from the audio out on your pc to the RCA input on the Valhalla. The HRT Music Streamer II DAC is ~ $160 (available at Audio Advisor), and the stand-alone ODAC (from JDS labs) is ~ $160 with the RCA output option. There are many more options on the market, but that's a start. If your PC is a desktop, you could always get a new soundcard like the Asus Xonar Essence ST.
This.
It's an S5 Electronics K12G. 8wpc. 
