If you think about a home system, there is a source component, a preamp, an amplifier, and speakers. The source component pushes an analog current to the preamplifier at a set voltage (dependent on the component); the preamp varies that voltage (either by attenuation or amplification) to make the sound level listenable; then the amplifier takes that voltage level and adds tons of current to drive the speakers. The voltage exists from the source, but if you hook the speakers up to the source they will draw far more current than the source was designed to handle.
Headphones typically have a much higher resistance than speakers, and so require less current, so there's less of a chance of hurting the source by experimenting. Also, since we're all talking about 1/8th inch connections, I imagine that portable sources with line-outs are designed to not blow up when connected to a headphone rather than an amplifier, whereas a home component engineer may safely assume nobody will plug their bananna plugs into an RCA jack. Not so many worries here as with a home system.
Still, the bottom line is that every component is pushing out analog data, which by its analogous nature will produce sound when connected to a speaker. (Unless, of course, you're using a digital output. Then this only applies to the DAC.) But the source/preamp won't put out the kind of current a speaker was designed to produce sound with, and so won't sound as good as an amplifier that was.
Does that help any?