Yes, driving these impedances is about having enough voltage. You only ever want to match impedances if you're working with transmission lines.
This all comes down to electrical models. Basically, your preamp is getting simplified for circuit analysis purposes. We're using a model where it (the preamp) is just a voltage source (that's where your audio signal comes from) and a series impedance (that's the output impedance). Then, the input of your amplifier is getting simplified just to an impedance and ground, so when you connect the two, you have the voltage source->output impedance->input impedance->ground all in series.
A circuit like this is called a voltage divider, and the voltage starts high at the source (even if the highest is only 5V or so), then drops down to zero (ground is zero). The drop across each impedance is proportional to the resistance. Ideally, you want all the voltage drop to happen across your amp's input impedance, but since the world is not ideal, you just want the input impedance to be significantly higher than your output impedance. That ensures good transfer (efficiency) between the amp and preamp.
Also, this same principle applies to the output of the amp and your speakers. There's also some concern about electrical dampening, but you still want good power transfer to your driver.
This Wiki article actually might show it better. You want a big Vout, so Z1 should be much smaller than Z2. Z1 is your preamp's output impedance and Z2 is your amp's input impedance. Also, the cricuit model is called a
Thevenin Equivalent if you'd like to read up on it.