I use mostly FLACs from CDs. To go down from 1440kbps to 320kbps would lose information. And, I have a half-dozen 24/96 albums. I've seen a lot of debate about when data compression is audible. I don't think 320kbps loses much if anything. I'm just not willing to take the chance.
I find 320kbps to be perfectly fine. I have some CDs ripped to flac but most is mp3 and I really can't hear the difference. Maybe it's my equipment or maybe it's my hearing but I have Jackson Browne's "Running On Empty" from HDTracks and I can't tell the difference between that and the CD I have ripped to 320 mp3. Maybe if I had much more expensive headphones I might detect a difference but I'm happy with the sound I get from the X3 because it was such a huge upgrade over my old ipods. It was like that first blast of daylight when you leave a movie theatre on a sun-filled day.