I definitely think XO is the best place to start, but that doesn't mean I think it's his definitive album. I think Abbey Road is the best place to start with the Beatles, but I don't think it's their definitive album.
XO is very much the album which introduced him into the mainstream, not that it's a mainstream album.......but it has a mix of the early sound he was known for (the lo-fi acoustic) and the heavy production of Figure 8.....
I'm of the opinion that XO, Either / Or and Figure 8 are all equally great. And From a Basement is not far behind those 3. His self-titled is very very good, but I listen to it less than the 4 aforementioned. Roman Candle I own, but have not given it that much attention. New Moon is a superb companion to his work but I don't view as an album.
You will have a genuine idea as to whether you love Elliott Smith by purchasing Either Or / XO / and Figure 8..........all three albums are about as different as Rubber Soul / Revolver / Sgt Pepper.............and From a Basement was going to be his White Album.
Take a listen.
I've been talking headphones with you for years. Now I can help you with your purchase:)
Sales Specialist & Headphone Guru @ Headphones.com