Source is the most important part, but only insofar as the quality of the recording. Most differences with CD players seem to stem from the output level - different models and makes have different output. Two different CD players at the same volume level on the same amp can sound different because of this. If you level match digital sources, it becomes increasingly difficult to tell them apart. Don't take my word for it, look at the manufacturer's specs - most differences lie beneath the threshhold of hearing.
There are big differences in manufacturing build quality, aesthetics and reliability. If you want something that looks cool, with lots of aluminum and a sufficiently obscure brand to impress friends who mostly shop at Best Buy, you will pay for the privilege. There might be minor differences in sound quality, but I haven't heard any worth dropping a few thousand for. What I'd like to find is a SACD player built for studio/professional/radio station use. The pro gear tends to be extremely reliable, free of major defects, and affordably priced compared to audiophile gear. It also tends to hold value better and isn't subject to whims of audiophile fashion.