I love Joy Division. I followed the opposite path of you, first discovering Joy Division, actually through the movie 24 Hour Party People (which was an OK flick, but I am indebted to it for introducing me to Joy Division) and then moving on to the awesomeness of New Order. The transition the band made after Curtis's death is one of the most remarkable in the history of rock, IMO. Both bands are amazing, and both are absolutely seminal, for completely different reasons. For most fans, the
Heart & Soul Boxed Set is all you'll ever need. It comes with both their studio albums, a lot of their early stuff under their original name, Warsaw, as well as a live disc and a disc full of b-sides, demos, and other knick-knacks. The book inside is cool, though the box itself is a bit flimsy. Still, well worth the purchase!
I love it when I "discover" a really incredible band that maybe I'd heard of but never really listened to, i.e. I just started listening to Frank Zappa. Woah! It is a great experience, and one I envy of you.
Unknown Pleasures is really amazing to listen to when you consider the guy who produced it was so crazy, he even had separate tracks for each
different drum. So every drum was painstakingly recorded on a different track. This extreme instrument separation makes for the cold, distant feel of the entire record. BTW, make sure you check out the tune "Transmission" off of Unknown Pleasures (dance dance dance dance dance to the radio!!)
Generally my favorite Joy Division tracks are Disorder, Day Of The Lords, Atrocity Exhibition, Transmission, Isolation. The synth on Isolation is so freaking cool.
Enjoy!