I've used a friend's Touch, and own a D2, so it's pretty obvious which way I voted.
The biggest question is whether you are a fan of iTunes or not. I like owning the CDs for all my music. I decided I was happier using the many options available to rip FLAC files from my CDs, and I've been converting everything I own to OGG files. These formats are not available on the Touch, but Apple has lossless available, so it's not a big difference. However, I prefer organizing my music in Windows, using drag-and-drop.
The other major differences for me are that the Touch does a much better job with photos. My D2 will store and display them, but on the Touch the controls are much more cool (zoom and scanning through them). On the other hand, the D2 takes SD cards; 16 Gb cards are here and 32 Gb cards are coming, and the prices are falling...