Well, search through file system is faster in exFAT but for player this advantage is optional. Yet, there is another aspect - Windows and many third party formatting tools refuse to format big cards with FAT32. So player itself was the only option to format my 128GB card. No big deal but I don`t like to be dictated. Also player`s implementation of USB is very slow and I prefer work with cards in good USB adapter. And so before copying the files to a new card I have to insert it to player to format then insert to adapter to copy and then back to player - instead of copying in adapter (as card is factory formatted to exFAT) and inserting to player.
Almost every piece of gear I've purchased (never had a smart phone that uses external cards though) has recommended to format the card it uses with the device, including camera gear and the like. This is nothing out of the ordinary and certainly not a bug. I'd call it a feature as it has alleviated many format issues in the past. The majority of playback issues I've read since they've first released the X3 a couple years ago were form card format issues before they offered the built in format tool. There are many ways to format a card that make it unstable for the player. Count your blessings it's available.
I understand it's not optimal to remove the card but once it's loaded with the majority of music the first time it really doesn't take that long to add an album or two once in a while. I get it, it's inconvenient, but no where near a bug.