I will answer as a consumer, what I would love to see in a portable player:
1) Linux compatibility, and multi-OS compatibility in general . For daily life and for work I use linux, and use Windows for games also, I also know lots of people with mac, windows and diferent flavors of linux, so you can see the mess when something doesn't have cross-platform compatibility, and it's REALY anoying when you try to get support from a company and it says that you should use windows since THEY don't support cross/OS. Sorry for the caps, but it's how I feel for many years now after purchasing many products and having to relly in the (great, btw) linux comunity to do the job the companies didn't want to complete.
2) Multiple browsing and organization options: Diferent people like diferent things. In my case I wan't to have folder structure browsing as a must, but many people like either artist/album or playlist formation. I like particularly my Sansa Fuze 8GB folder structure, it's not perfect but it's simple enough. Also it would be interesting to have a "virtual merge" between the SD card and the internal memory folders, so you shouldn't go all the way up the folder tree just to change between the internal memory and SD card. Of course you should be able to see the memory as it is if you wish so.
3) As people said, support to all possible formats, but well, FLAC is a must, and I think mp3 & ogg also, still if you could make it to support APE and ALAC would be very nice, in the case of the last one, would probably atract lots of people that have macs/ipod right now and pose a possibility for them since they wouldn't have to change from ALAC to flac in their colection. One of the (many) reasons that I don't go to ipod classic is that it doesn't support flac out of the box, and I don't know how that rockbox support for it is going.
4) 16GB / 32GB / 64GB would be great, but if there is 16GB and SD(not micro-sd) card suport would be more than enough. Most consumers would be more than satisfied with 16GB, and some that want more storage can buy up to 128 GB of extra storage, and the cost stays with the consumer, and as so maybe it's possible to cut some dollars off if the player get too expensive.
5) Support for 24 bits/ 44.1 48 88.2 96 kHz files natively, I don't think that's necessary to go all the way up to 24 bits/192 kHz for a portable player, but this is my personal opinion. As a consequence choose a good DAC for them.
6) About conectors:
If you aim squarely as a memory in -> headphone out player, just focus on a good amp but I think that such a player could be more than just that.
To compete with the Ipod, I think that it should be able to use either an digital output or an usb output in order to have the music streamed out digitaly, and so use something like an Algorithm Solo or other portable DAC.
Besides that, I think it should have a line-out to use portable/fullsized amps or stereo systems without too much of a fuss, and maybe an digital-in to be used as an portable dac/amp. It can get bulky with this kind of conectors. If you could make multiple versions, I would make a consumer-focused version without this kind of conectors, and also with micro-sd card suport intead of SD card suport, and a higher-end version with them. If you can make it no bigger than Ibasso DX100, that would be enough.
Last but not least, it should use standard usb conectors, no proprietary conectors please. I don't know why use proprietary conectors, but I do know that you end up needing to use multiple conectors and adapters on the long run, and I personaly don't like it.
7) It's going to be feeding mainly Iems and portable headphones, so I think there is no need for something that can power up to 600 ohms cans. If you can get it to power up to 200-300 ohms max, would more than enough in my view. If you wan't to power something like a planar-magnetic or a 600 ohms can, I think it would be wiser to have a line out and a external dedicated amp.
8) Battery: If you can manage to have ~ 20 h with 16/44.1 flac music using a ~ 30 ohms iem, it would be a bliss!!
Idealy, something like 8-10 h of 24/96 flac, and 24 h of 16/44.1 flac with an portable can or with an iem in decent(not earblowing) volumes would be the perfect battery time in my opinion.
Also the basic auto-switch off after idle time is mandatory, perferably if configurable. Also it would be nice if you could choose if you want to charge or not the player when it's pluged in the computer. If you just want to get music in, you don't need to get it charged, principaly if the battery if already near full.
Edit: Also, if you could make the battery user-exchangable would be awesome.
9) Touch-Screen vs buttoms/wheels and physical UI :
Personaly I haven't had a good time with touch screens, usualy my fingers are too big for most of them so when I have to type something it sometimes get bothersome. Yesterday I saw the Samsung Galaxy S wifi interface and I think I'll be giving it a closer look soon, it seemed the best UI from touch-screen players that I tried, far better than the Ipod touch UI, cleaner.
If possible, try making a lower end with buttoms and a higher end with touchscreen. I like the sansa fuze buttoms and wheel, but sometimes I would like to have a nicer UI, it seems a bit old/limited interface from time to time.
Also a holder/locking buttom is a very good
10) A fine grained volume control is important if you and to get from 16 ohms iems to 300 ohms cans. If you don't want to get it there, you don't need a too fine grained, but still, I think something between 64/128 steps would be very very good, or even maybe 256 steps.
11) I don't think that most consumers would care about tweaking their player, but for a realy high end player, which I don't think it's the case but it would be If you would be aiming to compete with ibasso dx100, would be very good.
12) wifi Internet, games and other capabilities:
Many consumers love this things now a days, but if you think it you have to sacrifice some audio-quality to have it, I say forget it, make one good audio-player and you are good to go. Also, this things eat battery like nuts, so be carefull if you wish to include them.
13) Price:
I know it's dificult, but if you could make it near the ~ US$ 400, it would face directly the Ipod touch and I think it would be near unbeatable in the price-range. If you have to choose, make a lower end player near the US$ 300 and a higher-end player near the US$ 600-800, but make the later a real threat to the DX100, if not in SQ, in usability and other features.
Well, good luck with your market research, and, if I may add, once you make your player, if has most if not all the above, I will seriously consider buying it.