I think it comes down to 2 questions:
1. how much difference does it make to you personally
2. how much money are you able and willing to spend on that difference
Reg 1, would be best if you could listen to some friend's equipment, maybe best a price class above what you might be willing to spend.
The more and the longer you can listen, the better.
For example, I have the Cayin N3 ($130), the HiBy R3 ($230) and the Chord Mojo DAC ($500).
The N3 is a nice, natural player for the money, nice upgrade to phone, the R3 sounds bold and strong at first, but for me it's a bit "too much" and my ears get a little tired after a short while. The Mojo in comparison is just so smooth, open and flowing, maybe less "spectacular" at first but I can listen for hours on end.
With a budget of $100, I would spend all on headphones/earphones, and maybe later add a small DAP or DAC/AMP.