The X3 can hit hearing damage levels on Grados without distorting. It's not going to be a lot of difference as Grados have very high sensitivity as well as low impedance. You'll be at the impedance range where most amps produce the most power, which they won't even need. There will be some improvements as even DAPs might not have the capacitance that good amps (including portables) have but don't expect anything drastic.
The Vali2 might be something to consider over the Magni though as it has enough tube coloration to smooth out a few rough edges. And while it doesn't have the Magni's monster power output it will still be enough for most other headphones you might upgrade to.
As for the Modi, you already have the X3 - you can use that as a DAC with a PC. Or you can get the DK1 dock/stand so you can get a line out signal out of the USB port on the bottom (it has some pins where the signal can be rerouted, kind of like the iPod except it has a dedicated 3.5mm line out), so any Fiio DAP can be propped up with cables running behind the DK1 and the desktop amplifier.