+1 on running the Magni out of E10, etc. It works fine running out of my E10 and I'm still debating what DAC to get. The 770s sound great out of the Magni as do most of my headphones.
I originally delayed the Modi purchase because I wanted to buy a tube amp to 'smooth out' the DT990's & Superlux 681s a little instead. That tube amp was a very good purchase for me.
That said, I haven't felt the pressing need to upgrade the DAC. In fact, delaying the Modi / other DAC purchase has allowed me to pick up a couple really good headphone/iem deals.
Not to say that Fiio E10 or the Schiit Modi are bad amps -- much like yourself, I never felt a pressing need to upgrade from my Modi -- but when I picked up a Lyr + Bifrost and did some A-and-B one variable at a time back-to-back comparitive listening, I was blown away by the extra levels of details and revelations, clarity, transparency, separation of instruments, and super airy sound stage of the Bifrost.
The comparisons that really stood out for me were the following:
1) Modi + Magni versus Bifrost + Magni
2) Modi + Lyr versus Bifrost + Lyr
In both cases, qualities that I mentioned earlier, namely detail, revelation, clarity, transparency, separation of instruments, and sound stage, were all noticeably better with the Bifrost. When matched with the Lyr, in particular, it because painfully obvious how much the DAC mattered -- the Modi was really holding Lyr back, and the Bifrost simply allowed the Lyr to soar in all kinds of acoustic manners.
Again, I've spent a lot of time with my Modi, and it is definitely a great amp. But the higher end the rest of your equipment are, the more important it is for you to complement it with a capable DAC.