I am in no way familiar with the functions and specific features, quality etc of the 4i4 so my comments should be read in that light. I also don’t know what you are doing with the 4i4 and if or why that is an important part of your set up.
What you are currently doing is exactly as you said. That is adding the Magni + to the headphone out of the 4i4 so it is literally just amplifying the existing signal. The Magni + is a fairly uncoloured amp so if volume matched the sound with the Magni + will likely sound all but identical to that straight out of the 4i4, which is exactly as you have found.
If you are using the 4i4 for some specific use other than just listening to music and it is an important part of your system, you should be using one of the line out jacks on the back. I imagine you would need new cables comprising 2 x 1/4 inch TS to RCA for that purpose. That should, all things being equal, connect the Magni + to the DAC portion of the 4i4 rather than the amplification circuit.
HOWEVER, if the line out is variable and volume controlled from a knob on the front it may be more or less exactly the same as connecting to the headphone out as you currently are. If that were the case the audio quality is going to be determined by the 4i4 and the Magni + will effectively achieve nothing at all.
The only difference in that instance might be the output impedance of the headphone out versus the line outs but that in itself may not make any obvious difference either.
If that is the case, and given the 4i4 is somewhat budget oriented and designed for a different use case than a specific audio only DAC, it may be that the output of it either via headphones or the line outs isn’t especially good, I don’t know, that is an assumption on my part.
If you are just listening to music, and all the other features of the 4i4 are not important in that application, you may very well get better audio quality results using a Schiit Modi + DAC instead of the 4i4. That is use a Modi + and the Magni + and set the 4i4 aside for whatever else it is you do with it.