All my troubles was gone after flashing the Franco Kernel (I'm using R155). I got both the headphone amp and volume boost on, I´m still experimenting with what amount that would be ideal. I´m highly recommend anyone with the N4 to install this kernel since its way better then the stock in all categories.
All in All, the nexus 4 is a solid source and does produce more then resonable sound without an amp.
Edited by SimonReborn - 6/19/13 at 5:13am