Hmm? Did you do measurements for side speakers by having speakers to your side (in which case the obvious answer is to re-measure with the speakers in a new better position), or are you trying some virtualization on top of virtualization?
If you measured speakers from one position and the binaural reproduction images them from a different position, something is wrong with either the measurements or the binaural rendering/mixing. You can only move the virtual speakers forward if you have corresponding measurements for those HRTFs. For example, since I used https://www.earfish.eu/ to capture my whole HRTF, I have enough data points to do a "binaural pan" of the virtual speakers to almost any direction. If you for example only have side and front-center measurements, then you would need to do a special volume pan to move the left and right virtual channels closer along the line between the side speakers and the front-center speaker.
