Right, as frequency is a function of amplitude cycles over time, the derivation of frequency requires no assumptions to conduct the Fourier transform for the purpose of easily analyzing the frequency content of the complex signal captured in the Dirac delta IR.
The issue now is that converting from FR to IR requires converting the imaginary time domain represented in the FR to the real time domain in the IR, which requires assumptions on when the recorded amplitudes occured in the original IR used to obtain the FR. All the information on amplitude over time is technically preserved in the FR, but there is no way to derive the real time domain component back out of the imaginary time domain without assuming it's position in the phase.
If the system is LTI and minimum phase, you can convert back and forth with no issue because the assumption of phase position is correct, so theoretically you have no loss of information. As the system deviates from perfect phase alignment, the phase error incurred by converting the FR to IR via IFT becomes larger because of this.