1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.

    Dismiss Notice

A layman multimedia guide to Immersive Sound for the technically minded (Immersive Audio and Holophony)

Discussion in 'Sound Science' started by jgazal, Nov 25, 2017.
2 3 4 5 6 7 8 9 10 11
  1. jgazal
    This post was written to try to answer the following questions:

    Since immersive audio is a broad subject, I believe it deserves its own thread.

    Question 1


    There would be something fundamentally wrong if all types off recordings suffered from such unique distortion. But each kind of recording has its own peculiar set of distortions.

    Spatial distortion

    Spatial distortion could refer to:

    a) what makes stereo recordings played back with loudspeakers different from stereo recordings played back with headphones;


    B) what makes our perception while listening to real sound sources different from listening virtual sound sources reproduced via electro-acoustical transducers.​

    Even if we define spatial distortion as the difference in perception of space of a real sound versus a reproduced sound, chances are every recording format played on every system would suffer spatial imprecision varying by degree, as described below.

    But do not interpret the word distortion in this post as negative alteration, because although varying by degree, they are not a "fatal" types of distortion, as recordings were made with full knowledge that the reproducing systems probably were not going to match the creative environment exactly. Perhaps with cheaper digital signal processing we have a chance to even both environments.


    As per item 1 you need to know how your recording was made and what level of spatial precision you want to achieve, because you need to know which kind of distortion you need to address.

    A layman multimedia guide to Immersive Sound for the technically minded (Immersive Audio and Holophony)

    Before you continue to read (and I don’t think anyone will read a post so long), a warning: the following description may not be scientifically and completely accurate, but may help to illustrate the restrictions of sound reproduction. Others more knowledgeable may chime in, filling in the gaps and correcting misconceptions.

    First of all you need to know how your perception works.

    How do you acoustically differentiate someone stroking a piano 440hz note from someone blowing an oboe and pressing its 440hz key?

    It is because the emitted sound: a) has the fundamental frequency accompanied by different overtones, partials that are harmonic and inharmonic to that fundamental, resulting an unique timbre (frequency domain) and b) has an envelope with peculiar attack time and characteristics, decay, sustain, release and transients (time domain).

    When you hear an acoustical sound source how can your brain can perceive its location (azimuth, elevation and distance)?

    Your brain uses several cues, for instance:

    A) interaural time difference - for instance sound coming from your left will arrive first at your left ear and a bit after at your right ear;​

    B) interaural level difference - for instance sound coming from your left will arrive with higher level than at your right ear;

    C) spectral cues - sound coming from above or below the horizontal plane that cross your ears will probably have not only a fundamental frequency a complex set of partials and then your outer ear (pinna) and your torso will change part of those frequencies in a very peculiar pattern related to the shape of your pinna and the size of your torso;

    D) head movements - each time you make a tiny movement with your head you change the cues and your brain track those changes according to its head position to solve ambiguous cues;​

    E) level ratio between direct sound and reverberation - for instance distant objects will have lower ratio and near sources will have higher ratio;​

    F) visual cues - yes visual cues and sound localization cues interact in the long and the short term;​

    G) etc.​

    A, B and C are mathematically described as a head related transfer function - HRTF.

    Advanced “watch it later” - if you want to learn more about psychoacoustics and particularly about the precedent effect and neuroplasticity, watch the following brilliant lectures from Chris Brown (MITOpenCourseWare, Sensory Systems, Fall 2013, published on 2014):

    So you may ask which kind of recording is able to preserve such cues (spatial information that allows to reconstruct a lifelike soundfield) or how to synthesize them accurately.

    One possible answer to that question could be dummy head stereo recordings, made with microphone diaphragms placed were each eardrum should be in a human being (Michael Gerzon - Dummy Head Recording).

    The following video from @Chesky Records contains a state of the art binaural recording (try to listen at least until the saxophonist plays around the dummy head).

    Try to listen with loudspeakers in the very near field at more and less +10 and -10 degrees apart and with two pillows one in front of your nose in the median plane and the other at the top of your head to avoid ceiling reflections (or get an IPad Air with stereo speakers, touch your nose in the screen and direct the loudspeakers sound towards your ears with the palm of your hands; your own head will shadow the crosstalk).​

    Just post in this thread if you perceive the singer displacing his head while he sings and the saxophonist walking around you.

    However, each human being has an idiosyncratic head related transfer function and the dummy head stays fixed, while your listener turn his/her head.

    What happens when you play binaural recordings through headphones?

    The cues from the dummy head HRTF and your own HRTF cues don’t match and as you turn your head cues remain the same and the 3D sound field collapses.

    What happens when you play binaural recordings through loudspeakers without a pillow or a mattress between the transducers?

    Imagine a sound source placed to the left of a dummy head in an anechoic chamber and that, for didactic reasons, a very short pulse, coming from that sound source is fired into the chamber and arrives at the dummy head diaphragms. It will arrive first at its left diaphragm and after and lower in its right diaphragm. End of it. Only two pulses recorded because it is an anechoic chamber with fully absorptive walls. One intended for your left ear and the other for your right ear.

    When you playback such pulse in your listening room, first the left loudspeaker fires the pulse into your listening room, it arrives before at your left eardrum, but also after and lower at your right eardrum. When the right speaker fires the pulse (the second arrival at the right dummy head diaphragm), it arrives first at you right ear and after and lower at your left ear.

    So you were supposed to receive just two pulses, but you end receiving four pulses. If you now give up the idea of very short pulse and think about sounds, you can see that there is an acoustic crosstalk distortion intrinsically related to loudspeakers playback.

    Since the pinna filtering from the dummy head fired into the listening room and interact with its acoustics, even not attempting to tackle acoustic crosstalk, there is a tonal coloration, that engineers try to compensate to make such recording more compatible with loudspeakers playback:

    To make things worse there are early reflections boundaries in typical listening rooms. Early reflections arrive closely enough at your eardrum to confuse your brain. More “phantom pulses” or distortions that you have in the time domain. A short video explaining room acoustics:

    There is also one more variable, which is speaker directivity. One concise explanation:

    A rather long video, but Anthony Grimani, while talking at Home Theater Geek by Scott Wilkinson about room acoustics, gives a good explanation about speaker directivity (around 30:00):

    Higher or lower speaker directivity may be preferred according to your aim.

    Finally, the sum of two HRTF filterings (the dummy head and yours) may also introduce comb filtering distortions (additive and destructive interactions of sound waves).

    With "binaural recordings to headphones" and "binaural recordings to loudspeakers" resulting no benefits, what could be an alternative?

    Try something easier than a dummy head, such as ORTF microphone pattern:

    For didactic reasons I will avoid going deeper into diaphragm pick up directivity and the myriad of microphone placing angles that one could use in a recording like this (for further details have a look at The Stereo Image - John Atkinson - stereophile.com).

    So for the sake of simplicity, think about such ORTF pattern, as depicted above: just two diaphragms spaced at the average size of an human head so you can skip mixing and record direct to the final audio file.

    Spectral cues are gone. Some recording engineers place foam disks between the microphones to keep the ILD closer to what would happen with human heads.

    An example:

    What happens when you play such “ORTF direct to audio file” with loudspeakers? You still have an acoustic crosstalk, but you have the illusion that the sound source is between the loudspeakers and at your left.

    What happens when you play such “ORTF direct to audio file” with headphones? You don’t have acoustic crosstalk distortion, but ILD and ITD cues from the microphone arrangement don’t match your HRTF and as you turn your head cues remain the same and the horizontal stage collapses.

    Are there more alternatives?

    Yes, there are several. Some are:

    A) Close microphones to stereo mix.​

    Record each track with a microphone close to the sound source and mix all of them into two channels using ILD to place them in the horizontal soundstage (panning - pan pot).

    Reverberations from the recording venue need to be captured in other two tracks from an microphone arrangement that allow then preservation of such cues an that is mixed into those two channels.

    The ILD of each instrument track (more precisely the level such track will have the right and left channel) and the ratio between instrument tracks and reverberation tracks is chosen by the engineer and do not necessarily match what a dummy head would register if all instruments were playing together around it during the recording.

    You still have acoustic crosstalk when playing back with speakers and soundstage will collapse with headphones.

    In this case, if the ILD/ITD levels are unnatural, when using headphones, as @71 dB advocates, adding electronic crossfeed may avoid the unnatural perception that sound sources are only at the left diaphragm, at the right diaphragm and right in the center:

    How many recordings have unnatural ILD and ITD? IDK.

    B) Mix multichannel with level panning.
    @pinnahertz describes some historical facts about multichannel recording here.

    Even with more channels, there is still some level of acoustic crosstalk when playing back through loudspeakers in regular room, unless all sounds/tracks are hard-panned.

    You just can’t convey proximity as one would in reality (a bee flying closely your head; or the saxophonist from Chesky recording above if the speakers are placed further than the distance the saxophonist was playing when he was recorded...).

    C) Record with Ambisonics.

    This is interesting as the recording is made with a tetrahedron microphone (or an eigenmike, as close as possible to measure a sound field at a single point) and the spherical harmonics are decoded to be played back into an arrangement of speakers around the listener.

    So with Ambisonics the spatial effect is not derived from two microphones but at from at least four microphones that encode height spatial information (The Principles of Quadraphonic Recording Part Two: The Vertical Element By Michael Gerzon) and the user HRTF is acoustically filtered at playback.

    There are two problems.

    The first one is that you need a decoder.

    The second one is that at high frequencies the math proves that you need too many loudspeakers to be practical:

    Although the spherical harmonics seem more mathematically elegant, I still do not figured out how acoustic crosstalk in listening rooms - or instead the auralization with headphones without adding electronic crosstalk - affects the possibility of conveying sound fields and proximity, neither if crosstalk cancellation in high order ambisonics ready listening rooms with high directivity loudspeakers is feasible.

    Let’s hope third order ambisonics, eigenmikes and clever use of psychoacoustics are good enough! See item 2.C of the post #2 below or here to have an idea of such path.

    If ambisonics spherical harmonics decoding already solves acoustic crosstalk at at low and medium frequencies, then the only potential negative variables would be the number of channels for high frequencies and the listening room early reflections. That would be in fact an advantage of convolving a high density/resolution HRTF/HRIR (when you can decode to an arbitrary higher number of playback channels) instead of interpolated low density/resolution HRIR or BRIR (of sixteen discreet virtual sources for instance), when binauralizing ambisonics over headphones. Current, state of the art, example of High-Order-Ambisonics-to-binaural rendering DSP:

    Problem is that, currently, HRTF are acoustically measured in anechoic chambers, a costly and time consuming procedure:

    In the following video Professor Choueiri demonstration of a high density/resolution HRTF acquisition through acoustical measurements:

    That is one of the reasons why this path may benefit from easier ways to acquire high density/resolution HRTF such as capturing biometrics and searching for close enough HRTF in databases:

    Fascinating research in 3D Audio and Applied Acoustics (3D3A) Laboratory at Princeton University:

    D) Wavefield synthesis.

    This one is also interesting, but complex as the transducers tend to infinity (just kidding, but there are more transducers). You will need to find details somewhere else, like here. And it is obviously costly!

    E) Pure object based.

    Record each sound source at its own track and don’t mix them before distribution. Just tag them with metadata describing their coordinates.

    Let the digital player at the listening room mix all tracks considering the measured high density/resolution HRTF of the listener (or lower density/resolution with better interpolation algorithms) and room modeling to calculate room reflections and reverberation (“Accurate real-time early reflections and reverb calculations based on user-controlled room geometry and a wide range of wall materials” from bacch-dsp binaural synthesis).

    Playback with crosstalk cancellation (or binaural beamforming with an horizontal array of transducers such as the yarra sound bar; more details forward in this post) or use headphones with head-tracking without adding electronic crossfeed.

    This is perfect for the realms of virtual environments for video games in which the user interact with the graphic and narrative context determining the future states of sound objects.

    What are the problems with a pure object based approach? As we have just seen, it is costly and time consuming to measure the HRTF from the listener. It is computational intensive to mix those tracks and calculate room reflections and reverberation. You just can’t calculate complex rooms. So you miss the acoustic signature of really unique venues.

    Atmos and other hybrid multichannel object based codecs use also beds to preserve some cues. But such beds and the panning of objects between speakers also introduce the distortions from the chains before mentioned (unless it also relies in spherical harmonics computation?).​

    Going the DSP brute force route to binauralization and to cancel (or avoid) crosstalk

    Before whe start talking about DSP, you may want to grasp how they work mathematically and one fundamental concept to do that is the Furrier transform. I haven’t found better explanations than the ones made by Grant Sanderson (YouTube channel 3blue1brown):

    A. The crosstalk cancellation route of binaural masters

    So what Professor Edgar Choueiri advocates?

    Use binaural recordings and play them back with his Bacch crosstalk cancellation algorithm (his processor also measures a binaural room impulse response to enhance his filter and use headtracking and interpolation to relieve head movement range restrictions one would have otherwise).

    Before we continue with Bacch filter, a few notes about room impulse response and a recovery of speaker directivity.

    There is a room impulse response for each (Length x Width x Height) coordinates of a given room. A RIR can be measured by playing a chirp sweep from 20hz to 20khz from a source in a given coordinate. The microphone will capture early reflections and reverberation at another given coordinate. Change those source and microphone coordinates/spots and RIR is going to be different. Room enhancement DSPs use RIR to compute equalization for a given listening spot (Digital Room Equalisation - Michael Gerzon), but if you want even bass response across more listening spots then you probably need more subwoofers (Subwoofers: optimum number and locations - Todd Welti). A BRIR is also dependent of the coordinates in which it is measured (and looking angle!). But instead you measure with two microphones at the same time at the entrance of a human head or dummy head. That is one of the reasons why you need to capture one BRIR for each listening spot you want the crosstalk canceled filter to work. Such BRIR integrates not only the combined acoustic signature of loudspeakers and room, but also the HRTF of the dummy head or the human wearing the microphones.

    Here an speaker with higher directivity may improve the performance of the algorithm.​

    Dummy head HRTF used in the binaural recording and your own HRTF don’t match, but the interaction between speakers/room and your head and torso “sum” (filter) your own HRTF.

    The “sum” (combined filtering) of two HRTF filterings may also introduce distortions (maybe negligible unless you want absolute localisation/spatial precision?).

    Stereo recording with natural ILD and ITD, like the ORTF discribed above, render an acceptable 180 degrees horizontal sound stage. Read about the concept of proximity in the Bacch q&a Professor Choueiri has in the 3d3a of Princeton website.

    Must watch videos of Professor Choueiri explaining crosstalk, his crosstalk cancellation filter and his flagship product:

    Professor Choueiri explaining sound cues, binaural synthesis, headphone reproduction, ambisonics, wave field synthesis, among others concepts:

    B. The crosstalk avoidance binauralization route

    B.1 The crosstalk avoidance binauralization with headphones

    And what you can do with DSPs like the Smyth Research Realiser A16?

    Such processing circumvent the difficulty in acquiring an HRTF (or head related impulse response - HRIR) by measuring the user binaural room impulse responses - BRIR (or also personal room impulse response - PRIR, when you want to say that it refers to the unique BRIR of the listener) that also includes the playback room acoustic signature.

    Before we continue with the Realiser processor, a few notes about personalization of BRIRs. Smyth Research Exchange site will allow you to use your PRIR made with a single tweeter to personalize BRIRs made by other users in rooms you may be interested to acquire. So the performance will be better than just use that BRIR that may poorly match yours.​

    So after you measure a PRIR, the Realiser processor convolves the inputs with such PRIR, apply a filter to take out the effects of wearing the headphones you have chosen, add electronic/digital crossfeed and dynamically adjust cues (headtracking plus interpolation) with headphones playback to emulate/mimic virtual speakers like you would hear in the measured room, with the measured speaker(s), in the measured coordinates. Bad room acoustics will result bad acoustics in the emulation.

    But you can avoid the addition of electronic/digital crossfeed to emulate what beaforming or a crosstalk cancellation algorithm would do with real speakers (see also here and here). This feature is interesting to playback binaural recordings, particularly those made with microphones in your own head (or here).

    The Realiser A16 also allows equalization in the time domain (the latter is very useful to tame bass overhigh).

    Add tactile/haptic transducers and you feel bone conducting bass not affected by the acoustics of your listening room. Note also that the power requirements to feed the headphone cavity are lower than feeding your listening room (thanks to brilliant @JimL11 for elaborating the power concept here) and that the intermodulation distortion characteristics of the speakers amplifier may then be substituted by those from the headphone amplifier (potentially lower IMD).

    In the following (must hear) podcast interview (in English), Stephen Smyth explains concepts of acoustics, psychoacoustics and the features and compromises of the Realiser A16, like bass management, PRIR measurement, personalization of BRIRs, etc.

    He also goes further and describes the lack of absolute neutral reference for headphones and the convinience of virtualizing a room with state of the art acoustics, for instance “A New Laboratory for Evaluating Multichannel Audio Components and Systems at R&D Group, Harman International Industries Inc.” with your own PRIR and HPEQ for counter-filtering your own headphones (@Tyll Hertsens, a method that personalizes room/pinnae and pinnae/headphones idiosyncratic coupling/filtering and keeps the acoustic basis for Harman Listening Target Curve).

    Extra interview at CanJam SoCal 2018 (by @kp297):

    Stephen Smyth at Canjam SoCal 2018 introduces the Realiser A16 (thanks to innerfidelity hosted by @Tyll Hertsens):

    Stephen Smyth once again, but now introducing the legacy Realiser A8:

    Is there a caveat? An small one but still yes, visual cues and sound cues interact and there is neuroplasticity:

    Kaushik Sunder, while talking about immersive sound at Home Theater Geek by Scott Wilkinson, mentions how we learn since our childhood to analyze our very own HRTF and that such analyses is a constant learning process as we grow older with pinnae getting larger. But he also mentions the short term effects of neuroplasticity at 32:00:

    Let’s hope Smyth Research can integrate the Realiser A16 to virtual headsets displaying stereoscopic photographs of measured rooms for visual training purpose and virtual reality.

    B.1 The crosstalk avoidance binauralization with an phased array of transducers

    So what is binaural beamforming?

    It is not crosstalk cancellation but the clever use of a vertical phased array of transducers to control sound directivity resulting a similar effect.

    One interesting description of binaural beamforming (continuation from @sander99 post above):

    Peter Otto interview about binaural beamforming at Home Theater Geek by Scott Wilkinson:

    Unfortunately the Yarra product does not offer a method to acquire a PRIR. So you can enjoy binaural recordings and stereo recordings with natural ILD and ITD. But you will not experience precise localization without inserting some personalized PRIR or HRTF.

    Living in a world with (or without) crosstalk

    If you want to think about the interactions between the way the content is recorded and the playback rig and environment and read other very long post, visit this thread: To crossfeed or not to crossfeed? That is the question...

    Question 2

    I find this question somehow harder.

    You already know mixing engineers place sound sources between speakers with level differences. Some may also use ITD in conjunction and that helps with a better rendering when you use binaural beamforming or crosstalk cancellation with loudspeakers playback.

    Recordings with coincident microphones direct to the final audio file incorporate early reflections and reverberation in a given venue spot.

    Instruments recorded with closed microphones in several tracks and then mixed with other two tracks or digital processing that incorporate early reflections and reverberation in a given venue spot may theoretically give a sense of depth with two loudspeakers playback. But what is the level ratio between those tracks?

    Can you mix them keeping the same ratio one would have with a binaural recording alone?

    But not treating early reflections and bass acoustic problems in the playback room may also null any intelligibility of such depth you could theoretically convey.

    I am curious to know how the processors that increase soundstage depth, that @bigshot mentioned here, work. An example would be the Illusonic Immersive Audio Processor, from Switzerland, a company that also provides the upmiximg algorithm in the Realiser A16, which extracts direct sound, early reflections and diffuse field in the original content and plays them separately over different loudspeakers:

    Anyway, without a binaural recording and crosstalk cancellation (or binaural beamforming), playing back plain vanilla stereo recordings, you could only hear an impression (and probably a wrong one) of elevation by chance, in other words, if some distortion downstream is similar to your very own pinna spectral filtering. With multichannel you can try the method @bigshot describes here.

    Going further in the sound science forum.

    The following threads are also illustrative of concepts involved in the questions raised:

    A) Accuracy is subjective (importance of how content is produced);
    B) How do we hear height in a recording with earphones (the role of spectral cues);
    C) Are binaural recordings higher quality than normal ones? (someone knowledgeable recommended books you want to read);
    D) About SQ (rooms and speakers variations)
    E) Correcting for soundstage? (the thread where this post started)
    F) The DSP Rolling & How-To Thread (excellent thread about DSP started and edited by @Strangelove424)

    To read more about the topics:

    P.s.: I cannot thank @pinnahertz enough for all the knowledge he shares!
    Last edited: Jul 3, 2018
  2. jgazal
    More examples of attempts on immersive audio:

    Last edited: Dec 17, 2017
  3. bigshot
    I actually recently stumbled across some recordings that seem to be trying to recreate a totally natural realistic sound. And they did it in a very non-intuitive way that doesn't follow the rules.

    I posted in the lounge about a deal at JPC on multichannel SACDs (4 euros apiece!) I got a big batch of them and after I listened, I immediately went back and ordered the rest. For $300 including shipping, I got about 70 SACDs. The performances are top notch. They were recorded in the 1990s as the Royal Philharmonic was preparing for a 50 year anniversary tour of the world. But as good as the performances are, that isn't what makes them interesting for us here.

    The liner notes says that they used as many as 45 microphones to record these. That normally would terrify me because I look to the Living Stereo recordings recorded with two or three mikes as the standard of quality for recording classical music. But they've done something very interesting with all those mikes. I've listened to several of them so far and they appear to be trying to recreate the sound of an orchestra from the audience in a great concert hall, which makes sense for multichannel. But there are differences between the way an orchestra sounds in a concert hall and how it sounds on a record.


    They have chosen somewhere towards the front of the orchestra section as their optimal point for the recording. It's close enough to get a very broad dynamic range, yet out in the open enough to pick up all of the hall ambience. The dynamics are very wide. These push the range of what I expect the dynamics to be on a classical recording. My spit in the wind estimate is somewhere between 60 and 70dB. This means that you have to listen to these LOUD. At a more moderate listening level, stuff in the quiet parts starts to fade into the distance, and the hall ambience isn't nearly as clearly presented. If you adjust it to the sweet spot, at least 10dB louder than most classical recordings, in a particularly dynamic piece, like Tchaikovsky's Pathetique, the peaks are right at the edge of being too loud. But the conductor modulates the performance so it isn't tiring to listen to. I think the ideal with most classical recordings is based on a theoretical listening position ten or fifteen rows back from the band where the hall has the opportunity to suck up some of the dynamics and the level of the dynamics is more in the 55 to 60dB range. But at the proper listening level, a little more dynamics certainly adds to the natural presence of the sound.


    The engineers on this recording have poured a lot of attention to the secondary depth cues. Instruments in the front of the band have less hall reverberation than ones in the back of the band. The percussion has a great deal of reverb on it- more than in any classical recording I've ever heard in fact. The percussion is also allowed to fall into the dynamics of the band. When there is a triangle hit, you don't hear it because of the volume of it, you hear it because it's a different frequency. Most classical recordings will mike things like that a little closer so they can get the fundamental frequencies of the triangle to cut through. But this recording allows the triangle to sound like a triangle from a distance. The same is true of the woodwinds. Even though you can hear the finest points of the phrasing, you don't hear any clacking of the keys. A lot of audiophiles judge the quality of a recording by whether they can hear tiny details like that, but I've never heard anything like that in a concert hall. The same goes for the sounds the conductor makes or squeaking of chairs... all those things are often audible in recordings, but when you're sitting 40 or 50 feet away from the band, there's no way you can hear any of that. The sounds that are projected sound like they are projected, the ones that are small and localized don't come through. Even though it seems counter-intuitive, details like this are what sound like close miking overlaid over distant miking. It doesn't sound natural. These recordings seem to have a very realistic presentation of depth.


    I'm still trying to get a grip on this aspect. I suspect that the unique sound of these recordings when it comes to directionality has something to do with the way distance is presented. The soundstage isn't as pinpoint clear as some classical recordings, for instance the three channel Living Stereo recordings. The sound is more diffuse at lower volumes and more directional at louder ones. I'm guessing that is a natural effect of the hall ambience. When something is loud, it cuts through like a laser beam, and when it is quieter, it tends to spread out into the massed sound of the band more. I've heard recordings with "spotlighting" where they will record a solo with a dedicated directional mike so they can mix it in clearly in position. This doesn't sound like that. A solo can go from loud to soft and the directionality will become more focused as it gets louder and more diffuse as it gets softer. Again, it goes against that old audiophile adage that everything in the soundstage must be pinpoint accurate at all times. But it works and it sounds VERY natural. I'm going to listen carefully and see if I can understand this better.


    The rear channels are almost silent at a moderate listening level. But when you get the volume up to a loud peak level, it blooms out and makes the back of the room feel live. It tends to pull upper mids and high frequencies out into the room a bit. For instance, trumpets seem to jump a little forward when they blare, and the triangle taps, even though they are low level, are echoed a little in the rear. But the lower frequencies tend to mass together into an undifferentiated reverberation. It's all gradated through the frequency range perfectly. I'm guessing this is also a very natural sort of way of handling the ambience. It definitely isn't the sort of ping pong surround that is common on a lot of multichannel mixes, but it sounds a little different than the way Pentatone or my blu-rays of classical music sound. The reverberation isn't as sharply defined below the midrange. It tends to bloom out into a sound that makes the room feel full, rather than echoing. I'm not sure how to describe it, except to say that it sounds very natural, even though it's not probably specific to the venue they recorded in. (C.T.S. Studios in London http://www.malonedigital.com/studios-cts.htm#.WixiGSPMyV4 ) I'm guessing that they analyzed some real concert halls and synthesized this sound from scratch, which is astounding considering how natural it sounds!

    So putting all these aspects together... The sound is more ambient and diffuse in some ways than most classical recordings, but the clarity is gradated along with the dynamics in a natural way. And the multiple channels provide more definition to put across a more sophisticated and natural soundstage like this.

    I'm guessing that they used so many microphones because they miked each section of the orchestra from several different distances with both broad and shotgun mikes. That way, if they needed to go in and fix the balance of a particular part, they could do it with an ambience that matched the perspective of the dynamics. I suspect that most of the time, engineers just take a wide, broad perspective of the orchestra as the foundation of the mix, and punch in with close miked details if they need to adjust anything. These engineers seem to be able to maintain the distance cues better when they do that so nothing sounds "spotlighted" and everything feels like a consistent, coherent distance. That would require several mikes on each section at different distances with different polar patterns optimized for what they were trying to isolate.

    I haven't listened to anything with a small band yet. I'm interested to see how they handle that. I would bet that they don't follow the big hall approach they did with the full orchestral recordings. I am interested to see what sort of environment they create for a more intimate scale.
    Last edited: Dec 9, 2017
  4. jgazal
    Yes, @bigshot, I was also surprised the National Orchestre of France recording linked above was mixed from several spot microphones.

    I was expecting coincident microphones, because I used to share the same fear you mentioned: more microphones, more variables can go wrong.

    Nevertheless, as Professor Choueiri said in one of the videos linked above room acoustics and HRTF are governed by linear physics, so it can be mathematically modeled. Even the HRTF can be mathematically modeled from the geometry of our heads and torsos (though computational intensive...).

    But I believe elevation is the worst part (The Principles of Quadraphonic Recording Part Two: The Vertical Element -Michael Gerzon). As I see it, spectral cues act together with ILD and ITD intricately.

    With binaural recordings the dummy head HRTF and particularly its spectral cues are convolved/convoluted (which is the right term?) and I guess the post equalization for speakers is an averaging. You would need to know the dummy head HRTF that was convolved/convoluted to neutralize the second HRTF filtering of the listeners at playback and that would be too computational intensive.

    That is why I guess Smyth Reaserch service to personalize BRIRs may be practically limited to ILD and ITD. Spectral cues may be not treated, which may explain the second tier performance.

    That is also why I believe acoustic crosstalk forbid sources to be perceived inside the limits of the surrounding speakers if you don’t use spherical harmonics. You can hard pan 12 instruments with 12 speakers in the horizontal plane around you but that is not very practical.

    With coincident microphones elevation information is encoded up to an optimal resolution depending on the order of microphones. And then the HRTF convolution happens at playback.

    Why I said all that?

    Because limiting my comments to an horizontal sound stage within the loudspeakers and channels in phase, then I agree that, even without crosstalk cancellation: (i) synthetic mixing of direct sound and reverberation can theoretically render distance and stage depth in the recording you mentioned; (ii) level and time differences can render directionality of instruments in the recording you mentioned; (iii) tracks from spaced microphones can add ambience.

    Nevertheless, I am sorry to say that, but I have never perceived depth in such circumstances within regular rooms without acoustic treatment. Until now I only perceived depth with binaural recordings with speakers in the very near field and two pillows, one blocking crosstalk and the other blocking ceiling reflections... I visited two rooms with acoustic treatments but it was so long ago I don’t remember...

    On the podcast above and in the Realiser A8 manual, Smyth Professors (I guess is fair to call both brothers professors also given their academic background) explains how we perceive distance: a mix of how plain is the wave reaching or head (so notice that time arrivals and envelope does matter) and the ratio between direct sound and reverberation (so in theory even with continuous sound waves we could perceive distance). And the Realiser alters the speaker distance by varying the second aspect (direct sound and reverberation ratio) just as you describe mixing engineers do.

    But it seems to me from your description that your listening room has well treated reflections in order to preserve the ratio of direct sound and reverberation. I don’t know if and how crosstalk alter such ratio.

    I still didn’t study how binaural synthesis mixing engines can change the perceived size of sound sources (sound objects). That is something curious because in some recordings I find the piano huge and in others I find it is thin and razor sharp. I will dig deeper.

    Interesting also that you find the dynamics somehow different from what you hear live. I don’t know if that alters the localization.

    I don’t know why @71 dB sees the crossfeed debate so negatively. It was one of the reasons that made me interested in comparing surround and ambisonics. I learned the effect of reducing crosstalk by hard panning in surround speakers with @pinnahertz. I had already read Daniel Levitin (This is your brain on music), but was @pinnahertz again that did alert me “don’t forget the envelope”.

    So I find the debates here really constructive.

    To sum up, I guess that coincident microphones or synthetic mixing, we need to test to say if the rendering is realistic. @Erik Garci like a PRIR that emulates the Hafler circuit, something that I would fear previously (perhaps because I don’t understand the mathematics of microphones directivities and angles). Visual cues do help to achieve a more secure reference (but stereoscopic headsets are another can of worms as @pinnahertz also alerted me).

    Please post the sacd booklet so others can test it also. One of the recording engineers of the studio you linked did some impressive work with motion picture sound tracks and Maurice Jarre. I have found this comment hilarious:

    But to be fair, I don’t know if post recording synthetic reverberation is as good as the reverberation in real time during the recording that would be correlated to the excitation music signal. If they play a sweep tone into the hall it is still related to one single spot, but the orchestral instruments are in several spots. Maybe a mix of several RIRs in several spots. I really don’t know... I just know that Maurice Jarre had a really strong opinion on the subject. :laughing: Perhaps he would change his opinion with RIR convolution reverbs instead of algorithmic reverbs that were available in his time?

    I would like to confess that I just can’t understand all those acoustical and psychoacoustical variables together... And I would also like to confess that I also find really amazing all that Rayleigh, Michael Gerzon, Choueiri and Smyth’s wrote and engineered. It is just unbelievable how humans can understand those patterns and digitally process them. To me such achievements are very similar to what made Max Planck solving the ultraviolet catastrophe or Einstein defining Newtonian mechanics as an special case of the more general relativity or fissioning an atoms in a chain reaction etc...

    Finally, thank very much for replying to this thread.
    Last edited: Dec 10, 2017
  5. bigshot
    I'm talking about speakers. None of that head stuff applies because I'm using my own head.
  6. jgazal
    Yes, I do agree. Either ambisonics through speakers or multichannel through speakers just convolve/convolute your head acoustically at playback, they are both personalized to your very own head.

    What I am trying to say is that your head would be fine if and only if the recordings were made with coincident microphones and the rendering had spherical harmonics computed to each speaker in order to recreated a the original sound field at a single point (the center of the listener’s head).

    But usually recordings are made with coincident pairs or mixed and playback adds acoustic crosstalk and reflections so that the ratio between direct sound and reverberation may not be as realistic as the first technique. But I do agree with you and @pinnahertz that multichannel has an edge over regular stereo, because the less you pan, the lower the crosstalk you introduce.
  7. bigshot
    Most mixes are constructed from mono tracks potted to where the engineer wants to place them. The shape of your noggin, crosstalk, or pattern of the mike are all irrelevant to it. You sit in front of the speakers and you get the sound. It works fine that way.
  8. jgazal
    So the mixing engineer adjust the “convolution reverb” and the dry spot microphone object ratio to define depth. With headphones the ratio is maintained. But with speakers, my gut feeling gives me the fear that crosstalk and reflections just don’t allow to keep the exact same ratio. Sure synthetic ILD and ITD may be enough for your brain to perceive an acceptable auditory illusion, so I will trust your experience.

    Nevertheless, consider the following two descriptions and after that just tell me if you are really sure that the interaction between “convolution reverb”, direct sound and crosstalk and at which point in the audio reproduction chain you convolve/convolute and HRTF are all completely “irrelevant” to where “the engineer want to place” the object:

    Since you may be the one of the most experienced with multichannel, I would like to ask you a couple of questions. Did you find multichannel recordings in which all sound objects are hard panned to each speaker? Do they convey the object size accurately? Are you able perceive the object inside the boundaries of the speakers around you? In other words, have you perceived proximity (see How Does BACCH™ 3D Sound differ from surround sound?)?

    Now consider something like the Microsoft HoloLens and that you want to connect the holographic world with the real world. Objects sizes must be visually coherent with the real objects in the scene:

    But the size of sounds and their proximity must also be coherent.

    Do you believe that multichannel is able to render size and proximity of audio objects with better coherence than (I) binaural through two loudspeakers with crosstalk cancellation, (ii) binaural through beaforming phased array of transducers, (iii) binaural with personalized headphone externalization and headtracking, (iv) Ambisonics through multiple loudspeakers or (v) Ambisonics downmixed to binaural using you own HRTF and playback with headphones or two loudspeakers with crosstalk cancellation or beaforming phased array of transducers?

    To be sincere, I still don’t know if option “iv” (coincident soundfield microphones or eigenmikes* and ambisonics decoding; the “pattern of the mic” variable on your terms) is also capable of conveying proximity in the sense Professor Choueiri describes. Dealing with reflections with increasing speakers is an increasing challenge. Perhaps option “v” with your own HRTF and headphone externalization with headtracking and without crossfeed? Maybe.

    * in other words, coincident microphones with 3 axis spatial information.
    Last edited: Dec 10, 2017
  9. bigshot
    Studio engineers mix on speaker systems, not headphones. If you play it back on speakers that are set up properly you aren't getting any distortion. It's the way they heard it in the studio. If you listen with headphones you're getting a certain amount of distortion by definition, because that wasn't what it was mixed for.

    There are ping pong recordings where everything comes from a different speaker. That was more common in the early days of quad when they didn't really know how to use multichannel yet. And yes, when something is thrown to just one speaker, it does sound smaller. Almost all current mixes either use the "Pink Floyd" approach (a bed of bass and drums and perhaps rhythm guitar in a natural acoustic and solo elements potted around) or the "Sound Field" approach (a natural acoustic with fixed positions for the instruments in two dimensional space between front and back) or a "Sound Stage" approach with fixed positions up front and just hall ambience in the rears). There are blends of the different approaches as well. Perhaps the most common approach for rock is a sound field up front with background vocals and fills thrown occasionally to the rears ala "Pink Floyd"

    If you want to associate a sound with an image in three dimensional space (like a hologram), it would require more speakers and more carefully meshed channels. At this point Atmos is the technology that can do that. Atmos is scalable up to 128 channels and 68 speaker feeds. In the home it's much more limited. It can support up to 35 channels there.

    Multichannel can do just about anything better than two channel speakers or binaural. Object size is determined by the number and placement of the speakers. You can use DSPs to optimize the room acoustics and synthesize any size space you want, from a tiny box around you all the way up to an arena. Atmos would be the most effective at this because it includes the vertical dimension, not just right and left and front and back. The advantage to speakers is that you are creating real sound in real space. You don't have to worry about head shapes or head tracking because all that is handled naturally by the listener.

    You're misunderstanding what I'm talking about with the number and types of microphones. The purpose isn't to capture a multichannel ambience like binaural. It's to capture sound objects that can be assembled into a multichannel ambience in the mixing board. When I talk about the pattern and placement of the microphones, I'm referring to the ability to isolate specific groups of instruments and the ability to capture secondary distance cues from the actual recording venue. Secondary cues are natural and real and add a level of verisimilitude to the ambience. Distance cues like reverb and changes in frequency response can be subtle and very specific. When you build a synthetic ambience using digital reverb, it can tend to create monochromatic ambiences where all instruments feel like they are the same distance away from the listener.

    You can also get disconnects in secondary sound cues when you mix them haphazardly. Say you want to boost the woodwinds. You take a close miked isolated track of them and mix them into the wide perspective to pull it up in the mix. Perhaps you add a digital reverb to make the close miked feed sound more distant. This certainly works, but it isn't quite the same as having a more distant isolated track with natural secondary distance cues to mix in. When you have these natural details recorded, you can sprinkle them into the mix and get an even more vivid perception of depth.

    I think what they did in these Royal Philharmonic SACDs was to mike the band a dozen different ways- close and distant, sections isolated, wide panoramas of the whole band, mikes dedicated to room ambience, etc. Then they took the 45 different channels and started assigning them to slots in the mixing board, which at the studio they recorded at had a whopping 96 channels! That would allow them to add a specific stereo ambience to each one of the 45 discrete channels and still have six channels left for 5.1.

    I'm guessing they created the depth of the mix in layers, isolating each section- percussion in the rear, woodwinds in the middle, strings in front of that, soloists in concertos right up front- then they applied digital hall ambience combined with natural secondary distance cues to each layer according to how far away they wanted it to feel. When they had the depth worked out for the three front channels, they synthesized a rear channel hall ambience with a 3 or 4 second decay and delays varied by frequency to allow higher frequencies to bounce clearer than lower ones.

    In the ideal world, all you would really need is five microphones placed in the positions of the speakers for the listener and a room big enough to have the natural ambience you are going for. That would give you a pretty exact recreation of the sound- just like the three channel Living Stereo sessions back in the 1950s. But if you did it that way you would have no flexibility at all to correct level imbalances or change an instrument's perspective or make it sound like a different hall ambience. By recording from 45 different vantage points on the band in the room and bringing it into a mixing board capable of adding a unique digital stereo ambience to almost all of those discrete channels, they would have complete control over the way distance, direction and ambience was rendered. It's like a film shoot with three cameras where one shoots the closeup, one shoots the two shot and one shoots the wide... you can capture a single performance three different ways so it can be assembled into a filmed presentation. Except here they are recreating a single perspective in great detail, not switching between different perspectives. Another analogy would be HDR photography. They take three different photos at three different exposure settings and merge them together into a photo that has the best elements of all three.

    It has nothing to do with head shapes or crosstalk or headphones. They are creating a synthetic auditory environment in the mixing board by pulling the sound apart into small bits, reassembling them in the mixing board layer by layer and sprinkling in real depth cues to make the synthetic ambience sound vividly real. Then they are channeling the sound out into three front speakers, two rears and a sub to create real dimensional sound surrounding the listener. This is the holographic sound you're talking about. If they upped the complexity and did an 8.2.6 mix like this, it would be even better than 5.1.

    By the way, I haven't listened to the 2 channel mixes on these SACDs, but if they are simple fold downs of the 5.1, I bet they sound really thick and awful. You need the multichannel to separate out all the sound objects properly.
    Last edited: Dec 10, 2017
    Strangelove424 likes this.
  10. jgazal
    I see that, likewise @gregorio and @pinnahertz, your reference is the mastering room. I cannot argue that, it is a more down to earth reference given the current mainstream equipment in consumer environments.

    So with that peculiar premise in mind, I agree with all you say, except the following statement:

    When Professor Choueiri explains his crosstalk cancellation algorithm, he likes to describe it as an “stereo purifier”.

    And he does that because he states that the standard Blumlein stereo is corrupted by crosstalk.

    So he does not even consider the word holophony and simply restores the meaning of word stereo:

    So thinking about your analogy to holograms, I am inclined to believe that holophony would be a better word than expressions like “immersive audio” or “immersive sound”.

    I didn’t want to use the word holophonia or holophonics since it was coined by Hugo Zuccarelli to describe his system that, AFAIK, is binaural recording without crosstalk cancellation.

    Apparently, Helmut Oellers in Germany is using the word to refer to Wave Field Synthesis: Holophony - the path to true spatial audio reproduction.

    Anyway, I am not an audio engineer or researcher, so I will live the thread title as is, since it was based on the “Immersive Sound” book.

    Having said that, I feel the urge to insist on this:

    Are you really sure that currently object based proprietary codecs, like Atmos, Auro and DTS:X, are capable of such holophony?

    Do you know why Professor Choueiri do not mention those codecs as the most promising methods for generating 3D soundfields?

    @gregorio and @pinnahertz, you both also know the music industry and mastering engineer: do you agree with @bigshot that Atmos, Auro and DTS:X are capable of conveying proximity (or holophony in a broader concept) without relying on crosstalk cancellation and convolution of objects with a generic HRTF?

    I think that being capable to convey proximity is a very important engineer specification of such codecs and that need to be really clear here.
    Last edited: Dec 16, 2017
  11. bigshot
    Atmos is an object based system. You take an individual track (like a specific musical instrument) as a discrete channel and the system processes it to place it anywhere within the three dimensional sound field. The channel doesn't necessarily relate to a specific speaker, rather it's plotted to groups of speakers that represent a specific point in space. The size and definition of the sound field is governed by the size of the room and the number of speakers in the installation, but the mix is the same regardless of the number of speakers. The more speakers you have, the more precise the placement in space. It's like being in a rectangular cubic grid of sound.

    Then they can add distance cues in three dimensions- say for instance a plot of how reflections, reverberation and decay work in a Gothic cathedral- and they can wrap that ambient envelope around the objects. That creates scale. The rectangular cube of sound defined by the size of the listening room is now able to create an environment of any size, shape or acoustic. Again, the precision is based on the number of speakers, and the mix is the same for a small installation as a big one.

    Now if that isn't holographic sound, I don't know what is. I'm sure all the research into crosstalk, phase and all that stuff can be a part of the processing of Atmos as well. It's a lot easier to get two speakers to mesh to create a coherent phantom center than it is to get a cube covered with 64 speakers to create coherent phantom centers between every possible pair of speakers. I'm sure there is a lot of room tuning going on to make it work in the real world. But since the placement and ambience is completely object based, rather than baked into the individual channels, it allows for infinite flexibility. You could theoretically take a Rolling Stones album recorded in Atmos and have them performing in an arena acoustic on stage a hundred yards from you, then change a DSP and have them standing in a recording studio six feet in front of you. I would bet that eventually, they could even create alternate mixes using the same recorded channels to make the instruments move around you in a circle or fly over your head. Since it's object based, the placement can be anything they want.

    Someone once mentioned to me that the order of magnitude of improvements to the quality of a sound field is related to the doubling of channels. So stereo is an order of magnitude better than mono, quad is an order of magnitude better than that, 8 channel, 16, 32, 64, etc. It's based on evenly spaced channels defining the four walls. You have to have each of the four walls equal to create a sound field that is significantly better.
    Last edited: Dec 10, 2017
  12. jgazal
    I admit is somehow difficult to describe what I mean. But my next two questions are very objective.

    I will quote Professor Choueiri and Lavorgna again to describe my doubt:

    So your Atmos horizontal speakers are seating in an imaginary circle of 2 meters in diameter. You are right at the center.

    The first question is: can you perceive David Chesky voice as if he were whispering next to your ear?

    By whispering next to your ear I mean more and less 10 centimeters away and not 1 meter away (since your are in the center of such imaginary circle).

    The second question is: is crosstalk cancellation currently part of Atmos?

    Edit - this is what Dolby claims about Atmos:

    I just don’t understand how they do that without crosstalk cancellation or beamforming...
    Last edited: Dec 12, 2017
  13. bigshot
    I think the difference between what you're talking about and the way music is recorded is that you are talking about capturing a real acoustic and perspective. That is rarely the goal in music recording. Recorded music is about creating an optimized acoustic and perspective. It's not trying to capture small things to make real sound more real. It's about creating a synthetic reality that sounds real but is better than real.

    If you want to have a sound be right next to your ear, you have to have a speaker right next to your ear. Headphones can do that easily.

    I'm not sure if Atmos incorporates that particular technique, but I'm sure a VST plugin could be created if there was some way to apply it.
    Last edited: Dec 10, 2017
  14. jgazal
    Yes that is the difference and the reason why I say that your reference is the mastering room.

    It was not a question about what I want, but what is currently possible to do with speakers.

    Most headphones if not all don’t externalize sounds without DSP.

    So do you think Professor Choueiri and Lavorgna were not precise when they described the proximity?

    What I want, in case you wonder, is to emulate with headphones what Professor Choueiri claim to do with speakers. That is currently possible.

    But I wish the Realiser to allow just a little more freedom in the mix block (or ILD) and in the ITD of the PRIR’s (with two speakers PRIRs and Ambisonics speakers arrangement PRIRs).

    But I haven’t received any answer from them so I am just losing my hope.
    Last edited: Dec 10, 2017
  15. jgazal
    This is a concise counter argument to what I have been writing in this thread:

    There are certainly at least two drivers to keep the reference locked to the mastering room when we are dealing with music content. One is objective and the other is let’s say volitive: a) the consumer environment; b) preference for such best seat in the audience.

    @bigshot, @gregorio and @pinnahertz can correct me, but they seem to agree with the first one.

    Someone that shares the opinion of @castleofargh regarding the second driver:

    But there may be a third driver: apprehension that one single coincident microphone could detract the creative intent of artist, producers, recording, mixing and mastering engineers.

    I don’t believe that shifting the reference can in anyway detract the creative intent:

    Giving the user access to 360 degrees of freedom in the x, y and z axis of a sound-field may sound challenging, but do not detract the creativity. It may be just a new language. This analogy with the cinema limited angle of view versus VR freedom of view might be useful:

    Last edited: Jan 26, 2018
2 3 4 5 6 7 8 9 10 11

Share This Page