[krismusic]

Streaming is the miracle of my lifetime as far as I am concerned. To have access to an overwhelming amount of the world's music in my pocket. To be able to play an example of virtually any artist who crops up in conversation, is little short of astounding.

 

[sander99]

I have a question, very on topic of this thread I think.
Where could I find a suitable "test set" of audio fragments for use in controlled level matched double blind ABX tests, or somewhat simpler level matched blind tests?
I assume there are known "test sets", some more and some less extensive, that are suitable for finding all or most possible audible differences?
Instead of actual audio data a list of albums (including specific master if relevant), track, time interval in the track would also be appriciated (maybe even better for me).
(Of course I don't expect anyone to make such a list for me, just if you happen to know one...)

And maybe, not needed for convincing me but maybe needed for convincing others, it would be nice to have a "high res" test set, of real 24 bits 192 kHz audio fragments. (So that when not hearing differences in some test, nobody can blame it on using "inferior" audio fragments.)

 

[taffy2207]

Streaming is the miracle of my lifetime as far as I am concerned. To have access to an overwhelming amount of the world's music in my pocket. To be able to play an example of virtually any artist who crops up in conversation, is little short of astounding.

Each to their own but I hate streaming due to the low rates Streaming Services pay :-

https://www.digitalmusicnews.com/2018/12/25/streaming-music-services-pay-2019/

I tend to buy direct from Bands websites, via Bandcamp or occasionally buy a CD. Streaming is not for me anyway due to my predominantly obscure & diverse Music tastes

Spoiler: Off Topic Anecdote

Interesting anecdote. I had only 1 illegal download in my Music collection up until last year (Wayland - Sticked & Stoned). I'd searched for the Album for years and couldn't find it anywhere except as an illegal download. Last year I found the CD on Amazon for 72 pence, brand new. Go figure

 

[bigshot]

Where could I find a suitable "test set" of audio fragments for use in controlled level matched double blind ABX tests, or somewhat simpler level matched blind tests?

Just take any well regarded "HD Audio" recording and knock it down to the formats you're comparing to... 16/44.1, AAC, MP3, etc. Then bump them back up to HD Audio and normalize them to the same place. Any audio editing program will be able to do that, and you'll end up with files that all look the same, as long as the test participants don't peek at the waveforms.

All true, but your listening choices don’t generally represent the market that streaming is currently targeting. I don’t agree with your assessment that streaming services are largely good for “greatest hits”.

The definition of where the lines for esoteric and "greatest hits" lie on the spectrum depend on where the listener is on his own musical journey. I know that I'm not a typical consumer. Just pointing out that some people are at a point where streaming just can't serve all of their needs any more.

I also think we are going to see a contraction in streaming libraries. It happened in cable TV and home video. At first, they can't put out new content fast enough. Then they realize some things don't have as large of an audience and they start weeding.

Every time a new format comes in, a lot of stuff gets left behind on the old format. There's a vast amount of material that never made it from 78 to LP, from LP to CD, and from CD to streaming. When I was starting out in jazz, there were series of CDs with complete runs of music by a particular artist... Every Fletcher Henderson side in chronological order, every Cab Calloway, Fats Waller, Erskine Hawkins, even John Kirby and Ambrose and his Orchestra. These were hugely important to me in my research into the history of jazz. Now there are some greatest hits CDs, and those are on streaming. The rest of it is all out of print and unavailable on any format. It's sad, because the technology allows for more complete archives, not less. But that isn't how it works. If it isn't being licensed for sale any more, it isn't available. There's no archive of past streaming content like a used record store.

I remember going to Tower Records in the 80s and finding amazing new things every week. I would come home with arm loads of stuff. Then it started getting thinner and thinner until it seemed like everything was like the end caps on the aisles... hit of the day. That may happen in streaming too. They also might start dividing things up into corporate streaming services... Universal will have theirs, Sony will have theirs, Warner will have theirs... and you have to subscribe to all of them, or choose one. Or maybe slimmed down streaming libraries and you have to pay more for "all you can eat".

I really don't care how I get music or what format it is, just as long as I can get to the music I want. My media server library of rips seems to be the best choice to depend on for the long term. I'm really glad I built my CD library when I did. At least half of it isn't on streaming. Yes, the half of it that is streamable amounts to a huge chunk of listening time... perhaps 10,000 hours. But when I want to hear something from that other 10,000 hours that isn't on streaming... say a specific Erskine Hawkins B side mentioned in a book... listening to a greatest hits of Erskine Hawkins won't do the job.

I guess music serves different purposes to different people. I love art books that have catalogs with photos of every painting in the National Gallery of Art or the Louvre or the Hermitage too. The Taschen complete Rembrandt sketches books, or complete Michaelangelo, DaVinci and Schiele are my meat. Best of Van Gogh books don't serve the same purpose to me.

Edit: One side note... part of the explosion of back catalog historical recordings in the 80s had to do with the copyright law. There actually was a public domain back then. When the copyright extension bill passed, and the Naxos case came down, it all changed. Hopefully, with the clock ticking on copyright again, we will see more older recordings again bit by bit. But that will likely be on non-corporate controlled services. I'd like to see the internet go back to being the wild west again. It's good for culture.

 

[sander99]

Just take any well regarded "HD Audio" recording and knock it down to the formats you're comparing to... 16/44.1, AAC, MP3, etc. Then bump them back up to HD Audio and normalize them to the same place. Any audio editing program will be able to do that, and you'll end up with files that all look the same, as long as the test participants don't peek at the waveforms.

Oh, I did not want them specifically for comparing audio formats, but for comparisons in general, also to get some personal experience with these things.
And more specific I wanted to check the influence of adding a cheap little DSP box with analog inputs and outputs and hence AD and DA conversion (48 or 44.1 kHz) in my "system". (In fact I have 4 of those boxes with 4 inputs and 4 outputs and also wanted to try to loop a stereo signal through them all twice, so that it passes 8 AD and DA conversions in a row. I figured that if that doesn't give a clear audible difference then there is no need to worry at all about one such a box in the chain.)

 

[bigshot]

I don't think I understand what you are intending to compare. An ABX test is intended to compare two sound samples to see if a difference between them can be discerned. It tests to see if two sounds are the same or not. If a DSP box is doing its job, there should be a clear difference between the enhanced audio and the direct audio with no enhancement. The only question there is a subjective decision whether you like the enhancement or not.

If you want to see if conversion from 24/96 to 16/44.1 makes a difference, then the way I described before will do that. If your DSP boxes have a direct pass through with no filters applied, you could do a test switching between a direct level matched signal and the pass through from the DSP box. That would tell you if the DSP box added degradation. But I don't know if a direct pass through would necessarily apply the data rate conversion. You'd need to find that out from the manual.

 

[bfreedma]

Just take any well regarded "HD Audio" recording and knock it down to the formats you're comparing to... 16/44.1, AAC, MP3, etc. Then bump them back up to HD Audio and normalize them to the same place. Any audio editing program will be able to do that, and you'll end up with files that all look the same, as long as the test participants don't peek at the waveforms.



The definition of where the lines for esoteric and "greatest hits" lie on the spectrum depend on where the listener is on his own musical journey. I know that I'm not a typical consumer. Just pointing out that some people are at a point where streaming just can't serve all of their needs any more.

I also think we are going to see a contraction in streaming libraries. It happened in cable TV and home video. At first, they can't put out new content fast enough. Then they realize some things don't have as large of an audience and they start weeding.

Every time a new format comes in, a lot of stuff gets left behind on the old format. There's a vast amount of material that never made it from 78 to LP, from LP to CD, and from CD to streaming. When I was starting out in jazz, there were series of CDs with complete runs of music by a particular artist... Every Fletcher Henderson side in chronological order, every Cab Calloway, Fats Waller, Erskine Hawkins, even John Kirby and Ambrose and his Orchestra. These were hugely important to me in my research into the history of jazz. Now there are some greatest hits CDs, and those are on streaming. The rest of it is all out of print and unavailable on any format. It's sad, because the technology allows for more complete archives, not less. But that isn't how it works. If it isn't being licensed for sale any more, it isn't available. There's no archive of past streaming content like a used record store.

I remember going to Tower Records in the 80s and finding amazing new things every week. I would come home with arm loads of stuff. Then it started getting thinner and thinner until it seemed like everything was like the end caps on the aisles... hit of the day. That may happen in streaming too. They also might start dividing things up into corporate streaming services... Universal will have theirs, Sony will have theirs, Warner will have theirs... and you have to subscribe to all of them, or choose one. Or maybe slimmed down streaming libraries and you have to pay more for "all you can eat".

I really don't care how I get music or what format it is, just as long as I can get to the music I want. My media server library of rips seems to be the best choice to depend on for the long term. I'm really glad I built my CD library when I did. At least half of it isn't on streaming. Yes, the half of it that is streamable amounts to a huge chunk of listening time... perhaps 10,000 hours. But when I want to hear something from that other 10,000 hours that isn't on streaming... say a specific Erskine Hawkins B side mentioned in a book... listening to a greatest hits of Erskine Hawkins won't do the job.

I guess music serves different purposes to different people. I love art books that have catalogs with photos of every painting in the National Gallery of Art or the Louvre or the Hermitage too. The Taschen complete Rembrandt sketches books, or complete Michaelangelo, DaVinci and Schiele are my meat. Best of Van Gogh books don't serve the same purpose to me.

Edit: One side note... part of the explosion of back catalog historical recordings in the 80s had to do with the copyright law. There actually was a public domain back then. When the copyright extension bill passed, and the Naxos case came down, it all changed. Hopefully, with the clock ticking on copyright again, we will see more older recordings again bit by bit. But that will likely be on non-corporate controlled services. I'd like to see the internet go back to being the wild west again. It's good for culture.

The ability to manage both my local library of thousands of CD rips and integrate a streaming service is why I find Roon so compelling. Why have to choose one delivery model when I can have both? If things change in the future, I’ll adjust.

 

[sander99]

I don't think I understand what you are intending to compare.

The boxes only have analog inputs and outputs. Any signal going through is converted to digital and back to analog. So I just want to check, without actually applying any DSP processing, if there is degradation due to the AD and DA conversions.

 

[bigshot]

Is there a bypass button so you can turn off the processing? Then just get a y adaptor and feed direct to one input and run the other through the box. Then just toggle between inputs. You might have to level match.

 

[sander99]

@bigshot: First of all, of course thank you for all the time and effort reacting to my question. I understand most of what you said and have that figured out myself already. My question was not really about the specific test I want to do now, but more general. Maybe you remember I was the one with the silly idea some time ago to maybe do sighted testing before blind testing, because I thought that would increase the chance to find "suspected differences" and suitable audio fragments for possible confirmation in a controlled blind test. And of course if not confirmed in a controlled blind test then these "suspected differences" would have to be dismissed. The idea behind this was that some differences may only be audible with suitable audio content, not just any randomly choosen audio fragment. Then gregorio reacted with the following (amongst other things):

I can't think of a single published scientific audio double blind study where they used a "randomly chosen set of audio/music fragments", they always use a set of carefully chosen of audio fragments which maximise the chances of the differences being audible.

Re-readin the rest of that post just now I think he meant a set carefully choosen to test the audibility of an already known - from measurements(?) - specific problem. But I was thinking maybe there is a more general set of audio fragments that in general maximises the chance of finding any differences in any blind comparison listening test.
So that's why I asked the question: where could I find such a set of audio fragments? But it's not super important. It is not as if I am planning to do some very important scientific tests. But if I do a little test and share the results with a few people, I want to try to do it good enough to be taken a little bit serious by those people, that's all.

Just in case you are interested, this is the little 4 in 4 out DSP box I was talking about before, costing 77 Euro's:
https://www.thomann.de/gb/the_t.racks_dsp_4x4_mini.htm
It has a number of standard functions like delay and PEQ that can be configured from a pc via USB. The default settings are such that the signal passes unchanged, but it will always pass the AD and DA conversions and has some level drop that I counteracted by setting the ouput gains at about +7 dB (only for comparison, and while making sure I don't input a level that could cause clipping). I did this gain setting by ear for now but I have an idea how to do it more precise later.

 

[bigshot]

I usually test with music, not tones, because I listen to music. I have a bunch of tracks that have different characteristics that I can use to isolate specific parts of sound, but it isn't a fixed set of testing tracks. Just well recorded music with a full range of frequencies.

You shouldn't test to try to convince others though. Individuals can't test to the standards of peer reviewed scientific testing. Your own testing should be for your own edification. Listen to other people, and apply their suggestions, but you don't have to convince them. Some people can't be convinced no matter how good your test is. Just have a clear idea of the question you want answered and try to refine the test to answer it clearly.

 

[castleofargh]

@bigshot: First of all, of course thank you for all the time and effort reacting to my question. I understand most of what you said and have that figured out myself already. My question was not really about the specific test I want to do now, but more general. Maybe you remember I was the one with the silly idea some time ago to maybe do sighted testing before blind testing, because I thought that would increase the chance to find "suspected differences" and suitable audio fragments for possible confirmation in a controlled blind test. And of course if not confirmed in a controlled blind test then these "suspected differences" would have to be dismissed. The idea behind this was that some differences may only be audible with suitable audio content, not just any randomly choosen audio fragment. Then gregorio reacted with the following (amongst other things):

Re-readin the rest of that post just now I think he meant a set carefully choosen to test the audibility of an already known - from measurements(?) - specific problem. But I was thinking maybe there is a more general set of audio fragments that in general maximises the chance of finding any differences in any blind comparison listening test.
So that's why I asked the question: where could I find such a set of audio fragments? But it's not super important. It is not as if I am planning to do some very important scientific tests. But if I do a little test and share the results with a few people, I want to try to do it good enough to be taken a little bit serious by those people, that's all.

Just in case you are interested, this is the little 4 in 4 out DSP box I was talking about before, costing 77 Euro's:
https://www.thomann.de/gb/the_t.racks_dsp_4x4_mini.htm
It has a number of standard functions like delay and PEQ that can be configured from a pc via USB. The default settings are such that the signal passes unchanged, but it will always pass the AD and DA conversions and has some level drop that I counteracted by setting the ouput gains at about +7 dB (only for comparison, and while making sure I don't input a level that could cause clipping). I did this gain setting by ear for now but I have an idea how to do it more precise later.

when I don't know what I'm looking for, I just use a few of my favorite tracks because I knows them well enough that it might help me catch a difference(or so I hope^_^)
but maybe in your case you might want to measure(or at least listen), to some "silent" tracks(or something with a sine at -100dB), to test for added noise. and perhaps go the extra mile km and record the analog output with and without the box and check if there is any relevant change. then if you time and amplitude align the 2, you can play those in abx and still get a nice listening test.

 

[bigshot]

Why would something at -100dB matter?

 

[castleofargh]

Why would something at -100dB matter?

to have a signal we'd like to look at while near a chimney. no I was thinking about how some gears may put themselves on standby when nothing relevant is detected at the input. I only considered that because we're dealing with somehow special boxes compared to a typical amp or a DAC so maybe....?

 

[bigshot]

I guess that might be possible. My subwoofer does that. But putting a normal signal through it would fix that fast I guess.