[Nirmalanow]

The Tera player currently uses an algorithm that does not create evenly distributed random playback. After a lot of experimentation some of us came up with a system that adjusts for the algorithm on the Tera and creates perfectly even selection of songs in random play.
Note added: There is word that Charles has updated the algorithm used in more recent Tera players. You can probably check with him to see if your unit has the new algorithm. If a Tera is using the newer algorithm, then the instructions below become irrelevant. The new algorithm by itself should do a pretty good job of playing music randomly.
 
Here is a new set of rules that work at every level on the Tera (root, artist and album) to create perfectly random selection of songs:
 
1-If there are 1-4 folders or tracks in a particular folder, then add exactly 4 empty folders to that folder, and use numbers (00001, 00002, 00003...00010, 00011, etc.) to name the empty folders so they come before all other folders/tracks (this rule for numbering empty folders applies to all of the following rules).
2-If there are 5-8 folders or tracks in a a particular folder, then add exactly 8 empty folders.
3-If there are 9-16 folders or tracks, then add exactly 16 empty folders
4-If there are 17-32 folders or tracks, then add exactly 32 empty folders
5-If there are 33-64 folders or tracks, then add exactly 64 empty folders
6-If you have more than 64 folders or tracks at any level, then it is best to rearrange your music so that you have 64 or less at each level. This is because Tera is limited to 128 folders/files so you cannot add 128 blank folders to a folder with 65-128 real folders.
NOTE: You can stop here. If you just apply the first 6 rules to every folder on your card including the root folder, then you will experience a huge improvement in the randomness on the Tera...probably 90% of the possible improvement. The last two rules are only important if you want perfectly random playback of every song on the card.
7-For perfect randomness, all folders at the same level of organization should be in the same range: i.e. all artist folders should have either 1-4, 5-8, 9-16, 16-32 or 32-64 real album folders, and all album folders should have either 1-4, 5-8, 9-16, 16-32, or 32-64 real songs. Do not combine ranges at the same level. It is OK to have different ranges at different levels: i.e the root folder can have 32-64 artist folders in it and then all artist folders can have 1-4 album folders in them and that will still work. I would suggest dividing the folders/tracks in any folders with unusually high numbers of folders/tracks to bring them down to the range you plan to use at that level. Again this rule only matters if you want the randomness to be perfectly distributed across all of the songs.
8-Exception to rule 7: If your folders on the same level have contents that cross two of the lower ranges, then you can just use the higher range's number of empty folders. For example, if you have album folders with a wide range of say 3-16 tracks, then just add 16 empty folders to all album folders, and all of the tracks will have the same odds of playing. Or if instead you happen to have just a few very large albums with more than 16 tracks, then again I would suggest you divide them into two or more smaller album folders with 16 or less tracks each to bring them more in range with the other albums on your card.
 
If you play with the spreadsheet here (created by Suicidal_Orange), you will see that these rules always create a perfectly even selection of all of the "real" folders and also all of the "real" tracks at each level on your card. However, again I will point out that to have perfectly even randomness, you need to stay in the same range for all folders at each level: root, artist and album. You do not need to understand the mechanics, but if you follow the rules above and keep each level in the same range (i.e. all artist folders have say 1-4 real album folders and therefore 4 empty folders, and all album folders have say 9-16 real tracks and therefore 16 empty folders) then the randomness will be perfectly even throughout your card. Or just follow rules 1-6 and the randomness will be pretty much even across the whole card with only a few slight exceptions - the randomness will still be something like 90% better overall than it would be without adding empty folders at all.
 
A couple of extra  notes:
1-The root folder can have any number of real artist folders (except above 64) as it does not have to be similar to another root folder, because obviously there is only one root folder. Once you add the correct number of blank folders to the root folder, then all artist folders will be selected evenly in randomness. (And it may be obvious, but if there is only one artist folder in the root, then no need to add any empties, but for 2 or more artist folders, follow the rules above.)
2-When using blank folders as place holders, they should be named/numbered as follows 00001,00002,00003...00010,00011,00012, etc, so that they are sure to be arranged before any other folders/tracks on the card. This works as long as none of your tracks have a name that starts with more than 3 numbers.  Note that the Tera does not always see folder names in the order your computer sees them so a song that is named "03-name of song" may appear before a folder named 00004 on your computer but not on the Tera.
3-Having all of these blank folders in there does not seem to increase the time between tracks. The Tera seems to just keep very rapidly selecting until it finds a song, and it does not seem to take any longer than it used to between tracks (except maybe very occassionally, it will take a little longer to select a song).
4-The extra folders also do not seem to interfere with ordinary navigation. When you skip to the next artist folder or album folder using the navigation buttons, the Tera just skips to the next real folder. In other words, you never need to manually skip through 16 or 32 blank folders to get to the next one.
5- It speeds all of this up if you create 16 or 32 blank and properly numbered folders in a folder on your computer's desktop. Then you can copy and paste the blank folders as needed into each folder on your card. It would obviously  take a lot longer to manually create 16 or 32 new folders one at a time in each folder on your card, and since you may need to add the same large number of empty folders to lots of folders, copy and paste is definitely the way to go.
6-I used to think the first folder had to have a song in it for the Tera to start playing, but with empty folders instead of short silent tracks (which I tried first) as place holders, the Tera just skips ahead to the first folder with a song in it.
7- Avoid having real tracks mixed with real folders at any level, as this will mean those tracks will be selected much more often than the tracks in one of the folders. Best to stick with the Artist/Album/Track system of organization like that in iTunes so that all the actual WAV file songs are in an album folder, and to avoid having WAVs in your root folder or in an artist folder.
 
I will report that with my card setup according to these rules, I am now hearing lots of songs I have not heard in a very long time. I have been skipping through a lot of music to see if I can see any patterns, and so far I can't sense any pattern or emphasis on certain music. I also have not experienced any repeats of songs that I have already heard that same day or the day before which used to happen often (up to 7 or 8 times in a single listening session) before I figured this stuff out. It really seems now like I have a card with 2500 songs on it, instead of seeming like a card with 200-300 very familiar songs on it. Of course the old familiar songs still come up, but I would say they are maybe one in ten of the songs I hear which is about right. After all, they are part of the random selection process along with all of the music I used to hear less often.
 
Maybe we can call these the Rube Goldberg Rules for Randomness. Let me know how they work for you. And a big thank you to Suicidal_Orange. It was his spreadsheet that allowed me to figure out an approach that works without having to have an exactly even number of songs or albums in all folders, or other awkward limitations.

 

[Nirmalanow]

The above is based on how the Tera selects songs randomly. It uses a random number generator to come up with three 8-bit random numbers: one for the artist folder, one to select an album folder and one to select the particular track. The Tera can have up to 128 folders/tracks at each level: root, artist  and album, but the random number generator creates random numbers between 0 and 255. So there is an algorithm that first checks if there is a match. But if there is no match because the number is higher than the actual number of tracks/folders, then it drops one bit which is roughly equal to dividing by 2. Then it checks again for a match, but if there is no match it keeps dividing by 2 until a match is made.
 
So if there are 16 tracks and the Tera creates a random number of say 88, there is no match. So it tries 44...still no match. Then 22....still no match. Finally 11 and there is a match and track 11 plays. The problem is that this tends to always select the higher numbers before the lower numbers as they catch the process before the division gets to the lower numbers. So the low numbered tracks only play when there is a direct match, i.e. when the original number is a match. The higher number folders/tracks are therefore selected up to 63 times more often than lower numbered tracks (usually 15-31 times more often). You can see this by playing with the spreadsheet linked to in the previous post which shows the distribution of random selections when there are a specific number of folders/tracks (just enter different numbers in the black box to see how this affects the results).
 
The rules above compensate for this tendency to select higher numbers by inserting blank folders in front of the existing tracks and folders, so that all of the real tracks and folders are now in the higher numbers that get selected more often.
 
Here is how it works in real life using my own 128 GB card as an example:
 
I currently have 16 real artist folders along with 16 empty artist folders in the root folder of my card. This means that according to the spreadsheet, each real artist folder is selected 15 out of 256 times or about 5.9% of the time. The 16 blank folders are in positions 1-16 so according to the spreadsheet, they each get selected 1 out of 256 times or about .39% of the time per empty folder. So the empty folders are selected about 6.2% of the time in total (16 x .39%). If I had 10 real artist folders, they would still each get selected 5.9% of the time for a total of 59% of the time, and the empties would be selected the other 41% of the time. When an empty artist folder is selected, it does not matter what album or track is selected, since there are no albums or tracks in the empty artist folders. It seems the Tera just starts over with another set of three 8-bit random numbers when that happens.
 
Then I have an average of 11 real album folders (varies between 8 and 13) in each artist folder along with 16 empty album folders. Again each real album folder gets selected 5.9% of the time when its parent artist folder is selected. So with an average of 11 folders, a real album folder gets selected about 65% of the time and one of the empty folder gets selected the other 35% of the time.
 
Then each real album folder has 12-16 real tracks (average 14) along with 16 empty folders. Same thing happens when a particular album folder is selected, there is a 5.9% chance that each particular song would be selected also. And on average about 17% of the time one of the empty folders I have in with the tracks gets selected instead and the Tera starts over.
 
You put this together and each individual track of approx 2500 songs on the card has a 5.9% x 5.9% x 5.9% chance of being selected or about a .02% chance of playing. Every time the Tera selects, then every song on the card has a fresh .02% chance of playing, just like every time on the roulette wheel the red is about 50% odds even if it has come up 10 times in a row previously.
 
With 2500 songs, this means an actual song is selected on the card about 50% of the time (2500 x.02%). The other 50% of the time the Tera ends up in an empty folder and has to try again. But on each new selection, even after an empty folder has been selected, then each and every song once more has an equal .02% chance of being selected on the new try. Thus perfect randomness in the actual playback. And of course perfect randomness is not like shuffle play where every track plays once before any repeats. Random means random on every selection and so any song could actually come up twice in a row, although at .02% the odds are very low. I used to hear the same song twice in a row often on my Tera, and even more often twice in one listening session. Now with the empty folders added to my card, I have not had that happen even once in several days.

 

[Nirmalanow]

It crossed my mind that even if you just apply the first 6 rules and do not worry about the various levels all having the same range within all of the folders on that level, that you would still make a huge improvement to the randomness on the Tera (probably 90% or more of the possible improvement). So that could be the quick and dirty approach: Just add the correct number of empty folders to every one of the existing folders including the root folder of the card and enjoy it without worrying too much about the randomness being perfectly perfect

 
I added a note to this effect after rule 6 above.

 

[suicidal_orange]

Having done all of the above over three weeks ago I am now 100% sure that this works as intended - no more favourite artists let alone tracks as happens when you leave the Tera to it's own devices, instead you get perfect randomness.
 
As an added feature if you want to hear your new music more often you can put it directly into the "blank" folders straight on the card (with appropriate padding for the tracks as detailed above) and they will be played much more often than everything else.

 

[Nirmalanow]

Yes, it is still working great for me also. The difference is like night and day compared to before when certain songs would play much more often. Now I am getting the full benefit of a 128GB card.

 

[suicidal_orange]

My card got corrupted so I had to redo it, and now it's not so great - I'm experiencing long squeaky gaps between tracks much more often than before.  The tracks are still perfectly random but from thinking about it I've now got 12 "Albums" per "Artist" over 25% of the random selections will result in a no-play...
 
Now the question is do these empty "Artist" checks cause delay, or is it only when it hits a valid "Artist" and "Album" before finding an empty "Track"? 

 
If you've got any ideas how to test I'd love to hear them, in the mean time I will be fixing the spreadsheet to simulate it assuming any no-play takes time.

 

[Nirmalanow]

  My card got corrupted so I had to redo it, and now it's not so great - I'm experiencing long squeaky gaps between tracks much more often than before.  The tracks are still perfectly random but from thinking about it I've now got 12 "Albums" per "Artist" over 25% of the random selections will result in a no-play...
 
Now the question is do these empty "Artist" checks cause delay, or is it only when it hits a valid "Artist" and "Album" before finding an empty "Track"? 
 
 
If you've got any ideas how to test I'd love to hear them, in the mean time I will be fixing the spreadsheet to simulate it assuming any no-play takes time.

I do hear squeaky gaps occasionally but they are fairly quiet and rarely last longer than a fraction of a second. I have a lot of blank folders in my setup, and I would assume that any time the random number generator hits a blank folder anywhere it has to start over again. I wonder why you are experiencing longer gaps now. It should be possible to reduce the number of blank folders to the barest minimum by rethinking the arrangement some, but so far I have not given it any thought since the arrangement on my 128 GB card that I described above is working great for me.

Sorry to take so long to reply, but I am rarely on Headfi these days.

 

[suicidal_orange]

 
I do hear squeaky gaps occasionally but they are fairly quiet and rarely last longer than a fraction of a second. I have a lot of blank folders in my setup, and I would assume that any time the random number generator hits a blank folder anywhere it has to start over again. I wonder why you are experiencing longer gaps now. It should be possible to reduce the number of blank folders to the barest minimum by rethinking the arrangement some, but so far I have not given it any thought since the arrangement on my 128 GB card that I described above is working great for me.

Sorry to take so long to reply, but I am rarely on Headfi these days.

Have you had a play with the spreadsheet in the first post?  If you have more "artists" you get a higher chance of hitting an empty one (for 32 "artists" you have 32/256 compared to 2/256 if you only have 2 albums.  2 artists of 64+ albums (plus blanks at the end to pad to 128 (as few as possible)) results in minimal sqeak on a 128gb card but still a 3:1 chance of playing a later album... That's what I'm using at the moment.
 
As to the squeaks the volume varies with the sensitivity of your headphones and the length with the number of misses - pad your root to 128 and I'm sure you'll hear long ones

 

 

[Nirmalanow]

Yes, I only have 16 artist folders (they are really more like genre folders, but I call them artist because that is the way Charles refers to them) and so I added 16 blank folders to the root. With that ratio of blank folders to artist folders (16 to 16), a blank folder is only selected about 6% of the time. And as calculated above, when I hit a real artist folder, I have about a 65% chance of hitting a real album folder. When I hit a real album, I then have about an 83% chance of hitting a real song.
 
Add all of that up and it works out to a real song about 50% of the time (94% x 65% x 83%) and some kind of blank the other 50%, so roughly 50/50 like a coin toss. So about half of the time, I will hit a real song on the first try and there is no squeak. And about half of the time there is a slight pause as it has to try again. And just like it is possible although rare to flip a coin and have heads 5 or 10 times in a row, it is also possible for it to take several tries to hit a real song. That is when I hear a quiet squeaking sound as the Tera keeps trying until it finds a real song.
 
My Sony's have a sensitivity of 104db and I tend to listen fairly loudly so the volume is turned up between songs also, and I find the squeak to be very innocuous and quiet. I could see how it might be more annoying with other setups, but for me it is a very small price to pay for the true randomness I get with this system. I almost always listen in random mode, and it used to drive me a little crazy to hear the same damn songs over and over again on my 128GB card. I have not spent the time with the spreadsheet to see what is the best way to minimize the number of blank folders needed, but that is just because it is already working so well for me now.

I do think anytime it hits a blank at any level (artist,album or track) then the Tera just generates a whole new random number and tries again. So it should not matter really what level the blanks are at, and maybe having fewer artists with lots of albums in each artist just means having to add in more blank folders at the album level. Again I have not played with the numbers to see if there is an advantage one way or the other in the overall percentage of successful tries.

 

[suicidal_orange]

Haha, just noticed you made this thread, of course you've played with the spreadsheet!  Is it really that simple to combine percentages?  I had forgotten the basic maths 

 
The trouble seems to be that many of my albums have 9-13 tracks so (for 12) in addition to the 16/256 lost at the beginning to padding folders it also wastes another 60 for tracks 13-16 resulting in just 70% success (less with 9-11 tracks) when it does hit a valid "artist" and "album".  My longest squeak was nearly a minute!  How do you pad your albums to get 83% success?
 
To minimise the squeaks the best arrangement is to use 64+ "albums" in one "artist," padded to 128 at the beginning. This means no missed artists and minimal missed albums but does result in a 3:1 preference for the last 64 albums and makes it much harder to find a specific album if you want to.  As an alternative you can put the padding at the end so there's no delay when you switch it on but this will mean the later albums will be played noticeably more often (eg for 96 albums with the padding at the last 32 would be played 3x as often as the first 64 but if you pad the beginning 64 are played 3x as often as the first 32 so it's less noticeable) but as you get near 128 this become irrelevant and a good idea.  One I hadn't thought of until writing this out *corrects card*
 
If you don't want to do that or don't have a 64gb+ card there are so many things to consider that I fear my script will never be completed but the simple percentage combination will surely help.
 
 
 
I wonder what Charles' new algorithm looks like...

 

[Nirmalanow]

I have a kind of unique situation because I pretty much only ever listen in random mode. So I can simply put as many songs as I want in whatever arrangement of folders works for me. I am not constrained by the actual number of songs in any particular album as all of my songs get thrown in together. It is a long story how I ended up with this arrangement, but now I am used to it and it works for me. Anyone who likes to be able to navigate to a particular album or song would find it totally unusable.

Given this kind of setup, I could shoot for a perfect arrangement of 16 artists/8 albums per artist/16 songs per album for 2000 or so songs on a 128GB card. Then I would hit a real artist folder 93% of the time, a real album folder 97% of the times I hit a real artist, and a real song 93% of the time I hit a real album. This works out to a successful selection about 84% of the time (93% x 97% x 93%)
 
But I have closer to 2500 songs so it has not quite worked out that perfectly and as I mentioned above I have more like 8-13 albums in each of my 16 artist folders, and then 12-16 songs in each album. So the extra albums and odd amounts of songs means a lot more duds. But still I end up at a rough average of hitting a real song 50% of the time which I can live with. I have never had a full minute of squeaking. I would say the most has been 5-6 seconds and that is very extremely rare.
 
I am not sure why you would have such long squeaks with your setup as with just one artist folder it seems you would have a overall better chance than 50% of hitting a real song. Well lets see...with 64 albums in one artist, that is 75% and then with an average of  say 12 songs in each album that works out to 70%....so 75% x 70 %  still comes out about 50%. You should be having about the same "luck" as me.
 
You could pad each album of 9-13 songs with just 12 blanks. It would reduce the misses that cause squeaking a lot, but some songs would be about twice as likely to play. That would bring your odds of hitting a real song up to 96%. And then if you rearrange the 64 albums into 8 artists of 8 albums each, you would be looking at something very similar to my ideal above: 97% x 97% x 96% equals 90%.
 
Or even if you stick with 16 blanks in each album of 9-13 songs, but switch to 8 artists with 8 albums each, you would still do better: 97% x97% x 70% equals 65%.
 
Somewhere in the other Tera thread I shared what Charles was thinking of doing with the algorithm. I am not sure if he did use the one he told me about, but here it is again quoted from the other thread:
Charles is working on another algorithm. This one takes the random number and divides it by the number of tracks/files. Results of the division are discarded and then any remainder is used to select the track or folder. (i.e. with 22 tracks and a random number generated of 135: 135 divided by 22 is 6 with a remainder of 3. So track #3 would be selected.) Sounds to me like it should work much better, although there still might be a slight weighting. However, it would be many orders of magnitude less than the weighting of the current algorithm. For example with 100 folders, some would get 2 out of 256 selections and some would get 3 out of 256 selections, or with 20 folders, some would get 12 out of 256 and some would get 13 out of 256. Still pretty random especially compared to the current algorithm, and it would be perfectly random anytime it was selecting a folder/track out of a group that has 2,4,8,16,32,64 or 128 folders/tracks.

 

[suicidal_orange]

Ah I see... I'm only using random about half the time so want to keep the albums as they come.
 
The really long squeak was with 32 "artists" containing 7 or 8 albums each I think, needless to say didn't keep that setup for long!  With one "artist" there would very rarely be a squeak (I don't think you hear a single miss) and it's rare with my current setup of 2(+2) artists with around 120 albums in each.
 
That new algorithm sounds great but I'll probably never get to see/hear it in action.  Unless my jacks fail and Charles does a U-turn about needing the original invoice to transfer the warranty which I don't think he will.  That's what I get for not being willing to pay the original full price...