Frankly, I'm not sure where Sony really takes over the signal chain. A lot of the system messages the first time you boot it up are about the "aging cycle" (burn in) required for the fancy capacitors inside to reach their true audio potential. I would say that the balanced output is excellent, but there's nothing really wrong with the SE. I know the higher-grade model of the player uses specialty cables for the balanced connection internally, but I have a very hard time believing that 2" of higher grade cable will have a life-changing impact on sound. I use Apple Music in "Lossless" most of the time- I don't actually have a ton of FLAC stuff on my Walkman. It's just much more convenient that way, since I use an iPad as my primary source at my desk and I can keep an organized library between the various devices easily, plus stream to the home vintage HiFi or car stereo from any device easily.
I haven't used my A8000 balanced, so I can't speak to what it gains necessarily from the 4.4 jack. There's definitely much higher gain on the balanced, but I don't think the A8000 needs more than what the 3.5 SE output generates to sound really great. Flipside, that extra gain really pushes the D8000 the extra mile.
Ultimately, you could see the 4.4 jack as a "future upgrade" to push them further, and get a balanced connection down the road. For me personally, I like to use my A8000 with my Switch and Steamdeck occasionally, so the extra power and dynamic range I get from a balanced connection on my D8000 aren't necessarily worth the convenience of having the A8000 as single ended.