I'd imagine living in small spaces with a lot of neighbors adjoining your walls, above and below you, you wouldn't have the option to listen to speakers, let alone a large high end speaker system without others hearing it.
So limited space and crowded neighbors. Not a surprise at all why headphones are so popular in Japan.
I don't live in Japan, but I do live in a small condo in Southern California. So with many neighbors nearby and noise restrictions imposed by HOA rules, it's actually headphone listening heaven here. Sucks if you prefer speakers, but I prefer headphones.