Damping factor will be a variable. It tends to not be an issue with BAs but some dynamics can sound slow, soft and woolly from added resistance, even if they lose some bass quantity. It's why I'd rather deal with it after the fact on a lower impedance amp. Actually I prefer to not deal with it at all with so much to choose from today.
The other side of the story is that there are some decent devices out there with output impedances at 3 to 4 ohms and people jump all over them due to an expanded graph into difficult load that will make a 1 to 2 db variance look mountainous. Lower is always better in my book but not until the device itself is better. Like all things, it's a balancing act but all things being equal, I'll go low every time.
All that said, I'd like to see a graph of an x10 with a lot of impedance in front of it. Impedance vs response curve looks made for it.