You can build your own headphone volume control box and install as many good quality potentiometers as you need.
You should use a minimum of 1 watt power handling capacity, and the left and right channel pots must track each other if using a ganged pot. This eliminates cheap pots. A standard resustive track pot will change its impedance from minimum to maximum volume. Maximum impedance it at the ELECTRICAL halfway point (not halfway rotational point; you need an audio taper pot), and will be 1/4 of the impedance measured across the entire pot resistance. Impedance at full volume will be zero, but impedance to the amp will be headphone in parallel with the pot. Look up the Thevenin Equivalent formula, it is a simple mathematical formula, and you can figure impedance at various settings.
A 100 ohm pot at full volume in parallel with a 250 ohm headphone will show 71.4 ohms to your amp. Your headphone will see the impedance of the amplifier. At half volume the amp will see 91.7 ohms of impedance. The headphones will see 25 ohms + the amp impedance. This is the worst case (highest impedance) the headphones will be driven by, about a 10:1 damping factor, fully adequate. At zero volume the amp will see 100 ohms, the actual impedance of the pot alone, and the headphones will see 0 ohms as it is at electrical circuit ground.
You can keep the impedances the headphone sees at all volume levels by using a stepped attenuator with properly chosen resistance values in both legs. This is quite expensive to do it right and keep sound quality very high, so it might be less expensive to just use separate amplifiers instead. My Conrad-Johnson preamp volume control uses such a setup but takes it a step further by using computer control to pick 2 resistors from the resistor bank and then connect them properly through hermetically sealed gold plated contacts. That is probably beyond your ability to construct such a device, but it accomplishes a 100-step pot with far less number of expensive components and eliminates dirty pot contacts over time.
You can use a rather high quality pot of high impedance then connect the output to a buffer amp chip, which will require a DC power supply, which can be supplied to a Radio Shack battery holder and several D-cells, or get a high quality DC wall wart power supply.
If making your own gear, you can switch to high quality XLR connectors and wire your phones in balanced mode, etc. FWIW, Conrad-Johnson says that floating single ended circuits do much the same as balanced circuits, but use half the components and also eliminates matching of components on both sides of the circuit. That is why an isolated battery operated amp can do you just as well as a much more expensive balanced amp. Also, wire your interconnects in the configuration Conrad-Johnson suggests in their owners manuals. That is to connect the shield at ONLY one end and connect that end to a common point, normally the preamp in a home system. I was wiring my interconnects that way a long time before I owned my first C-J piece of gear, and it works, and common Mogami or Canare studio interconnect cable may just outperform very expensive audiophile interconnects if you give it a try.
Give simple pots in a box a try first, and use decent quality 1/4" phone jacks. You can use switched jacks, and when a headphone is pulled out of circuit it will connect a resistor of the same impedance as your headphones across the circuits which will eliminate volume changes.
The load your amp will see will be the impedance of the headphone + pot combo, divided by the number of circuits. That is 91.7/2 for the above example, or 45.85 ohms, at half volume for both cirrcuits. Much higher than a single 32-ohm headphone. Unless using very low impedance pots and headphones, any good headamp will drive a couple of headphones and a couple of pots.
