It could be possible, the manual for the BHA-1 Manual states that the 6W @ 32 ohms (or 3.8 into 50) is max power (i.e. peak). It states "nominal" power into 50 ohms to be 500mW. Technically, the gain of the BHA-1 should be enough depending on how loud you typically listen.
Then it just depends on if the amp can dish out enough current for the load depending on the gain level while maintaining acceptable distortion levels. My logic could be flawed though.