Double sided tape was brought up. Clarification: carpet tape. There are lots of kinds of double sided tape, but only carpet tape is strong enough to consider using it for mounting things.
Instead of superglue, use epoxy. It's better suited to the purpose. It's less brittle, and you can just slather it all over things to get a very strong mounting arrangement. It takes some time to cure, but once it does, it's very strong stuff.
As for permanently attaching things, I would only do that with the battery holders. The circuit you may want to repair or modify some day.
I prefer standoff or spacers for mounting boards to cases:
- Put short small (#4, #6) machine screws through the board at strategic points, put a threaded standoff on each one, and then dab epoxy on the bottom of each standoff, then set the assembly down in the case. Use a toothpick to dab a bit more epoxy around the edges of the standoffs and let them cure. When it's done, you should be able to remove the screws and lift the board off. Be sure to use screws short enough that they don't come too near the bottom of the standoff, so they don't get epoxy on their threads.
- Alternately, use long flat-head screws and spacers. The screws go in through the bottom of the case into a countersunk hole so the head is level with the bottom of the case. It goes up through a spacer and through the circuit board, and it's all held together with a nut. You can do something similar with two screws and a threaded standoff if you prefer. This is much stronger than the epoxy method. See the pic at the bottom of this page for an example:
http://tangentsoft.net/audio/mlxfeed.html
There are nylon spacers, aluminum spacers, steel threaded hex standoffs...lots of variety to choose from. You might pick up a Mouser catalog and go through the hardware section.