Hm...I think that the speaker/headphone/earphone route would probably require a bit of mechanical engineering expertise, although you would still need a strong electrical background (and possibly electromagnetic fields+waves specifically). On the DAP side, that would probably require a lot of embedded software, so I wouldn't call that "electrical engineering". My university would have called it "Computer Engineering". That program included a lot more software development coursework.
If you want to go the analog route like in the realm of amplifiers, that is definitely electrical engineering (at its finest
).
One last note, the audiphile electronics industry is relatively small, and I am sure that there are plenty of engineers just here on Head-Fi who would love working in the field (I know I would), so if you really want to work in the industry, do two things:
1) Start looking for internships and special programs as soon as possible to shape your qualifications.
2) Whatever you do focus on, make sure that it is still general enough to find a job outside of the audiphile electronics industry if necessary.
Good luck!