"engaging" hobby I love
HEHAEHAEHAEHHAHEAHEHAEHEHA
Oh sorry.
It's expensive, you'll feel brilliant one day and competely stupid the next, it'll take up all of your time not just your "free" time, did I mention it's expensive?
There's two ways - there's building and there's understanding. The former is connect-the-dots, paint-by-numbers. you don't have to understnd a schematic besides what symbol represents what part. With that and a parts list, you're done! The latter - well, as you say that involves classes, expensive books, more time and even more money (mo money mo money mo money mo money mo money...)
If you're still amazed at how energy flows from the magical socket where everything plugs in, start with a novel like Amdhal's "There Are No Electrons" - it's funny as hell. Extremely helpful in understanding the magic, though.
Then, grab a copy of Horowitz and Hill's "The Art of Electronics"... it's under $100 but barely under 100 pounds... as books go it's more a collection of ramblings than sequential learning, but it's very comprehensive.
In a few weeks/months you'll know how much time effort and money you want to invest in this "hobby"... and we'll go from there.
Also, talk to your math professor about electronics classes to take... what? what do you mean, you don't like math!?!?