I wouldn't limit my choices to those two notetaking applications. Here's my advice:
First, the advice about paper-based notes for courses with lots of equations is solid. Don't fall for the gee-whiz factor of TabletPCs... they're not practical for notetaking at the speed you need to keep up in science courses (or other courses). If you really want digital versions of your handwritten notes in science classes, get one of the smart pens (Logitech, Anoto, etc.) where you write on real paper and then upload what you wrote to your computer later. In general, if you're even a half decent typist, it's far more practical to type than to use a TabletPC. There are a couple people in my law school with convertible TabletPCs, and all of them type for notetaking. It's the only way you can keep up. I don't recommend those machines at all.
The main difference between OneNote and other apps, including Word for Mac's notebook mode, is that it's unstructured. The entire metaphor is based on putting boxes in arbitrary places on a page, then filling them with content. This has its advantages and disadvantages. The main advantage is freeform content; you can arrange things however you like on the page. However, this comes at a cost. It is not as convenient to go back later in OneNote and do significant editing of your notes, because it's based on a boxes model, not a flowing text model. If you intend to do a lot of editing later, or if you do not have a need for graphically arranging your text and media on the page, OneNote is not the best choice. Also, because the OneNote model is not based on flowing text, data exchange with Word is not perfect (apparently this is much improved in ON 2007 however).
Obviously, if you intend to transfer data to Word for "prettying up", Word for Mac's notebook mode is the cleanest, because you never leave Word. It is good for structured and general-purpose text notetaking.
However, if you're on the Mac, there are also other good options you should consider. Circus Ponies Notebook and Aquaminds Notetaker are like a more structured version of OneNote. You can't put boxes anywhere, but that's by design. It makes shifting your notes around later much easier and the outlining functions are much more convenient and robust. Also, these two programs are very mature, dating back to the NeXT era.
Finally, if you want a pure, fully-featured outliner for notetaking, look at OmniOutliner. This is currently the most powerful outliner for any platform currently being sold (the closest thing for Windows is the rather poor and expensive NoteMap). It has some media functions, but the emphasis is outlining, moving words around, structuring your notes. This is what I personally use. However, I like to take notes in point form, so this fits my style. If you prefer to take notes in a more lengthy, paragraph-based form, I would recommend Word's notebook mode, but I would suggest looking at Notebook and Notetaker too. I do not recommend OneNote for class notes, but it makes sense for research notes if you're in grad school or if you have to make a lot of diagrams.
