In that case, yeah, I'd hold off running batches until I took a break from work to check email or eat lunch. It would have less to do with audio quality than with having the conversions slow down whatever you're working on in the foreground. (Edit:) In any event, there's no real point to quitting apps to run the converter. A well-behaved idle application doesn't consume significantly more in resources than no app at all.
For audio file transcoding, I use Max, which allows me to define how many files to encode in parallel. Dialing that down to one or two makes the app reasonably non-disruptive when I'm doing my own large-format image processing in Photoshop. If I'm not doing anything, I leave it at 4.
I'm not familiar with XLD. If it has similar settings, try experimenting with them.