asio4all is your friend. Get it set up and running right and it should eliminate the remaining latency dropouts if they are driver related.
The issues you had are just the kind of thing that would have me launching the device across the room. Out.