After some quick research online, I found that the dirty connections on the tubes can be a big cause for noise. Based on the overall condition of the inside of the amp (picture below), I guess the tube connection might not be too good. As you can see in the picture, there are plenty of dirt everywhere inside the amp. So I removed the tubes to check the conditions on their pins...
As expected, there are plenty of oxidation built up on the pins of the tubes. So I quickly got to work and started cleaning the tube pins. I used a small piece of sand paper to clean the oxidation, and then clean the residue using some rubbing alcohol. Then I repeatedly insert and remove the tube to their sockets to clean the tube sockets.
With some elbow grease, all 4 tubes are cleaned, and I also did a thorough dusting of the inside of the amp. Now everything (on the outside at least) are in pristine condition, it's time to test the result... (accidentally gotten some rubbing alcohol on the tubes, and it wiped out the paint on the tubes)
I cannot believe how big a change this simple procedure brings! The noise level is greatly reduced. I can barely hear any noise when I plug in my VE Zen 2.0 and the noise is even lower with my HD600. I wouldn't call this "pitch black background", but this is a huge improvement, especially considering how little work that i did to achieve it.
Now I am very intrigued on other stuff that I can do to further improve its sound...