[arnyk]

 
I would get the sound meter but be aware it could be off by a few dB and often are. However it should be consistently off in its measurements as long as you use the same meter on all the headphones they should end up the same. Two people with two meters and you could be way off. The down side of phone app's is keeping them calibrated in absolute SPL. They can be excellent if you have a calibrator, preamps and microphone. But that is starting at about $1k. A and C curves are both the same at 1KHz fast or slow should not matter for a sine wave but slow will average and might help be more consistent. 94dB would be a standard calibration level, but it is pretty loud might freak out people running their head phones that loud. You want to be louder then the room noise maybe 80dB is a good starting point.  

 
As is alludged to above there is a standard kind of audio tool that is called a microphone calibrator. They are basically a tiny sound system that puts out one or two known audio tonal sounds at known frequencies at one or two different SPL levels. Some put out only one tone at one level. The point is that they do this in a highly reliable, precise and repeatable way.
 
The electret microphones that are commonly used for audio measurements are generally pretty stable as to their frequency response and distortion, but their sensitivity can change spontaneously due to things like temperature and humidity.  Microphone calibrators thus are useful to keep track of those changes.
 
In modern times a measurement microphone, calibrator and mic preamp combo can be assembled for a few $100. 

 

[dprimary]

Arnyk I agree, though I have not seen any low cost calibrators. About $200 is the lowest I have run across, $400-500 is the typical cost. A calibrator is good sanity check anytime you do acoustical measurements. There has been few times over the years that I have setup to do measurement then check the calibration and realize I have something set wrong. 
 
 
For the $20 cost of the Dayton Mic, I am impressed that it looks like they really do calibrate every microphone. The calibration files look realistic when I picked a random serial number and looked at the file. 
 
Now if they came out with $50 calibrator.

 

[arnyk]

  Arnyk I agree, though I have not seen any low cost calibrators. About $200 is the lowest I have run across, $400-500 is the typical cost. A calibrator is good sanity check anytime you do acoustical measurements. There has been few times over the years that I have setup to do measurement then check the calibration and realize I have something set wrong. 
 
 
For the $20 cost of the Dayton Mic, I am impressed that it looks like they really do calibrate every microphone. The calibration files look realistic when I picked a random serial number and looked at the file. 
 
Now if they came out with $50 calibrator.

 
$50 seems elusive but $100 or so is as close as eBay:
 
http://www.ebay.com/itm/New-Digital-Sound-Level-Meter-Calibrator-94-114dB-Class-I-1-Fit-1-1-2-Mic-/291503919138
 

 
Recommended to me by someone with good technical chops who also has several others from the high priced brands.

 

[Speedskater]

If we are just matching, setup A to setup B, then accuracy and frequency response just don't matter.  As long as the SPL meter or voltmeter are capable of displaying a reasonable value (not one that is down in the mud) at the volume and frequency in question, then you are good to go. All we are looking for is a repeatable equal value from A to B.

 

[castleofargh]

  If we are just matching, setup A to setup B, then accuracy and frequency response just don't matter.  As long as the SPL meter or voltmeter are capable of displaying a reasonable value (not one that is down in the mud) at the volume and frequency in question, then you are good to go. All we are looking for is a repeatable equal value from A to B.

but what value should we take?
I tried going for 1khz, sometimes it really gives the wrong idea. it's fine for headphones with a relatively similar signatures though.
I tried matching whatever was the loudest, that pretty much never felt right.
 
I've been thinking about recording the sound coming out of both phones with a song, then use replay gain on the tracks and set whatever value it gives me. tried with a cheap headset microphone, but the thing just doesn't know what low frequency is, and that even after I macgyvered it all to pretend there was a good seal. so I don't know if it's a stupid idea or not.

 

[arnyk]

 
Here's what I think I don't know how to do:

1. Set the meter configuration (A, C, slow, fast, range, etc.)
2. What tones to use? (associated with meter configuration)
   1. Can I just use my audition music sample and capture peak?
3. How to move the volume back and forth accurately enough for me (basic testing shows I can tell 1db differences 100% of the time, but 0.5db about 65%)
   1. Simple visual marker like a post-it strip?
   2. Always have to measure when switching?
4. How to align the meter/microphone to the headphone cup?

 
I'm not looking to publish a paper or spend $100s, I'm looking to compare headphones to both train my ears further and make better purchasing decisions. Ideally I'd love to have something quick and portable that I could take to local meet-ups and vendor shows.

 
If you are comparing two components with the same frequency response then the frequency response of the meter being used or the weighting chosen is not highly important. You are just matching two like things. However, flat response meters can ease the work.  
 
If you are comparing two components with different frequency response then the frequency response of the meter being used or the weighting chosen is highly important. The meter response should be as flat as possible.  It may be practically impossible to sensibly match two components that have vastly different frequency response.
 
The test signal that is used needs to be as easy as possible to use to obtain consistent response. If you are doing acoustical measurements, then pure tones are usually poison because their amplitude will vary significantly for any number of relevant and irrelevant reasons. The usual signals used for acoustical measurements are broadband pink noise, pink noise limited to some fraction of an octave and with different center frequencies, multitones, or swept sine waves.
 
If you are comparing two components then having some kind of rapid switching facility which might be as simple as a DPDT or 4PDT switch, can be very helpful for both measuring or listening.
 
Unfortunately what you seem to want to do seems simple enough, but it trips over any number of potholes that have been out there for a long time and never really properly filled.
 
For example, when we devised ABX part of the plan was to restrict our comparisons to equipment that had identical or nearly identical frequency response. That made things a lot simpler!

 

[mikesale]

   
If you are comparing two components...

 
Might it be that you think I'm looking to compare components for measurements beyond SPL? I'm only looking to do what is viable and reasonable to do a subjective headphone comparison. 
 
Are we basically doomed to just deal with it when looking to upgrade?

 

[arnyk]

   
Might it be that you think I'm looking to compare components for measurements beyond SPL? I'm only looking to do what is viable and reasonable to do a subjective headphone comparison. 
 
Are we basically doomed to just deal with it when looking to upgrade?

 
You said it: " I'm only looking to do what is viable and reasonable to do a subjective headphone comparison."
 
What I'm saying is that this opens a big can of worms, like it or not.
 
(1) If you try to compare two different pairs of headphones, then even if they were identical except that had slightly different effiiciencies, they would always  sound different unless you tried to level match them.
 
(2) Level matching two sets of headphones whose frequency response is significantly different may make them sound less different, and probably make their FR differences more obvious. But they will probably still sound different. 
 
This discussion started out with an anecdote about comparing headphones that differed only in terms of their cables, and if the cables were even marginally technically competent, we are doing a comparison of two different things with the same efficiency and frequency response which is a whole 'nuther thing. The goalposts got moved, big time!

 

[ogodei]

 
I'm not looking to publish a paper or spend $100s, I'm looking to compare headphones to both train my ears further and make better purchasing decisions. Ideally I'd love to have something quick and portable that I could take to local meet-ups and vendor shows.

 

 

 

 
 
I think dprimary already summed this up.  SPL meter ($30-40) shoved through a CD blank ($0) held onto the meter by gaskets or a sheet of rubber glued to the CD ($3 in the plumbing section at Home Depot).   I like the Rat Shack meter because the mic is wide and short and it doesn't poke into the headphone driver (look at the last picture on that Neewer one).
 
Set meter to C weighting, slow response, play pink noise at whatever volume you want to listen to (listen to the pink noise in one of the cans at the meet to figure out how loud it needs to be). Center it as much as possible and let the pressure of the headphones hold the meter in place, or just set the meter face down onto an ear cup that is facing upwards. Holding it in place by hand sometimes gives you iffy results.
 
At a meet up those little post it note arrows might be your best bet, at home pieces of black tape are better.  Get markings for both cans and just go back and forth. 
 
This is quick and dirty of course but it gets you as close as possible in a meet environment.  You don't need a $500 meter or a calibrated mic, as you state you're not looking at publishing a paper of technical findings.
 
And just to pre-empt the comments I'm fully aware of the problems with rat shack meters. It's perfectly suitable for this purpose however.

 

[mikesale]

Thank you all for a fantastic learning experience!