Given that perceived sound quality is so inextricably linked with changes in loudness, and we are talking fractions of a dB, it would be amazing if some psycho-acoustics were not at play in your determination.
Specifically, adjusting volume controls ‘by ear’ is nowhere near accurate enough to level match. Therefore, if you are manually adjusting volume up and down to compare this with the volume levelled alternative, I doubt this would be accurate enough.