I think the “more to it” might be significantly crazier than the timing thing.

Or ears have unique complex shapes that attenuate certain frequencies and bounce sound around in complex ways depending on the direction they are coming from. And our brains instantly process all that stuff too. It’s why our sense of hearing isn’t just on a flat plane around our head.