The answer is that our way of keeping time assumes constants that aren't exactly constant. Earth's path around the sun isn't a circle, it's an ellipse. We orbit faster around the sun in winter when we're closest, and slowest when were furthest away in summer.
These little changes mean the sun is not at its highest point in the sky when we say it's noon -- it can be off a few minutes.
Check out this article below for even more details -- imagine.gsfc.nasa.gov