When the clock is going TICK TICK TICK, it can be hard to think clearly, because you’re anxious about the clock.
Math anxiety is well understood, and no small part of this comes from the pressure of timed tests. Ultimately, some people take tests faster than other people. I would hope that you want your tests to measure how much students have learned, not their ability to take tests under pressure. If this is the case, then everybody taking the test needs to feel that they have adequate time.