This correspondence paper focuses on the problem of exponential stability for neural networks with a time-varying delay. The relationship among the time-varying delay, its upper bound, and their difference is taken into account. As a result, an improved linear-matrix-inequality-based delay-dependent exponential stability criterion is obtained without ignoring any terms in the derivative of Lyapunov-Krasovskii functional. Two numerical examples are given to demonstrate its effectiveness.