This paper deals with the problem of exponential stability for a class of discrete-time recurrent neural networks with time-varying delay by employing an improved free-weighting matrix approach. The relationship among the time-varying delay, its upper bound and their difference is taken into account. As a result, a new and less conservative delay-dependent stability criterion is obtained without ignoring any useful terms on the difference of a Lyapunov function, which is expressed in terms of linear matrix inequalities. Finally, numerical examples are given to demonstrate the effectiveness of the proposed techniques.