What Is the LMS Algorithm?

The Least Mean Square (LMS) Algorithm was a contribution to filter theory advanced in a 1960 article by Bernard Widrow and Marcian E. Hoff, concerning the production of the least mean squares of an error signal.

Filter Theory

Filter theory is an area of applied mathematics that concerns the reception of signals. To understand the foundation of this theory, think of getting static on a radio, then processing the signal so that you can only hear the song, not the static.

Performance Feedback

The Widrow-Hoff paper described a new system for signal controls in anthropomorphic language, by noting, "The designer 'trains' the system to give the correct responses by 'showing' it examples of inputs and respective desired outputs. The more examples 'seen,' the better is the system performance."

Gradient Descent

The LMS is an example of what are known as "gradient descent algorithms," in which some undesired quantity---noise, in this case---is reduced by repeated steps, toward either a local or (preferably) a global minimum.

References

• "Adaptive Filter Theory"; Simon Haykin; 1996.
• Information Systems Laboratory, Stanford: Adaptive Switching Circuits
• "Neural Network Learning and Expert Systems"; Stephen I. Gallant; 1993