Gregory Kusnick said,
Going back to the 1960s, text editors such as TECO used regular alphabetic keys to invoke editing commands: S for search, D for delete, etc. The program was in this so-called command mode by default; to type in text, you had to enter insert mode by pressing the I key. Now your keystrokes would be interpreted as text input rather than as commands until you exited insert mode by pressing the Escape key.
By the late 1970s this so-called “modal” text-entry method had come to be regarded as bad UI design, and modeless interfaces came into vogue with the introduction of graphical displays and pointing devices such as mice. In modeless designs, alphabetic keystrokes are (almost) always interpreted as text input, and editing commands are invoked via on-screen menus or key combinations such as Ctrl+X.
As KWillets and MattF note, popup dialog boxes that force user interaction brought back the notion of UI modes in which normal text input is disabled, and alphabetic keystrokes may be reinterpreted as command shortcuts (“Press Y for Yes or N for No”). So these window types became known as modal windows in the UI design jargon of the 1980s.