late 14c., "physically weak, enfeebled, crippled," from Old French impotent "powerless, weak, incapable," from Latin imponentem (nominative impotens) "lacking control, powerless," from assimilated form of in- "not, opposite of" (see in- (1)) + ponentem (nominative potens) "potent" (see potent).
Meaning "having no power to accomplish anything" is from mid-15c.; that of "completely lacking in sexual power" (of males) is from mid-15c. Middle English also had a native term for this: Cunt-beaten (mid-15c.). The figurative sense in Latin was "without self-control, headstrong, violent." Related: Impotently.