c.1300, "incantation, magic charm," from Old French charme (12c.) "magic charm, magic, spell; incantation, song, lamentation," from Latin carmen "song, verse, enchantment, religious formula," from canere "to sing" (see chant (v.)), with dissimilation of -n- to -r- before -m- in intermediate form *canmen (for a similar evolution, see Latin germen "germ," from *genmen). The notion is of chanting or reciting verses of magical power.
Sense of "pleasing quality" evolved 17c. Meaning "small trinket fastened to a watch-chain, etc." first recorded 1865. Quantum physics sense is from 1964. To work like a charm (figuratively) is recorded by 1824.