Old English *steortian, *stiertan, Kentish variants of styrtan "to leap up" (related to starian "to stare"), from Proto-Germanic *sturtjan- (cf. Old Frisian stirta "to fall, tumble," Middle Dutch sterten, Dutch storten "to rush, fall," Old High German sturzen, German stürzen "to hurl, throw, plunge"), of unknown origin.
From "move or spring suddenly," sense evolved by late 14c. to "awaken suddenly, flinch or recoil in alarm," and 1660s to "cause to begin acting or operating." Meaning "begin to move, leave, depart" is from 1821. The connection is probably from sporting senses ("to force an animal from its lair," late 14c.).
Related: Started; starting. To start something "cause trouble" is 1917, American English colloquial. Starting block first recorded 1937.