expression meant to startle, early 15c., boh, "A combination of consonant and vowel especially fitted to produce a loud and startling sound" [OED, which compares Latin boare, Greek boaein "to cry aloud, roar, shout."]; as an expression of disapproval, 1801 (n.), 1816 (v.); hence, the verb meaning "shower someone with boos" (1893).
Booing was common late 19c. among London theater audiences and at British political events; In Italy, Parma opera-goers were notorious boo-birds, but the custom seems to have been little-known in America till c.1910.
To say boo "open one's mouth, speak," originally was to say boo to a goose.