c.1200, bagge, from Old Norse baggi or a similar Scandinavian source; not found in other Germanic languages, perhaps ultimately of Celtic origin. Disparaging slang for "woman" dates from 1924 (though various specialized senses of this are much older). Meaning "person's area of interest or expertise" is 1964, from Black English slang, from jazz sense of "category," probably via notion of putting something in a bag.
To be left holding the bag (and presumably nothing else), "cheated, swindled" is attested by 1793. Many figurative senses are from the notion of the game bag (late 15c.) into which the product of the hunt was placed; e.g. the verb meaning "to kill game" (1814) and its colloquial extension to "catch, seize, steal" (1818). To let the cat out of the bag "reveal the secret" is from 1760.