"crazy," 1846, from earlier be nutts upon "be very fond of" (1785), which is possibly from nuts (plural noun) "any source of pleasure" (1610s), from nut (q.v.). Sense influenced probably by metaphoric application of nut to "head" (1846, e.g. to be off one's nut "be insane," 1860). Nuts as a derisive retort is attested from 1931.
Connection with the slang "testicle" sense has tended to nudge it toward taboo. "On the N.B.C. network, it is forbidden to call any character a nut; you have to call him a screwball." ["New Yorker," Dec. 23, 1950] "Please eliminate the expression 'nuts to you' from Egbert's speech." [Request from the Hays Office regarding the script of "The Bank Dick," 1940] This desire for avoidance accounts for the euphemism nerts (c.1925).