1530s, "a boisterous, brutal fellow, one ready to commit any crime," from Middle French rufian "a pimp" (15c.), from Italian ruffiano "a pander, pimp," of uncertain origin, perhaps from a Germanic source related to rough (adj.), but Dutch roffiaan, German Ruffian are said to be from French. English meaning might have been influenced by similarity of sound to rough. Related: Ruffianly.
The Romanic words (e.g. Medieval Latin ruffianus, Provençal rufian, Catalan rufia, Spanish rufian) preserve the sense of "protector or owner of whores." For sense evolution in English, cf. bully (n.).