"rose garden," mid-15c., from Latin rosarium "rose garden," in Medieval Latin also "garland; string of beads; series of prayers," from noun use of neuter of rosarius "of roses," from rosa "rose" (see rose (n.1)).
The sense of "series of prayers" is 1540s, from Middle French rosaire, a figurative use of the word meaning "rose garden," on the notion of a "garden" of prayers. This probably embodies the medieval conceit of comparing collections to bouquets (cf. anthology and Medieval Latin hortulus animae "prayerbook," literally "little garden of the soul"). Sense transferred 1590s to the strings of beads used as a memory aid in reciting the rosary.