Rogers' first marriage was at age 17 to her dancing partner Jack Pepper (real name Edward Jackson Culpepper) on March 29, 1929. They divorced in 1931, having separated soon after the wedding. In 1934, she married actor Lew Ayres (1908–1996). They divorced seven years later.

In 1943, Rogers married her third husband, Jack Briggs, a Marine. Upon his return from World War II, Briggs showed no interest in continuing his incipient Hollywood career. They divorced in 1949. In 1953, she married Jacques Bergerac, a French actor 16 years her junior, whom she met on a trip to Paris. A lawyer in France, he came to Hollywood with her and became an actor. They divorced in 1957. Her fifth and final husband was director and producer William Marshall. They married in 1961, and divorced in 1971, after his bouts with alcohol, and the financial collapse of their joint film production company in Jamaica.