Following the birth of his son Sean in 1975, Lennon had put his career on hold to raise him. After five years of little musical activity aside from recording the occasional demo in his apartment in New York, Lennon felt ready to resume work. He was quoted as saying that when making the album, his ambition was to “do something as good as “Heroes””, the 1977 album by David Bowie.

In the summer of 1980, Lennon made a sailing trip through treacherous waters from Newport, Rhode Island, to Bermuda. Almost losing his life on the journey, he began to write new songs, occasionally reworking the earlier demos.

Ono also wrote many songs, inspired with new confidence after Lennon had stated that he believed that contemporary popular music such as the B-52's “Rock Lobster” bore similarities to Ono's earlier work.

The couple decided to release their work on the same album, the first time they had done so since 1972's politically charged Some Time in New York City. In stark contrast to that album, Double Fantasy (subtitled A Heart Play) was a collection of songs wherein husband and wife would conduct a musical dialogue. The album took its title from a species of freesia, seen in the Bermuda Botanical Gardens, whose name Lennon regarded as a perfect description of his marriage to Ono.