The French branch of the family launched legal action against the New York-based Solomon Guggenheim Foundation, which manages the collection.
The relatives are angry at the way the collection of paintings by artists including Picasso, Miro and Matisse are displayed and have called for it to be restored to its original configuration.
One of Peggy's grandsons, Sandro Rumney, and other family members complain that works from other collectors are now being displayed at the Palazzo, diluting the quality of the collection.
In the original hearing in May, lawyer Olivier Morice said the family felt moved to take the action "to respect the wishes of Peggy Guggenheim to see the collection intact".
But the Court of Appeal in Paris rejected the family members' case and ordered them to pay €30,000 in legal costs.
Peggy Guggenheim built up the collection with the enormous wealth she inherited at the age of 13 when her metal tycoon father Benjamin went down on the Titanic. She came to live in Paris in the 1920s, befriending many of the artists whose works are in her collection. She died in 1979 at the age of 81.
The Guggenheim Foundation hailed the court's decision, saying in statement it is "proud of having honoured the wishes of Peggy Guggenheim for more than 30 years, by keeping her collection intact in the restored Palazzo museum and by contributing to the knowledge of modern and contemporary art in Italy."