Five Star, led by Italian comedian Beppe Grillo, had the same day voted in favour of leaving the Eurosceptic bloc it is currently part of alongside Britain's Nigel Farage.
But after an outcry among members of the Alliance of Liberals and Democrats for Europe (ALDE), the group's leader Guy Verhofstadt said a deal for Five Star to join the liberals had fallen through.
"There are not enough guarantees to push forward a common agenda to reform Europe," said Verhofstadt, a former Belgian prime minister who is also the European Parliament's Brexit negotiator.
"There is insufficient common ground to proceed with the request of the Five Star Movement to join the ALDE Group. There remain fundamental differences on key European issues."
He said the two groups would "continue to work closely together" on issues such as the environment, transparency and the concept of direct democracy.
Anger over the proposed deal is likely to torpedo Verhofstadt's bid to become the new president of the parliament as it comes just a week after he said Europe was threatened by "nationalists and populists of all kinds".
The election to replace Germany's Martin Schulz as head of the parliament, the only one of the European Union's institutions that is elected by the public, takes place on January 17th.
Five Star party officials said 78.5 percent of members who voted in an online poll had backed a proposal for its MEPs to seek membership of ALDE, most of whose members are ardent enthusiasts for European integration.
Joining the liberals would have meant severing Five Star's alliance with the UK Independence Party (UKIP) in the Europe of Freedom and Direct Democracy (EFDD) group.
Grillo had forged an alliance with UKIP's then leader Farage after the 2014 elections to the pan-European assembly, but on Sunday he had argued that Britain's vote to leave the European Union had made UKIP redundant.
The U-turn is likely to be interpreted as a sign of Five Star watering down its own Euroscepticism as Italy heads for elections which must take place in the next 14 months.
Current polls suggest Grillo's party could emerge with the biggest share of the vote but its reluctance to forge alliances with other parties is an obstacle to it forming a government.
Grillo is a long-standing opponent of Italy's membership of the euro and has called for a referendum on withdrawal from the single currency.
But he does not question Italy's membership of the European Union and his younger lieutenants rarely address the euro issue.