Diplomatic sources said discussion at a late-running summit dinner became increasingly heated and "emotional" as Renzi accused his peers of looking after only their own interests.
"If that's your idea of Europe, you can keep it," Renzi told his 27 counterparts, Italian sources told AFP. "Either give us solidarity or don't waste our time," they quoted him as saying.
A European source said the EU's two top figures, European Council President Donald Tusk and European Commission chief Jean-Claude Juncker, had clashed at the meeting but the two rejected the suggestion.
"Don't believe those who are tweeting, leaking info. Whenever we have difficulties, we discuss these things between ourselves," Juncker told a press conference with Tusk early on Friday.
What really mattered instead was that EU leaders had approved the numbers of asylum seekers to be resettled as proposed by the Commission.
"I don't care if it is to be done voluntarily or not because the most important thing was that we agreed on the 60,000," hesaid.
Juncker is the driving force behind proposals for 40,000 Syrian and Eritrean asylum-seekers who have already arrived in Europe to be redistributed under a compulsory system.
Another 20,000 Syrians living in camps outside Europe will also be resettled across the 28-nation bloc.
Tusk, a former Polish premier whose country opposes the quota system, said "migrants with no legal right to enter the EU will be returned" and negotiations with countries of origin would be speeded up to take them.
Member states will in the meantime agree how many asylum seekers they are willing to take on a voluntary basis by mid-July, he added.
Renzi said the outcome of the meeting fell short of what Italy needed. "Relocation is a first step but for us this is not the solution to our problems."
More than 100,000 migrants have crossed the Mediterranean so far this year, most of them landing in Italy, Greece andMalta who say their EU partners should share more of the burden.
EU leaders agreed an action plan at an emergency summit in April after around 800 migrants died when their boat sank in one of the worst disasters of its kind in the Mediterranean.
Tusk said earlier on Thursday there was "no consensus" on quotas, meaning member states opposed to the idea should make realistic voluntary offers of refuge to deal with the upsurge in migrants.
"The voluntary scheme cannot be an excuse to do nothing. I can understand those who want this voluntary mechanism but they will only be credible if they give precise and significant pledges by the end of July at the latest," he said.
"Solidarity without sacrifice is pure hypocrisy. Now we don't need empty declarations on solidarity, only deeds and numbers," said Tusk.
Britain, Hungary and other newer east European member states have balked at quotas due to growing opposition to increased migration at home.
The migration row managed to overshadow both the Greek debt crisis and British Prime Minister David Cameron's push for EU reforms ahead of a referendum on membership of the bloc.
One official said a brief dinnertime speech that Cameron made to launch his plan at the summit "was very useful... it gave us time to rewrite the migration draft."