The bloc's 28 leaders held the final summit of a year that opened with the threat of a Greek euro exit and ended with the Paris attacks, a historic refugee crisis as well as Britain facing a possible "Brexit".
But their pronouncements of unity on key issues were undermined by splits between Merkel, the powerful leader of Europe's biggest economy, and Renzi over migrants and a controversial gas pipeline to Russia.
"The friendship and respect that I have for the German chancellor don't stop me from asking questions," a combative Renzi told reporters at the end of the two-day meeting.
A small group of states led by the Italian premier severely criticised the German Nordstream pipeline project with Russia, accusing Berlin of securing its own energy supplies while pressuring other countries to back economic sanctions against Moscow over the crisis in Ukraine.
EU nations eventually agreed to extend the sanctions by six months on Friday, sources told AFP, despite a push by Italy earlier this week to delay the rollover, amid reports that Rome was frustrated with what it saw as a hypocritical stance by Berlin.
"It was an emotional discussion," said EU President Donald Tusk, a former Polish prime minister who has been tough on Russia since taking office just over one year ago.
The stoical centre-right Merkel however sought to play down the divisions with centre-left leader Renzi, saying: "It is not the first time in my experience that we've had differences."
The row overshadowed a statement by EU leaders on Friday vowing an "uncompromising" fight against terrorism following last month's Paris attacks claimed by the Islamic State group, which killed 130 people.
"The Paris terrorist attacks of November 13th 2015 have only strengthened our resolve to continue our uncompromising fight against terrorism and to make full use of all the tools at our disposal," it said.
It said several EU measures that have stuck in a legislative pipeline, including a plan to track airline passenger names in close coordination with the United States, should be pushed through.
Meanwhile British Prime Minister David Cameron said there had been "good progress" towards an EU reform deal that he hopes to seal by the next summit in February.
On Thursday he urged his counterparts to work with him on his demands including a controversial four-year limit before EU migrants to Britain can claim welfare payments.
On Friday, Cameron gave his strongest hint yet that the referendum he has promised on a possible "Brexit" from the EU could be held in 2016. He had previously only vowed that it would be sometime before the end of 2017.
"I believe that 2016 will be the year we achieve something really vital, fundamentally changing the UK's relationship with the EU and finally addressing the concerns of the British people about our membership," Cameron told a press conference.
The British dilemma stole the limelight from fresh efforts to tackle the migration crisis, which has seen nearly a million people flood to Europe's shores, many of them fleeing the Syrian civil war.
Tusk said on Thursday leaders agreed to push through by June 2016 a plan for a new border and coastguard force that could intervene in member countries - even without their consent -- in order to shore up the EU external frontier.
He said they also agreed to protect the Schengen area, the cherished European passport-free zone that has been threatened by the huge movements of people across the continent.
Wide rifts emerged over Merkel's decision this summer to open Germany's doors to Syrian refugees, prompting several overstretched transit countries to suspend the Schengen rules and reintroduce border checks.
The tide of crises has led to fears of what lies ahead for the bloc, formed in the ashes of World War II in a bid to unite the continent.
"I am under no illusions," said European Commission head Jean-Claude Juncker when asked about his hopes for a smoother 2016.