Speaking from Palazzo Chigi at 00:30, the premier said he took "full responsibility" for the defeat.
"My government ends here," Renzi told reporters, confirming a promise he made early on in the campaign and which he later admitted had been a mistake, that he would resign if defeated at the polls.
The PM said he would officially hand in his resignation on Monday after the "extraordinarily clear" victory for the No camp. "I wanted to reduce the number of seats," said Renzi, referring to the reform which would have seen the number of senators cut significantly. "In the end, the seat which is going is mine."
Polls for national broadcaster Rai and the La7 television channel both gave the No camp as winners by a margin of at least 54 to 46 percent in Sunday's referendum.
One poll, from the IPR Marketing Institute for Rai, put No at between 57 and 61 percent.
The higher percentage came from the first poll based on actual vote count, suggesting that Renzi may be in for an even heavier defeat than predicted in initial exit polls. These had given the No camp a lead of between six and 14 points.
Turnout was over 69 percent in the vote on constitutional changes which over the course of the campaign had morphed into a referendum on Renzi's administration and the prime minister himself.
The projected result was in line with what opinion polls had been indicating up until November 18th, after which the media were banned from publishing survey results. However, these polls showed up to a third of Italian voters undecided, and Renzi had said he was confident that a "silent majority" would back his reforms.
During the day his hopes were raised as turnout in the prosperous north of the country far exceeded that in the south - a pattern which was seen as a potential boost to the premier's survival hopes.
But it appeared in the end that voters on both sides were equally energized, leaving the balance of forces much as it had been at the start of the campaign.
About an hour after polls closed, Renzi sent out a cheery tweet, complete with smiley face, saying: "Anyway, thank you everyone. In a few minutes I'll be live from Palazzo Chigi. Long live Italy! PS. I'm on my way, I'm on my way."
In the press conference, though visibly emotional, he said: "Anyone who fights for an idea cannot lose."
'Victory of the people'
Opposition parties denounced the proposed amendments to the 68-year-old constitution as dangerous for democracy because they would have removed important checks and balances on executive power.
Spearheaded by the populist Five Star Movement, the biggest rival to Renzi's Democratic party, the "No" campaign also capitalized on Renzi's declining popularity, a sluggish economy and the problems caused by the arrival of tens of thousands of migrants in Italy from Africa.
The leader of Italy's far right Northern League, Matteo Salvini, framed the vote as a "victory of the people".
Salvini said Renzi should resign immediately and called for early elections; Renato Brunetta of Berlusconi's Forza Italia party had echoed the call for Renzi's resignation on Sunday night, but did not call for snap elections.
French far-right leader Marine Le Pen congratulated "our friend" Salvini on the result in a Tweet, while Salvini was clearly delighted by the result, posting: "Long live Trump, long live Putin, long live Le Pen, long live the Northern League!"
However, the No campaign was very much a cross-party effort, including several members of Renzi's own Democratic Party as well as all the major opposition parties. One MP from the No camp of the PD, Roberto Speranza, described the victory as "a beautiful page of democratic participation" and said: "we were right to defend our beliefs."
The PD's deputy secretary, Lorenzo Guerini, said the party's leadership would likely meet on Tuesday to "evaluate the outcome of the vote" and to decide the next steps.
Five Star Movement leader Beppe Grillo was slower to make a statement on the result, but published a blog post with the title 'Democracy has won', in which he declared "the times have changed" and called for elections to take place as soon as possible.
In the run-up to the vote, there was much debate about whether Renzi would stay on in the event of a loss.
It was feared that his resignation could lead to political and financial instability, with some No campaigners, including former prime minister and economist Mario Monti, arguing that the premier should stay on.
So what happens now?
Renzi said he would be visiting President Sergio Mattarella on Monday to hand in his resignation following a final meeting of his cabinet. Mattarella will then be charged with brokering the appointment of a new government or, if he can't do that, ordering early elections.
Most analysts expect Renzi's administration to be replaced by a caretaker one dominated by his Democratic Party which will carry on until an election due to take place by the spring of 2018.
Finance Minister Pier Carlo Padoan is the favourite to succeed Renzi as the President of the Council of Ministers, as Italy's premier is formally titled.