Renzi, who has vowed to step aside if he loses, was counting on a last-minute turnaround in voter sentiment to win backing for his proposals to streamline parliament and centralise some regional powers in the name of more effective and stable government.
Turnout was very high by Italian standards, with 57.24 percent of voters having cast their ballots by 7 p.m. (1800 GMT), according to the Interior Ministry.
Nearly two thirds of the electorate had voted in some parts of prosperous northern Italy but the turnout was much lower in the south -- a pattern which was seen as a potential boost to the premier's survival hopes.
Renzi was in jovial mood as he cast his vote in his native Tuscany.
"Have you decided how you are going to vote, prime minister?" a female voter cheekily asked outside a voting station in his home town of Pontassieve, near Florence.
"Now I'm thinking about it!" Renzi, 41, quipped back before spending ten minutes queuing to register his vote.
Opposition parties have denounced the proposed amendments to the 68-year-old constitution as dangerous for democracy because they remove important checks and balances on executive power.
Spearheaded by the populist Five Star Movement, the biggest rival to Renzi's Democratic party, the "No" campaign has also sought to capitalize on Renzi's declining popularity, a sluggish economy and the problems caused by tens of thousands of migrants arriving in Italy from Africa.
"God willing it's over. A new era starts tomorrow I hope," said Matteo Salvini, leader of the far-right Northern League, after voting in Milan.
Five Star leader Beppe Grillo, who had urged Italians to follow their gut instincts, used his voting appearance to joke about some voters' claims that the pencils being used to vote meant ticks on the ballot papers could easily be rubbed out.
"Us oldies knew what to do in this digital age. I sucked the pencil and it was working," the comedian told reporters in Genoa.
Outside a polling station in Rome, business owner Raffaele Pasquini, 37, told AFPTV he was voting "Yes" in the interest of his two-year-old son. "We are voting to try and change a country that has been stalled for far too long."
Turnout in Rome may have been affected by the vote coinciding with the local football derby match.
Matteo Rossi, 25, voted "Yes" before heading to the Stadio Olimpico to watch his side, Roma, beat arch rivals Lazio 2-0.
"I hesitated but when you look at the people who are for 'No', I couldn't be with them," he told AFP.
Fellow fan Guido wasn't planning to vote. "For me, the important thing today is what happens inside this stadium."
Polls close at 11 p.m. (2200 GMT) with a reliable projection of the result not expected until the early hours of Monday.
If Renzi goes, some short-term market turbulence looks inevitable. Some analysts fear a deeper crisis of investor confidence that could derail a rescue scheme for Italy's most indebted banks, triggering a wider financial crisis across the eurozone.
If he wins, Italy's youngest ever prime minister could be emboldened to accelerate promised reforms in areas such as public administration, the judicial system and education.
"If we miss this chance it won't come back for 20 years," Renzi warned voters before campaigning was suspended at midnight on Friday.
Up until polls were banned on November 18th, the "No" camp was leading comfortably - but with a quarter of the electorate undecided.
After the Brexit vote and Donald Trump's victory in the US presidential election, populism has been a factor, and Grillo's Five Star would see a "No" vote as its stepping stone to government.
But the campaign has also caused many voters to reconsider the merits of a much-loved constitution, crafted in the aftermath of World War II and the bitter experience of fascist dictator Benito Mussolini's disastrous rule.
Under the proposed reform, the Senate, currently a body of 315 directly-elected and five lifetime lawmakers, would have only 100 members, mostly nominated by the regions.
The chamber would also be stripped of most of its powers to block and revise legislation, and to unseat governments.
Other envisioned changes involve transferring some regional powers to the national government, making it easier to get major infrastructural works approved, and abolishing a costly policy agency in Rome.
By Angus MacKinnon