Pitting environmentalists against the government and big industry, the referendum was seen a key challenge for Renzi's centre-left Democratic Party (PD) after two years in office during which he has pushed through a string of pro-business and political reforms.
About 90 minutes after polls closed at 11:00 pm (2100 GMT), the interior ministry said near-complete figures showed turnout stood at around 32 percent.
Under Italy's referendum law, a referendum is only valid if more than 50 percent of the country's nearly 47 million registered electors cast their vote.
The result will be savoured by Renzi, who had openly called on Italians to shun the ballot in the hopes of sinking the referendum, a stance that frustrated the opposition and deepened a rift within his own camp.
The referendum centred around whether Italians wanted to repeal a new law about drilling near the country's coast.
The current legislation, passed in January, says existing concessions within 12 miles (19 kilometres) of the coast should remain valid until the fields are depleted - to the dismay of campaigners for renewable energy.
Environmentalists claim platforms near the shore present risks to health and protected habitats.
They argued that reversing the law would send a clear signal the country wants to go green and stop what they call "dirty deals" that benefit oil companies, in the wake of a recent scandal that saw a top minister resign over alleged favours to French oil giant Total.
But Renzi dismissed the referendum as "a hoax" this week and said a win for the "Yes" camp - backed by environmentalists and opposition parties - would lead to rig closures and massive job losses.
The referendum also fuelled a bitter internal battle within Renzi's party, with some PD members saying it was unacceptable for the premier to be the head of the pro-abstention campaign.
"Those who wanted to settle accounts at all costs have lost. The winners are the 11,000 people whose jobs were at risk," Renzi said in a speech after the polls closed.
He added that he was ready to continue the dialogue about the country's energy policies and that he wanted Italy to become "the greenest country in Europe".
A "Yes" victory would have been a heavy blow to the premier ahead of a constitutional reforms referendum in October, on which the 41-year-old has bet his political career.
As for the energy companies, it would mean that the 92 platforms within the 12-mile limit would have to start closing from 2018, with the last in 2034.
Observers had said ahead of the vote that the referendum outcome would have little effect on government income.
Concessions within the 12-mile band brought in a relatively modest €38 million ($43 million) in royalties in 2015, according to official data.
Opponents had warned that if the referendum failed, the door would be flung open to underhand dealings.
"The biggest favour to oil and gas - extending extraction indefinitely - allows them to avoid shelling out money to dismantle the platforms once the concessions have expired," said Michele Emiliano, president of the Puglia region.
The "Yes" camp had argued that far fewer than 11,000 jobs were at risk, as many of the 92 platforms in question are unmanned.
They were also furious that Renzi refused to hold the referendum on the same day as local elections later this year, a decision which will cost the taxpayer €300 million ($338 million), according to the regions and environmentalists.
Genoa Mayor Marco Doria said it was "outrageous" they had been kept separate in the hope Italians would head to the beach instead of voting.