Berlusconi's grouping was expected to win between 31 and 41 percent of the vote, followed by the anti-establishment Five Star Movement with 29-32 percent, according to an exit poll by Rai public television.
The ruling centre-left Democratic Party, which struggled to get across its message of steady economic handling, was left in third place, according to the exit polls.
AS IT HAPPENED: The Local's live blog of the Italian election
Photo: Miguel Medina/AFP
Anger at the hundreds of thousands of migrant arrivals in Italy in recent years fired up a campaign, along with frustration about Italy's slow economic recovery.
The boost for far-right and populist parties has drawn comparisons to Britain's vote to leave the European Union and the rise of US President Donald Trump.
"The European Union is going to have a bad night," Marine Le Pen, leader of France's National Front, tweeted.
The Five Star Movement and the far-right League party, a member of Berlusconi's coalition, are both eurosceptic.
"Everyone is going to have to come and speak to us," Alessandro Di Battista, a leading member of the Five Star Movement, told reporters at the group's Rome rally.
The League, whose leader Matteo Salvini has been criticized for his Islamophobic rhetoric, was polling neck-and-neck with Berlusconi's Forza Italia (Go Italy). One prediction said the League may even come first which would change the balance of power in the coalition.
Berlusconi, a flamboyant three-time former prime minister, cannot hold elected office because of a fraud conviction but has put forward European Parliament President Antonio Tajani as his prime ministerial nominee. Salvini has said he should receive the prime ministerial nomination if his party comes in front -- a prospect that has spooked investors and European capitals.
READ ALSO: The Local's complete guide to the Italian election
The campaign was a gloomy one marred by clashes between far-right and anti-fascist activists, as well as a racist shooting spree by an extreme right sympathiser last month. The right has promised to deport 600,000 "illegal" migrants -- a prospect judged unfeasible by its rivals.
Former White House adviser Steve Bannon -- the man who helped Trump ride a populist wave to power -- characterised the election as "pure populism".
"The Italian people have gone farther, in a shorter period of time, than the British did for Brexit and the Americans did for Trump," Bannon, who was visiting Italy for the election, told the New York Times.
Bannon called a possible post-election deal between the Five Star Movement and the League "the ultimate dream".
'Berlusconi, you have expired'
If no grouping wins an overall majority, analysts say one scenario could be a grand coalition between the ruling centre-left Democratic Party and Berlusconi's Forza Italia -- a prospect that would reassure investors.
Berlusconi, who won his first election in 1994, has returned to the limelight at the age of 81 despite a career overshadowed by sex scandals and legal woes.
The billionaire was ambushed as he cast his vote in Milan by a topless woman from the Femen activist group who had "Berlusconi, you have expired" scrawled across her torso.
- 12 pictures that tell the story of the Italian election campaign
- Who's who in Italy's election?
- 17 of the most memorable quotes from the Italian election
In the event of a stalemate, President Sergio Mattarella will have the key role of choosing a prime ministerial nominee who could command a majority in parliament but negotiations could take weeks or even months.
"The verdict in Italy is always the same: the country is in constant instability. Being ungovernable has become endemic," said Claudio Tito, columnist for La Repubblica.
The Five Star Movement, which was founded in 2009 by a web entrepreneur and a former comedian, has tapped into disillusionment with traditional parties.
"I voted for the right and Berlusconi in the past... but this time I'm voting Five Star Movement to be against the parties that have always stolen," said 24-year-old pastry chef Francesco Tagliavini at a polling station in Rome's Tor Marancia neighbourhood.
"We hope something will change because until now things have been very bad," said Enzo Gallo, an elderly shopper at a street market in Milan.
"The middle class no longer exists, the poor are becoming poorer, the rich are becoming richer and there is no social justice," he told AFP.
Photo: Miguel Medina/AFP
By Dario Thuburn