"Over the next week, I will launch a new round of consultations to see if the possibility of forming a government, which today has not materialised, has advanced," Mattarella told reporters.
But anti-establishment and far-right leaders stood firm in a battle for the upper hand.
Five Star Movement (M5S) leader Luigi Di Maio called on right-wing leader Matteo Salvini to ditch coalition partner Silvio Berlusconi, and "change Italy" with the M5S.
Di Maio's M5S, Italy's largest single party after picking up just under 33 percent of the vote in last month's election, is staunchly opposed to working with the 81-year-old media magnate and wants to break up the right-wing coalition that garnered 37 percent of votes at the polls.
"I don't recognise the right-wing coalition. They had three different prime ministerial candidates and three different programmes," said Di Maio to reporters after meeting President Sergio Mattarella on Thursday afternoon.
"They also have completely different ideas, such as on working with the M5S."
Ahead of fresh talks, Di Maio and Salvini will continue their negotiations.
Any deal between the right and the M5S hinges on former prime minister Berlusconi, who is a fierce rival of Di Maio and a long-time target of barbs from M5S founder Beppe Grillo.
Earlier on Thursday Berlusconi said that his Forza Italia party was "not ready to form a government in which the politics of envy, social hatred, pauperism and judicial witch-hunts prevail", in a dig at the M5S.
Berlusconi is firmly against some of the M5S's flagship policies, such as the proposal for a basic income which would extend to nine million people.
However earlier on Thursday Salvini reiterated his loyalty to his coalition, saying he wanted to bring a "solid" right-wing group to government.
Should Salvini break from Berlusconi and sign what Di Maio called a German-style "government contract", he would likely be forced to play second fiddle to Di Maio as the weaker coalition partner.
Di Maio also appealed to the up to now intransigent centre-left Democratic Party (PD) to come to the negotiating table.
The PD's coalition came third but has enough seats in both houses of the Italian parliament to prop up a M5S or right-wing government.
However PD leader Maurizio Martina confirmed that his party would not work with either the right or M5S, saying that it was up to their opponents to try to form a government.
By Terry Daley