Letta said he will submit his resignation to President Giorgio Napolitano on Friday after less than a year at the head of an uneasy left-right coalition and just as Italy is beginning to emerge from a painful recession.
Members of the centre-left Democratic Party's governing directorate voted 136 for and 16 against to a motion requesting a change of government submitted by Renzi in a dramatic climax to a weeks-long feud with Letta.
The party thanked Letta for his "positive work" but called for "a new phase with a new executive".
Ever since being elected to lead the party in December, the ambitious and media-savvy Renzi has accused Letta of dragging his feet on crucial political reforms and failing to do enough to combat rampant unemployment.
Napolitano will now have to formally name a candidate to replace Letta, virtually certain to be Renzi, and a new cabinet could be in place by next week.
The resignation is due to be preceded by Letta's final cabinet meeting, starting at 1030 GMT on Friday.
Renzi said at the beginning of the Democratic Party's meeting that the party and the country were at a "crossroads" and there was a "necessity and urgency" of changing the government but ruled out early elections.
"We have to offer a way out of the marshes with a radical programme and profound change," he said, calling for a new government that could govern until the end of the parliament's mandate in 2018.
But the meeting also exposed divisions within the party with one dissenter against Renzi, Pippo Civati, likening the internal struggle between his party's leader and Letta to the 1980 horror film "The Shining".
Renzi, a photogenic former Boy Scout, who doubles as the mayor of Florence, is high in the opinion polls because he is seen as an outsider on the national political scene with on-the-ground local experience.
He has never been in government or parliament and often rails against the costs of Italy's political machine.
With his catchphrases, hashtags and informal style, he appeals particularly to a younger electorate and regularly communicates through social media.
A poll last month found 54 percent of Italians liked his leadership style, a much higher proportion than the roughly 25 percent who would vote for his party.
Stock markets fell sharply during trading but later gained ground and the FTSE Mib index ended the day down 0.17 percent, while borrowing costs rose for Italy on the bond market, reflecting investor nerves.
Tensions had been rising in the Democratic Party ever since Renzi won the party leadership in December.
Letta and Renzi failed to resolve their differences at talks on Wednesday and a defiant Letta ruled out resigning at a press conference later in the day in which he presented his reform programme for the year.
Letta supporters accuse Renzi of a "betrayal" that breaks a gentleman's agreement that Letta would be allowed to govern until after the conclusion of Italy's presidency of the European Union in 2014.
Renzi would be the youngest prime minister in the European Union and Italy's third government leader in a row who has not been directly elected – the last one being billionaire tycoon Silvio Berlusconi in 2008.
Letta was named by Napolitano to end a long impasse after inconclusive elections in 2013 only narrowly won by the Democratic Party and was preceded by economist Mario Monti, also appointed by the president.
Renzi has been at the centre of media attention in Italy for months but is still little known internationally – although he has said he looks to former British prime minister Tony Blair as a model.
With his mamma's boy good looks and populist touch, Renzi is considered a politician with "transversal" appeal who can win over part of the centrist electorate and has even been spoken highly of by Berlusconi.
A meeting with Berlusconi last month to agree political reforms caused controversy in his party, as does the fact that one of his main spin doctors is Giorgio Gori, a former head of Berlusconi's television stations.