Concern focused on Italy's third-biggest bank, Monte dei Paschi di Siena (BMPS), which analysts said is one of the country's most vulnerable large lenders.
Banking stocks went through a rollercoaster ride on the Milan bourse following Italian voters' rejection of constitutional reform that prompted Renzi's announcement that he would quit.
They opened sharply lower on Monday, and then recovered, before plunging again.
BMPS shares closed the day down 4.2 percent. Shares in Italy's biggest bank, UniCredit, shed 3.4 percent.
Meanwhile Banca Popolare di Milano tumbled 7.9 percent, Banco Popolare 7.4 percent and Mediobanca 4.2 percent.
BMPS, which is the world's oldest bank, has lost nearly 85 percent of its market capitalization since the start of the year. It also emerged as the worst performer from European Banking Authority (EBA) stress tests in July.
To ensure its survival, the bank has launched a plan involving the spinoff of 27.6 billion euros ($29.4 billion) worth of non-performing loans, combined with a capital increase of up to five billion euros. But the question is now whether political uncertainty will scare investors off.
"If there is no solution to the government crisis within a couple of weeks, financial markets will start getting jittery again," said Lorenzo Codogno, a former Italian Treasury official and now a professor at the London School of Economics.
"Probably the capital increase of Monte Paschi will be postponed or outright cancelled, and all other operations would be stalled."
While investors focus mostly on BMPS for now, the rest of the sector is far from immune to further panic, analysts said, pointing to the Milan's FTSE All-Share banking index, which has lost 47 percent of its value since January.
The referendum result is adding to deep worries about the failure of the Italian banking sector - which features no fewer than 700 banks - to make meaningful progress towards consolidation. And this, despite non-performing loans on their books amounting to a combined 360 billion euros, roughly a third of the eurozone's total bad debt.
A step back
Ahead of the referendum, BMPS managed to raise one billion euros via a voluntary conversion of bonds into capital. But all bets are off for the remaining four billion.
"The real risk lies with Monte dei Paschi di Siena. It's easy to imagine that investors who wanted to take part in the capital increase will now take a step back," said Umberto Borghesi, fund manager at Albemarle Asset Management.
"What is needed is a clear response on any government intervention, bail-in decision or any other solution," he said.
A "bail-in" forces existing creditors to write off part of their claims.
"No political formation can shoulder the responsibility of Monte dei Paschi failing," Borghesi said.
If investors really get cold feet, then UniCredit could be the next giant to feel the pain, analysts said, as it undergoes a massive strategy review.
It hopes to raise up to 13 billion euros of fresh capital early next year and analysts agree that the outlook for UniCredit is still favourable as it could also sell assets to fill capital gaps.
"An early sign of potential spillover of the referendum result to the economy will be if it damages plans to recapitalise some Italian banks, notably Banca Monte dei Paschi di Siena SpA and UniCredit SpA, and further weakens institutional investor willingness to provide funding and capital to the broader banking sector," said Fitch Ratings.
The agency put Italy's BBB+ rating, which is three notches above junk level, on negative outlook in October.
Fitch noted a bail-in of creditors, which is now standard practice in the eurozone, "would be politically problematic as bank debt is widely held by retail customers."
Analysts said smaller banks such as Carige or Veneta Banca are in much more immediate danger of falling victim to the ongoing crisis.
By Celine Cornu