Speaking at a conference in Rome, Prodi pointed to the successful nuclear agreement reached earlier this week between Iran, the US and five other world powers as an example of the progress that can be achieved when major players come together.
He added that the Iran accord is "important" and shouldn't be underestimated.
But he said the ongoing tension between the US and Russia is "poisoning" international relations and preventing issues in Libya and other areas of the Middle East from being resolved.
"To eradicate terrorism, it's necessary that the big powers reach an agreement," Prodi said.
"But several things are preventing this - one is the US and Russia confronting each other over Ukraine - it's keeping the level of tension too high."
"Why? Take Libya and the Middle East - you can't get peace in these places without the US and Russia taking the same position."
Prodi, who was president of the European Commission between 1999 and 2005, was speaking during a conference on the controversial TTIP (Transatlantic Trade and Investment Partnership) between the US and the EU.
The EU claims the agreement will create jobs and reduce the cost of living while detractors argue that it's a threat to public services and only favours big corporations.