The former AC Milan and Inter midfielder, who claimed the abuse also came from children, angrily confronted Cagliari fans, shouting at them: "This is my colour."
Muntari, 32, could now face disciplinary measures after deciding to leave the field seconds before full-time - a decision that earned him a second yellow card, and then a red, from referee Daniele Minelli.
Earlier, he had spoken directly with a section of the Cagliari supporters targeting him with racist abuse.
"They shouted at me from the beginning of the game, and in the first half I saw children in the group, so I went to their parents and gave them my jersey to set an example and to say you are not supposed to do that," he said.
"I tried to reason with them but the referee told me I had to stop. That's when I got pissed off. Why did he not stop the match?
"I am not a victim, but I am convinced that if they stopped the matches, this kind of thing would not happen any more."
While leaving the pitch in frustration and murmuring "Basta ! (Enough)", Muntari went to address more supporters, showing them his arm and shouting: "this is my colour, this is my colour."
His protest was later backed by Pescara coach Zdenek Zeman, although in comments to Sky Sport the Czech veteran said Muntari should not have quit the game.
"He asked the referee to intervene, but he (said he had) neither heard nor seen anything," Zeman said. "Muntari was right, but he shouldn't have left the pitch."
Muntari was at AC Milan when former teammate Kevin Prince Boateng angrily walked off the pitch while being targeted with racist abuse during a friendly with lower league side Pro Patria in January 2013.
It prompted a wave of support on social media, although football's ruling body FIFA, while applauding Boateng's principles, said it did not condone his decision to walk off.