Integration Minister Cecile Kyenge, originally from the Democratic Republic of Congo, had asked party leader Roberto Maroni to call an end to the attacks saying otherwise she would not attend the conference.
"Since there was no intervention, the minister has decided to decline the invitation" for the meeting on Saturday, the minister's spokesman told reporters.
Kyenge last month was compared to an orangutan by a leading member of the Northern League, deputy speaker of the Senate Roberto Calderoli, who later said he had not intended his comments to be racist.
Kyenge also had bananas thrown at her during a political rally and one Northern League member said she should be raped so she understands how victims of crimes committed by undocumented immigrants feel.
Kyenge has called for reforms to Italy's immigration laws, which currently grant citizenship to the children of foreign-born immigrants only when they turn 18.
Italy's main political forces have condemned the racism against Kyenge. The Northern League won four percent of the vote in a February general election.