Firefighters were called to the scene after a passer-by spotted the blaze at around 6.00am in the textile town of Prato, just north of Florence in central Italy, according to local television.
Vincenzo Bennardo, who heads the town's firefighter service, told journalists that beyond the five bodies retrieved, "the remains found suggest that two others" died in the blaze.
The nationalities of the deceased have not been made public but local media reported that they are all Chinese nationals.
"We saw a column of smoke and we ran towards it. There were two Chinese, a man and a woman, completely covered in soot. They were screaming, flames were coming out of the warehouse," police officer Leonardo Tucia told the Corriere
The cause of the blaze was not yet known but the flames were thought to have led to the collapse of a portion of the building housing a dormitory for workers, local media said.
Some 14 Chinese workers are understood to have been living at the factory site.
For Prato city hall's security official Aldo Milone, it was an accident waiting to happen.
"I am not surprised by this tragedy… We have always underlined the risks that could happen in these factories that have been partially transformed into dormitories, with old electrical installations and people who smoke non-stop," he told journalists.
The two seriously injured men sustained severe burns, local media reported. A third victim, a woman, suffered from smoke inhalation but was not considered to be in a serious condition.
For the past two decades, Chinese immigrants have played a dominant role in Prato's garment industry.
The city is officially home to some 17,000 Chinese nationals, according to official data from 2010, but local sources say the real figure is closer to 50,000.
The presence of the Chinese garment workers is not always welcomed in the city, where numerous Italian firms have been forced shut as they were unable to compete with the immigrants.