Deporting the Tarakhel family, which includes six children, without first getting assurances from Italy they would be taken care of and kept together is a violation of Article Three of the European Convention on Human Rights, the court ruled.
The decision is final and can serve as a precedent for other migrants in a similar situation.
The court noted that the UN refugee agency had previously raised concerns over Italy's procedures for caring for refugees.
"The possibility that a significant number of asylum seekers may be left without accommodation or accommodated in overcrowded facilities without any privacy, or even in insalubrious or violent conditions, cannot be dismissed as unfounded," the court ruled.
Italy has seen a massive rush of migrants over the past year trying to reach the country in highly dangerous sea voyages. Authorities have rescued over 150,000 migrants since two deadly shipwrecks in October 2013 killed 400 people.
The Tarakhel family was to be deported to Italy, where they had landed in July 2011, under the EU rule referred to as the Dublin Regulation. The regulation says asylum seekers like the Tarakhels must have their case judged in the European country where they first landed.
After landing in Italy the Tarakhels went to Austria to apply for asylum, but it was rejected and referred to Italian authorities. The family then headed for Switzerland where they lodged a new asylum request in November 2011.
Swiss officials decided to simply pass the family's asylum application to Italy. Despite their case being passed to Italy, the family and its six children, the youngest of which was born in 2012, continue to live in the Swiss city of Lausanne.
German European Deputy Ska Keller noted the court raises the question of whether Italy has the capacity to handle other countries' migrants.
"The ruling shows the 'Dublin Regulation' is not working at all anymore. It is time to replace it with a reasonable system," instead of "holding to a system that discharges on the southern countries of the EU all responsibility for refugees," Keller said in a statement.