Italy's interior ministry said the 60 tribal leaders - notably the Tuareg of the southwest, the Toubou of the southeast, and the Arab tribe of Awlad Suleiman - had reached the 12-point deal after 72 hours of secret talks in Rome.
A representative from Libya's UN-backed Government of National Accord, which is based in Tripoli and controls western Libya, was also present.
"A Libyan border patrol unit will be operational to monitor Libya's southern border of 5,000 kilometres (3,100 miles)," Italian Interior Minister Marco Minniti told Italy's La Stampa newspaper, one of several Italian media outlets reporting on the deal on Sunday.
"Securing Libya's southern border means securing Europe's southern border," Minniti said.
READ ALSO: Italy unveils plan to better integrate fewer migrants
Southern Libya is criss-crossed by smuggling routes for people, drugs and weapons. Since the 2011 uprising that ousted Moamer Kadhafi, a mosaic of tribal and ethnic forces is fighting for control of illicit trade and oil fields in the region.
Tuaregs control the border with southern Algeria, while further east, the Toubou operate along the borders with Chad and Sudan.
Arab tribes in the region have supported the authorities in western Libya, but they also maintain ties with a rival administration that holds sway in the east - and regularly clash with the Toubou.
Fayez al-Sarraj, chief of the fragile GNA, has struggled to impose the government's authority, despite its backing by many political and military leaders.
The accord, whose details have not yet been released, is the latest in a series of deals European countries have sought to reduce migration from Libya, which has increased sharply in recent months.
READ ALSO: Mediterranean interior ministers meet to discuss migration flows
The deal aims to combat "an economy based on illicit drugs, which causes hundreds of deaths in the Mediterranean, thousands of desperate people looking for a better life, a populist push (in Europe) and a jihadist threat in the desert," according to the text of the agreement, quoted in the Corriere della Sera newspaper.
It also calls for job training programmes to keep young people from criminal activities.
Some 24,200 people have been rescued from the Mediterranean and registered at Italian ports so far this year, according to the Interior Ministry.
As part of an earlier agreement with the European Union, about 90 members of the Libyan coastguard are currently completing training under the EU, and Italy is preparing to return ten coastguard boats to Libya that it seized in 2011. They are expected to be operational by the end of April or in early May.
In March, interior ministers from several EU and North African countries reached a deal with the GNA to stem flow of migrant and human smuggling, which included pledges of money, coastguard training and equipment for Libya.