The activists' "Defend Europe" mission has been financed by a crowd-funding initiative organised by young anti-immigration campaigners from France, Italy and Germany.
Their 40-metre (130-foot) ship named C-Star, hired by "Generation Identity", arrived on Saturday in an area where tens of thousands of migrants have been rescued from unseaworthy trafficker boats over recent months and years.
The boat spent 30-45 minutes tracking the MS Aquarius at a distance of a few hundred metres, before continuing to follow it from further back.
The Aquarius, a converted coastguard patrol boat, is operated by French aid group SOS Mediterranee and the international humanitarian organisation Doctors without Borders (MSF).
Its crew would not comment on whether they regarded the C-Star's proximity as intimidatory. Maritime charts indicated the NGO boat's speed doubled in the time the far-right vessel was close to it.
The two boats were about 20 nautical miles off Libya in an area east of the capital Tripoli.
On its website, the Defend Europe alliance accuses NGOs of "smuggling hundreds of thousands of illegal migrants to Europe, endangering the security and future of our continent" and vows to "do something against it."
French activist Clement Galant posted a video from the boat on Twitter on August 1st, in which he says the C-Star will accompany any migrant boat it comes across back to the African coast.
The Defend Europe initiative has been denounced by humanitarian organisations as a potentially highly dangerous publicity stunt.
Forcing a migrant boat that had reached international waters back to Libya, where most depart from, would be illegal under international law.
NGO-chartered boats have rescued over a third of the nearly 100,000 people who have been picked up from often distressed trafficker vessels off Libya this year and taken to Italy.
But the involvement of privately-funded boats in an operation mainly conducted by Italian navy and coastguard vessels has become subject to increasing scrutiny.
Critics say the NGOs are making it too easy for the traffickers to guarantee would-be migrants safe passage to Europe, creating a "pull" factor at best and operating a taxi service at worst.
Italian authorities last week impounded one NGO boat, the Iuventa, which is operated by German association Jugend Rettet, and accused its crew of being in direct contact with traffickers to organise pick-ups of boatloads of migrants from locations very close to the Libyan coast.
The NGO is challenging the seizure of its boat, saying it wants its crew to get back to saving lives as soon as possible.
Other organisations say they are happy to comply with tighter operational rules set by the Italian authorities but insist they will not give up their missions, saying thousands more people would have drowned but for their presence.
Some 600,000 mainly African migrants have reached Italy from Libya since the start of 2014, leaving the country's asylum facilities stretched and politicians under pressure to end the influx.