"I would never, ever have believed a documentary would win the Venice film festival," said an emotional Rosi, who won the top prize out of 20 films in competition with his tale, which peered into private moments in houses, workplaces and gardens.
The director said he had spent years preparing and researching for the movie and thanked "the characters for allowing me into their lives".
"Holy GRA" was one of an unprecedented two documentaries in the running for the Lion, and Rosi said it had been a "huge act of courage" to let them compete.
The film glimpses into the lives of a former aristocrat in a kitsch palace, an ambulance man who rushes to save car crash victims on the ring road and an elderly bushy-bearded father who shares a tiny apartment with his daughter.
"I spent a long time getting to know the areas and the characters, getting rid of all my pre-conceptions. I relied on instinct to draw out the story," he said.
With simple, pared-down shots, it quietly reveals the vast range of personalities living in tower blocks or country houses under airport flight paths.
From an eel fisherman on the Tiber River to a man who catalogues palm trees ravaged by weevils, the film uncovers unexpected sources of wit, eccentricity and kindness.
"I wanted to be surprised and Rosi's film was surprising. It was a unanimous choice," said jury head, Italian cinematic master Bernardo Bertolucci.
"Like a one-man orchestra, Rosi managed to reveal these characters to us within this ring road which circles Rome like a ring around Saturn. The world, the spaces in the film, had something Franciscan about them. There was a real sense of purity, the jury felt a poetic force," he said.
Italy last won the Golden Lion in 1998 with Gianni Amelio's "The Way We Laughed".
The best actor prize went to Greece's Themis Panou for his role as father to a family which harbours a deep secret in Alexandros Avranas's "Miss Violence".
Director Avranas was awarded the Silver Lion for best director for the film, which shocked audiences with a chilling tale of incest and child suicide.
"I believe this film will help tackle domestic abuse. Art can play a fundamental role in uncovering personal secrets and society's secrets," Panou said.
The award for best actress was picked up by 82-year-old Elena Cotta for her portrayal of a stubborn old lady in southern Italy who refuses to back down from a staring feud in Emma Dante's "A Street in Palermo".
Best screenplay went to Steve Coogan and Jeff Pope for Stephen Frears's "Philomena", a charming tragi-comedy starring Judy Dench, based on the true tale of mother Philomena Lee's search for her son after he is given up for adoption.
Coogan, who could not be present, sent a message saying: "Jeff and I are thrilled. It's truly been a labour of love. We are indebted to Judy Dench for giving heart and soul to the story, and Philomena for allowing it to be told."
Best young actor went to Tye Sheridan for his depiction in "Joe" of a young man desperately trying to save his family from an alcoholic, abusive father.
The grand jury prize went to Tsai Ming-liang's "Stray Dogs", the incredibly slow-moving story of a homeless father who earns a meagre living as a human billboard while his children wander the streets of contemporary Taipei.
Philip Groning's controversial "The Police Officer's Wife", a series of episodes which explore a mother's struggle to protect her child from domestic violence, was awarded the special jury prize.
Noaz Deshe's "White Shadow", the tale of a young Albino in Tanzania who is hunted by local witch doctors who want to use his body parts for their potions, was given the Lion of the Future award for a debut film.
The festival brought Hollywood stars including George Clooney, Sandra Bullock, Nicolas Cage and Scarlett Johansson by speed boat and gondola to Venice's Lido, as well as art house auteurs from around the globe.
Bertolucci, best known for his raunchy 1972 "Last Tango in Paris", was joined on the jury by British director Andrea Arnold ("Red Road"), German actress Martina Gedeck ("The Lives of Others") and US actress Carrie Fisher.