The eccentric film, which features salesmen flogging novelty items, singing bar women and a sex-mad flamenco dancer, was hailed by critics in Venice for its distinctive look and moving exploration of what it means to be human.
Critics had widely tipped Venice's top prize for Joshua Oppenheimer's powerful "The Look of Silence" documentary on the Indonesian genocide, which missed out but instead took home the prestigious Grand Jury prize.
Jury head Alexandre Desplat, a French film score composer whose works include the music for "Harry Potter and the Deathly Hallows", said "Look of Silence" and "A Pigeon" were "two sides of the same coin. Both works had a an incredibly powerful impact."
Sweden's Andersson told the audience in English that he had been inspired by Vittorio De Sica's 1948 "Bicycle Thieves" — an emblem of Neorealism filmmaking — particularly the scene in the pawnshop, because the character "discovers Rome's poor".
"It's a humanistic scene, it's true empathy. That's what a movie should be," he said.
The work is the final part of a trilogy including "Songs from the Second Floor" and "You, the Living" and has a theatrical feel, with touches of Monty Python-esque comedy.
Props are limited to the odd briefcase in surreal scenes which jump from "Limping Lotta's bar" to a cafe where Sweden's 17th-century militarist King Charles XII seduces a waiter.
"It's the first time a Swedish film wins the Golden Lion, I'm hugely proud," Andersson said.
The 71st edition of the world's oldest film festival featured a line-up heavy with tales of war, loneliness, old age and unexpected friendships.
US actor Adam Driver, from the HBO series "Girls", won best actor for his portrayal of a desperate father trying to save his baby son's life in Italian Saverio Costanzo's New-York based tale about an eating disorder "Hungry Hearts".
Italian actress Alba Rohrwacher won best actress for her performance in the same film.
Oppenheimer, who was grounded in Chicago airport by bad weather, accepted his award by video link for "The Look of Silence", in which Adi, the brother of a murdered man, confronts his killers.
"Adi made this film because he wanted to meet the perpetrators and have them accept what they'd done so he could pardon them," Oppenheimer said.
The only person in the documentary who accepts responsibility, however, is the daughter of an ageing murderer.
"We in the West should follow this daughter's example of dignity, acknowledge our own role in the genocide and our collective responsibility for these crimes," Oppenheimer said.
British actor Tim Roth, part of the nine-member jury, described it as "a masterpiece, something spectacular that moved me immensely. I loved it and think it has a dignity beyond words".
Russian director Andrei Konchalovsky took home the Silver Lion for best director for "The Postman's White Nights", a warm-hearted, critically applauded tale about the loneliness of life in a remote lakeside community.
"The feeling is of a child getting a present from under the Christmas tree. Each of us does films for the child inside of us," an emotional Konchalovsky said in English.
Rising French star Paul Romain picked up the prize for best young actor for his role as a boy who meets his father in Alex Delaport's "Le Dernier Coup de Marteau".
The Lion of the Future for best debut film went to Indian director Chaitanya Tamhane's film "Court", set in Mumbai, about the trial of an ageing folk singer.
First-time feature "Sivas" by Turkey's Kaan Mujdeci — a coming-of-age tale about a young boy who befriends a dog he saves — won the Special Jury prize, while Rakhshan Bani-Etemad's "Tales", a collection of stories about the difficulties of life in contemporary Iran, won best screenplay.
Critics were surprised no prizes were awarded to American comedy "Birdman or the Unexpected Virtue of Ignorance", directed by Mexico's Alejandro Gonzalez Inarritu, which opened the festival with a punch and received rave reviews across the board.
Desplat defended the jury's decision, saying Inarritu is "a fantastic director, he has a talent and an eye few people have, but among twenty very different films, documentary-style films, poetical, dark ones, in the end only one really stood out."