PIGEON: Predicting Image Geolocations

Lukas Haas, Michal Skreta, Silas Alberti, Chelsea Finn
Stanford University
CVPR 2024
Paper Code arXiv YouTube

PIGEON deployed in a live GeoGuessr game against professional player Trevor Rainbolt.

Abstract

Planet-scale image geolocalization remains a challenging problem due to the diversity of images originating from anywhere in the world. Although approaches based on vision transformers have made significant progress in geolocalization accuracy, success in prior literature is constrained to narrow distributions of images of landmarks, and performance has not generalized to unseen places. We present a new geolocalization system that combines semantic geocell creation, multi-task contrastive pretraining, and a novel loss function. Additionally, our work is the first to perform retrieval over location clusters for guess refinements. We train two models for evaluations on street-level data and general-purpose image geolocalization; the first model, PIGEON, is trained on data from the game of GeoGuessr and is capable of placing over 40% of its guesses within 25 kilometers of the target location globally. We also develop a bot and deploy PIGEON in a blind experiment against humans, ranking in the top 0.01% of players. We further challenge one of the world's foremost professional GeoGuessr players to a series of six matches with millions of viewers, winning all six games. Our second model, PIGEOTTO, differs in that it is trained on a dataset of images from Flickr and Wikipedia, achieving state-of-the-art results on a wide range of image geolocalization benchmarks, outperforming the previous SOTA by up to 7.7 percentage points on the city accuracy level and up to 38.8 percentage points on the country level. Our findings suggest that PIGEOTTO is the first image geolocalization model that effectively generalizes to unseen places and that our approach can pave the way for highly accurate, planet-scale image geolocalization systems. Our code is available on GitHub.

NPR

Geoff Brumfiel

The Register

Thomas Claburn

This AI is better than you at figuring out where a street pic was taken just by looking at it

PIGEON homes in on your geolocation

A trio of Stanford computer scientists have developed a deep learning model to geolocate Google Street View images, meaning it can figure out generally where a picture was taken just by looking at it.

The software is said to work well enough to beat top players in GeoGuessr ...

NPR

Sabrina Ortiz

This AI can find your location just by looking at a few photos

Developed by Stanford graduate students, an AI model can determine – with impressive accuracy –
a specific location simply by looking at Google Street View.

Overview

Administrative boundary and training set metadata are hierarchically ranked, clustered, and Voronoi tessellated to create semantic geocells. The geocell labels are then used to create continuous labels via haversine smoothing. Additionally, we pretrain CLIP via geographic synthetic captions in a multi-task setting. The pretrained CLIP model together with an OPTICS clustering model are employed to generate location cluster representations. During inference, an image embedding is computed and first passed to a linear layer to create geocell predictions and to identify the topK geocell candidates. The embedding is also used in our refinement pipeline to refine predictions within and across geocells. This is achieved by minimizing the embedding L2-distance between the inference image embedding and all location cluster representations across the topK geocells. Finally, predictions are refined within the top identified cluster to generate geographic coordinates as outputs.

PIGEON Pipeline

Image Geolocalization Results

PIGEON

PIGEON is a model trained on an original dataset of 100,000 randomly sampled locations from the game of GeoGuessr which each consist of four images spanning an entire “panorama", or a 360-degree view, for a total of 400,000 training images. The following is an example model input from a location in Pegswood, England

PIGEON Input Example

Evaluated on a holdout dataset of 5,000 Street View locations, PIGEON is capable of placing >40% of all predictions within 25 kilometers of the target location, and more than 5% within one kilometer. This corresponds to a median error of only 44.4 kilometers and an average GeoGuessr score of 4,525.

PIGEON Benchmark Results

PIGEOTTO

PIGEOTTO was trained as a general image geolocator which is able to generalize to any distribution of images. The model was trained on a combination of 4,166,186 images from the MediaEval 2016 dataset (Larson et al., 2017) consisting of geo-tagged Flickr images from all over the world, and 340,579 images from the Google Landmarks v2 dataset (Weyand et al., 2020) derived from Wikipedia. Unlike PIGEON, PIGEOTTO takes a single image per location as input, some examples of which are shown below

PIGEOTTO Input Example

When evaluated on common image geolocalization benchmarks, PIGEOTTO reduces the median kilometer error by 2-5 times on all benchmarks not solely focused on landmarks (IM2GPS). PIGEOTTO achieves stateof-the-art (SOTA) performance on every single benchmark dataset and on the majority of distance-based granularities. The model is truly planet-scale in nature, exhibits robust behavior to distribution shifts, and is the first geolocalization model that effectively generalizes to unseen places.

PIGEOTTO Benchmark Results

Match against a Professional GeoGuessr Player

As part of our evaluations, we challenge one of the world's foremost professional GeoGuessr players, Trevor Rainbolt, to a match and win six out of six planet-scale, multi-round games. PIGEON is the first model to reliably beat a GeoGuessr professional.

Additional Information

Model Access

For privacy and safety reasons, we do not make our model weights or dataset publicly available. However, we collaborate with institutions on a case-by-case basis if we determine the use case to be socially beneficial. For example, we have previously worked with INTERPOL on a criminal investigation. If you believe that access to PIGEON would be helpful and your use case is socially beneficial, please contact Lukas Haas, Michal Skreta, or Silas Alberti.

BibTex

@misc{haas2023pigeon,
        title={PIGEON: Predicting Image Geolocations}, 
        author={Lukas Haas and Michal Skreta and Silas Alberti and Chelsea Finn},
        year={2023},
        eprint={2307.05845},
        archivePrefix={arXiv},
        primaryClass={cs.CV}
    }