Mapping the Future: An Expert Update on Mapillary, Open Data, and Meta’s Strategy

What is Mapillary? Building the World’s Database with Street-Level Imagery

The core idea of Mapillary, which started in 2013, remains highly relevant today. Mapillary is a platform designed for people to easily contribute to building a database of the world using street-level imagery.

The platform’s success relies on a convergence of technology: the widespread availability of smartphones with GPS and better cameras, improvements in computer vision technology, and scalable cloud services.

Once imagery is uploaded, computer vision is used to derive critical map data from that imagery. This map data serves diverse applications because the majority of contributors are using Mapillary to understand the world around them for updating or building a map.

Key use cases for Mapillary data include:

• Asset Management: Cities use it to track assets, such as the location of traffic signs.
• Humanitarian Efforts: Identifying buildings susceptible to flooding.
• Infrastructure Analysis: Working out where good and bad cycling infrastructure exists.

Since Mapillary joined Meta in 2020, much of the data has become more open and available to anyone, including companies, researchers, and individuals interested in using the imagery.

The Evolution of Contribution: From Phones to 360 Cameras

While smartphones were central to Mapillary’s early days, Ed notes that a major change since 2019 has been the proliferation of 360-degree cameras. These cameras, pioneered by companies like GoPro and Insta360, were originally popular for action videos but have become essential for mapping due to the inclusion of GPS.

Mapillary now supports a full range of devices, moving beyond cell phones to action cameras like the GoPro Max and professional-grade cameras (e.g., Trimble and Mosaic cameras) that capture high-resolution imagery for detailed asset management. Higher resolution cameras are particularly important because they allow for the extraction of more detailed map features, such as enabling Optical Character Recognition (OCR) to read restaurant names or street signs.

Low-Friction Uploading and Video Support

Mapillary has prioritized making the contribution process easy, minimizing friction for community mappers.

• Smartphones: Imagery captured via the Mapillary app uploads automatically when the device connects to Wi-Fi.
• Action Cameras: Users can simply transfer files from the camera’s SD card to the Mapillary Uploader desktop software using a drag-and-drop interface.

Furthermore, Mapillary now supports video uploads. This is crucial because relying on time-lapse intervals for traditional photos (e.g., every five seconds) can result in large gaps, potentially missing important objects like traffic signs. Video allows the system to determine the useful frame rate during processing, making the capture process seamless for the user.

Mapping in 3D: Structure from Motion and Data Extraction

A core technical function occurring behind the scenes is the creation of a 3D model of the world.

When images are uploaded, Mapillary first processes them to blur faces and license plates for privacy purposes, then deletes the original image. Next, the system builds a 3D model using Structure from Motion technology. This process involves identifying the same points (like the corner of a building) across multiple images to construct a 3D point cloud.

Once the 3D point cloud is built, computer vision and semantics are applied to identify objects—like a trash can or a crosswalk—and position them in the 3D space, providing an estimated latitude and longitude for the object.

Despite the technical complexity introduced by the wide variety of sensors and lighting conditions (ranging from inexpensive Android phones to expensive Trimble cameras), the derived map features are consistent:

• Extracted Map Features: Crosswalks, traffic signs, trash cans, bicycle parking, streetlights, and utility poles.
• Data Quality: The metadata of derived features allows users to filter data by the specific camera type or user who contributed the high-quality imagery.

Mapillary and the OpenStreetMap Ecosystem

Meta, which acquired Mapillary in June 2020, utilizes maps and location services across various surfaces (e.g., Facebook recommendations, Instagram maps, Ray-Ban Meta glasses AI assistant) to answer location questions like “Where am I?” and “What’s around me?”.

Mapillary is just one source Meta uses to enhance map data. Meta actively contributes to open data sources, supporting the OpenStreetMap (OSM) Foundation and collaborating through the Overture Maps Foundation.

The features Mapillary extracts are made available for OSM via several channels:

1. Directly through the Mapillary API.
2. Via the web interface (where users can zoom, filter, and download data).
3. As a vector tile layer in OSM editors, such as the Rapid Editor.

It is important to note that these extracted features are not automatically added to OpenStreetMap. Because OSM relies on human-curated data, the features are made available for mappers to review the corresponding imagery, confirm the feature is present (avoiding false positives), and then add it to the map.

Real-World Impact: Crowdsourced Data in Action

Crowdsourcing map data is challenging, potentially leading to quality or vandalism issues, but the diversity of knowledge and opinion gained from contributors outweighs these minor issues.

Mapillary data is solving critical business and public service problems globally:

• B Group (Vietnam): This growing ride-sharing company uses Mapillary to tackle the rapidly changing urban environment of cities like Ho Chi Minh City and Hanoi. They mounted 360-degree cameras (GoPro Max) on their riders’ scooters to capture almost the entire city. They use the data to:
  • Determine where cars/scooters can travel and identify time-based road restrictions.
  • Position building entrances more accurately and categorize places using OCR.
• City of Detroit (USA): The city implemented a strategy to move away from expensive, infrequent third-party surveys. They invested in a high-quality vehicle equipped with a Trimble camera and Lidar. By collecting data year-round and making it open, Detroit has been able to perform multiple surveys (lidar, street-level imagery, manual counting) in a single drive, efficiently using taxpayer funds. Detroit has successfully used the imagery to update census records, proving the population was higher than thought, and even helping reduce insurance rates.

Gazing into the Future of Mapillary and the Metaverse

Looking ahead, Mapillary is exploring technology that supports Meta’s vision for immersive experiences. While Lidar support is not currently on the roadmap, one highly anticipated area is NeRFs (Neural Radiance Fields).

NeRFs are a computer vision technique used to create highly immersive 3D scenes from images as input. Unlike existing 3D point clouds, NeRFs excel at the visual experience and can create “novel synthesis” (new views) by filling in missing elements using AI and ensuring consistent lighting across the scene.

This is exciting for the Metaverse, as NeRFs could allow users wearing headsets to “teleport” to real-world places (like the Great Ocean Road or Sydney Harbour Bridge) and explore them remotely with high realism.

Regarding new collection devices:

• Ray-Ban Meta Glasses: Mapillary has tested the glasses, but challenges remain, including battery limitations, reliance on phone GPS synchronization, and the difficulty of ensuring a consistent gaze angle required for reliable 3D reconstruction. They might be useful for limited bursts of capturing specific points of interest.
• Drones: Mapillary generally discourages drone uploads, although some success has been seen with low-level drone imagery (e.g., flying slightly higher on a busy European street to gain a better perspective blocked by parked cars). Drone imagery complicates the 3D model processing due to the shift in altitude.

Ed also identified the accurate identification of places (Points of Interest) as the single biggest impact Mapillary could have moving forward, given the rapid change rate of places globally (some statistics suggest 30% change per year). Challenges include reading various typographies and languages, determining the location of the relevant building entrance (customer vs. delivery), and conflating new data with existing 2D maps.

Get Involved!

Mapillary’s success is built on the contributions of its community over the last decade. Ed encouraged listeners to use the data, explore the licensing, and push the boundaries of what is possible with street-level imagery.

The team is constantly learning from users and welcomes feedback. To provide feedback, you can contact Mapillary via:

Previous episodes that you might find interesting