Mapping the Future: An Expert Update on Mapillary, Open Data, and Meta’s Strategy
What is Mapillary? Building the World’s Database with Street-Level Imagery
The core idea of Mapillary, which started in 2013, remains highly relevant today. Mapillary is a platform designed for people to easily contribute to building a database of the world using street-level imagery.
The platform’s success relies on a convergence of technology: the widespread availability of smartphones with GPS and better cameras, improvements in computer vision technology, and scalable cloud services.
Once imagery is uploaded, computer vision is used to derive critical map data from that imagery. This map data serves diverse applications because the majority of contributors are using Mapillary to understand the world around them for updating or building a map.
Key use cases for Mapillary data include:
- Asset Management: Cities use it to track assets, such as the location of traffic signs.
- Humanitarian Efforts: Identifying buildings susceptible to flooding.
- Infrastructure Analysis: Working out where good and bad cycling infrastructure exists.
Since Mapillary joined Meta in 2020, much of the data has become more open and available to anyone, including companies, researchers, and individuals interested in using the imagery.
The Evolution of Contribution: From Phones to 360 Cameras
While smartphones were central to Mapillary’s early days, Ed notes that a major change since 2019 has been the proliferation of 360-degree cameras. These cameras, pioneered by companies like GoPro and Insta360, were originally popular for action videos but have become essential for mapping due to the inclusion of GPS.
Mapillary now supports a full range of devices, moving beyond cell phones to action cameras like the GoPro Max and professional-grade cameras (e.g., Trimble and Mosaic cameras) that capture high-resolution imagery for detailed asset management. Higher resolution cameras are particularly important because they allow for the extraction of more detailed map features, such as enabling Optical Character Recognition (OCR) to read restaurant names or street signs.
Low-Friction Uploading and Video Support
Mapillary has prioritized making the contribution process easy, minimizing friction for community mappers.
- Smartphones: Imagery captured via the Mapillary app uploads automatically when the device connects to Wi-Fi.
- Action Cameras: Users can simply transfer files from the camera’s SD card to the Mapillary Uploader desktop software using a drag-and-drop interface.
Furthermore, Mapillary now supports video uploads. This is crucial because relying on time-lapse intervals for traditional photos (e.g., every five seconds) can result in large gaps, potentially missing important objects like traffic signs. Video allows the system to determine the useful frame rate during processing, making the capture process seamless for the user.
Mapping in 3D: Structure from Motion and Data Extraction
A core technical function occurring behind the scenes is the creation of a 3D model of the world.
When images are uploaded, Mapillary first processes them to blur faces and license plates for privacy purposes, then deletes the original image. Next, the system builds a 3D model using Structure from Motion technology. This process involves identifying the same points (like the corner of a building) across multiple images to construct a 3D point cloud.
Once the 3D point cloud is built, computer vision and semantics are applied to identify objects—like a trash can or a crosswalk—and position them in the 3D space, providing an estimated latitude and longitude for the object.
Despite the technical complexity introduced by the wide variety of sensors and lighting conditions (ranging from inexpensive Android phones to expensive Trimble cameras), the derived map features are consistent:
- Extracted Map Features: Crosswalks, traffic signs, trash cans, bicycle parking, streetlights, and utility poles.
- Data Quality: The metadata of derived features allows users to filter data by the specific camera type or user who contributed the high-quality imagery.
Mapillary and the OpenStreetMap Ecosystem
Meta, which acquired Mapillary in June 2020, utilizes maps and location services across various surfaces (e.g., Facebook recommendations, Instagram maps, Ray-Ban Meta glasses AI assistant) to answer location questions like “Where am I?” and “What’s around me?”.
Mapillary is just one source Meta uses to enhance map data. Meta actively contributes to open data sources, supporting the OpenStreetMap (OSM) Foundation and collaborating through the Overture Maps Foundation.
The features Mapillary extracts are made available for OSM via several channels:
- Directly through the Mapillary API.
- Via the web interface (where users can zoom, filter, and download data).
- As a vector tile layer in OSM editors, such as the Rapid Editor.
It is important to note that these extracted features are not automatically added to OpenStreetMap. Because OSM relies on human-curated data, the features are made available for mappers to review the corresponding imagery, confirm the feature is present (avoiding false positives), and then add it to the map.
Real-World Impact: Crowdsourced Data in Action
Crowdsourcing map data is challenging, potentially leading to quality or vandalism issues, but the diversity of knowledge and opinion gained from contributors outweighs these minor issues.
Mapillary data is solving critical business and public service problems globally:
- B Group (Vietnam): This growing ride-sharing company uses Mapillary to tackle the rapidly changing urban environment of cities like Ho Chi Minh City and Hanoi. They mounted 360-degree cameras (GoPro Max) on their riders’ scooters to capture almost the entire city. They use the data to:
- Determine where cars/scooters can travel and identify time-based road restrictions.
- Position building entrances more accurately and categorize places using OCR.
- City of Detroit (USA): The city implemented a strategy to move away from expensive, infrequent third-party surveys. They invested in a high-quality vehicle equipped with a Trimble camera and Lidar. By collecting data year-round and making it open, Detroit has been able to perform multiple surveys (lidar, street-level imagery, manual counting) in a single drive, efficiently using taxpayer funds. Detroit has successfully used the imagery to update census records, proving the population was higher than thought, and even helping reduce insurance rates.
Gazing into the Future of Mapillary and the Metaverse
Looking ahead, Mapillary is exploring technology that supports Meta’s vision for immersive experiences. While Lidar support is not currently on the roadmap, one highly anticipated area is NeRFs (Neural Radiance Fields).
NeRFs are a computer vision technique used to create highly immersive 3D scenes from images as input. Unlike existing 3D point clouds, NeRFs excel at the visual experience and can create “novel synthesis” (new views) by filling in missing elements using AI and ensuring consistent lighting across the scene.
This is exciting for the Metaverse, as NeRFs could allow users wearing headsets to “teleport” to real-world places (like the Great Ocean Road or Sydney Harbour Bridge) and explore them remotely with high realism.
Regarding new collection devices:
- Ray-Ban Meta Glasses: Mapillary has tested the glasses, but challenges remain, including battery limitations, reliance on phone GPS synchronization, and the difficulty of ensuring a consistent gaze angle required for reliable 3D reconstruction. They might be useful for limited bursts of capturing specific points of interest.
- Drones: Mapillary generally discourages drone uploads, although some success has been seen with low-level drone imagery (e.g., flying slightly higher on a busy European street to gain a better perspective blocked by parked cars). Drone imagery complicates the 3D model processing due to the shift in altitude.
Ed also identified the accurate identification of places (Points of Interest) as the single biggest impact Mapillary could have moving forward, given the rapid change rate of places globally (some statistics suggest 30% change per year). Challenges include reading various typographies and languages, determining the location of the relevant building entrance (customer vs. delivery), and conflating new data with existing 2D maps.
Get Involved!
Mapillary’s success is built on the contributions of its community over the last decade. Ed encouraged listeners to use the data, explore the licensing, and push the boundaries of what is possible with street-level imagery.
The team is constantly learning from users and welcomes feedback. To provide feedback, you can contact Mapillary via:
Previous episodes that you might find interesting
In Conversation
How Ed Got Involved with Mapillary
Daniel: You’re a product manager at Meta, and you’ve worked on Mapillary for around 10 years. How did you first get involved?
Ed: I moved to Sweden in 2014, studying something completely unrelated — I was interested in foreign tech companies in China. I needed a job, started looking at interesting startups in Sweden, and found Mapillary. I emailed them almost exactly 10 years ago, said I loved the idea of building a database of the world using street-level imagery and phones, and asked if there was a role. Luckily they got back to me soon after, and that was the beginning of an incredible journey.
Daniel: So you had no prior background in mapping or geospatial?
Ed: None — other than loving maps and atlases since I was a kid. That was the seed. When I saw a maps-related startup in Sweden, it jumped straight to the top of my list. It was very early days — about six people on the team — but there was already an incredible community swell, with images coming in from around the world. That was intriguing, and I think a lot of future employees and contributors resonated with it too.
What Mapillary Is, and Who Uses the Data
Daniel: For anyone who doesn’t know — what was Mapillary when it started, and what is it now?
Ed: What’s funny is I went back and listened to your 2019 episode in preparation, expecting it to be out of date, and hardly anything in it wasn’t still relevant. Even the original 2013 idea holds up. Map data is hard, and collecting it is hard. But in 2013 we suddenly had smartphones with GPS in our pockets, cameras were getting much better, computer vision was improving, and cloud services made it possible to scale a platform quickly. So the idea of Mapillary is a platform people can easily contribute to with their cameras and phones, building a street-level imagery platform — and then using computer vision to derive map data from that imagery.
Daniel: What’s that map data used for, and who gets access?
Ed: Most people contributing want to understand the world around them — building or updating a map. That ranges from cities doing asset management, keeping track of where traffic signs are, to humanitarian use cases identifying buildings susceptible to flooding, to working out which parts of a city have good or bad cycling infrastructure. Mapillary used to make data available for OpenStreetMap under the ODbL licence and charge for commercial use. When we joined Meta in 2020, we were able to open up a lot of the data and make it available to anyone interested — so we’re seeing a resurgence of exciting use cases.
The Rise of 360 Cameras and Easy Uploading
Daniel: We’ve talked about phones — what other devices can upload imagery?
Ed: The big change since 2019 is how many more 360 cameras there are. When I joined in 2015, we’d get these weird, clunky, expensive 360 cameras in the office — many of them Kickstarter campaigns with fairly bad software. Since then, largely thanks to GoPro and Insta360, there’s been a resurgence of good 360 cameras. Their primary use case was nice ski and surfing videos, but we’ve benefited — particularly the ones with GPS. A lot of our imagery now comes from cameras like the GoPro Max, an easy-to-use action camera, all the way up to professional-grade Trimble and Mosaic cameras that capture high-resolution imagery. Higher resolution matters because you can extract more detail — like optical character recognition to read a restaurant name, which is hard on a grainy image.
Daniel: I tried it myself. Someone passed me a 360 camera on the promise I’d pass it on, I put it on my helmet, and started biking around mapping my local town. The uploader was magical — literally drag and drop the file into a folder and off it goes. Compared to contributing to other projects, which was painful, the whole experience was seamless.
Ed: Huge credit to the team. There’s a series of Python scripts called Mapillary tools — the underlying logic for taking imagery or video and processing it, and understanding all the different file formats — built into that desktop uploader. It’s really important, because that’s what makes people want to go out and do it again. If you had to spend your whole weekend writing scripts to process the imagery, you probably wouldn’t do it very often. The cameras enable it too — they’re smaller and less expensive now.
Behind the Scenes: Structure from Motion and Feature Extraction
Daniel: Are you also building a 3D model of the world in the background?
Ed: Yes. Once you upload images, the first thing we do is blur faces and license plates, then delete the original image, so we don’t keep anything privacy-compromised. Then we build a 3D model using Structure from Motion, which looks for points across multiple images that computer vision can identify as the same — say, the corner of a building — and reconstructs the world in 3D as a point cloud. Once we have that point cloud, we apply semantics and our understanding of an image to position objects: if we think we’ve identified a trash can or a crosswalk, we position that feature in the point cloud and give you an estimated latitude and longitude.
Daniel: That must be a huge task given all the different sensors you support.
Ed: It’s both a benefit and a challenge — so many different cameras and GPS thresholds. Even internally there are conversations about what the quality of Mapillary is, and it’s hard to give a straight answer. We have $50 Android smartphones contributing, which is still useful — we’re getting photos that show what the world looks like — and places like Detroit capturing nearly every day with a very expensive Trimble camera. Making sense of that full range and building a point cloud from different sources and lighting conditions is probably one of the most technically challenging aspects. The extracted features are pretty consistent though: crosswalks, traffic signs, trash cans, bicycle parking, streetlights, utility poles — each with an estimated lat-long. And the metadata of a feature contains the image it was detected in, so you can filter by users or camera types you know contribute higher-quality imagery.
Mapillary, Meta, and the OpenStreetMap Ecosystem
Daniel: How does Mapillary fit in with what Meta is doing?
Ed: Meta uses maps and location in a variety of ways — Facebook recommendations, a map appearing when you search a restaurant on Instagram, the AI assistant on Ray-Ban Meta glasses — all answering “where am I?” and “what’s around me?” Mapillary is just one source for that. But Meta is very involved in using and contributing to open data: we make donations to the OpenStreetMap Foundation and OSM US, build editing software like Rapid, and are part of the Overture Maps Foundation, a multi-company collaboration. We want to do this in the open, because maps are incredibly hard and the world changes rapidly — I’ve seen stats that places change 30% a year, sometimes higher.
Daniel: And the features Mapillary extracts go into OpenStreetMap?
Ed: They’re available for it, through several channels — our API, our web interface where you can zoom into an area and download filtered map data, and as a vector tile layer in OSM editors like the Rapid Editor. But they’re not added automatically. OpenStreetMap is very much about human-created data, so if you’re mapping crosswalks you’d toggle the map feature layer in Rapid, see where they are, review the image to make sure it’s actually there and not a false positive, and then add it.
Crowdsourced Data in Action: Vietnam and Detroit
Daniel: Are there cities or organisations investing in collecting data this way?
Ed: Two stories. One is B Group, a growing ride-sharing company in Vietnam. A ride-sharing company really needs people and packages to get from A to B efficiently, so they’ve had to get into the mapping business — you’ve seen the same with Uber and Grab. They put 360 cameras — GoPro Max — on some of their riders’ scooters and captured almost all of Ho Chi Minh City and Hanoi. They use it to work out where scooters can go but cars can’t, time-based road restrictions, and OCR to position building entrances and categorise places. There’s a clear business return for them, and they’re contributing back — anyone interested in Vietnam can use that same data.
Ed: The other is the City of Detroit. A GIS officer there, Dexter, started by just driving around with a GoPro Max, proved he could collect useful data, and won a mayor’s innovation grant. That turned into a better camera, more sponsorship from the city and General Motors, and today they have a dedicated vehicle with a Trimble camera and LiDAR, two drivers, capturing most of the year. Their philosophy: instead of paying companies to survey once every few years, do it continuously and openly — collecting LiDAR, street-level imagery, and traffic-sign counts in a single drive. They’ve used the imagery to update census records, proving the population was higher than thought, and even helped reduce insurance rates. It’s being smart with taxpayer funds.
The Future: NeRFs, Smart Glasses, and Drones
Daniel: Let’s do some future-gazing. Where are things going — we’re not doing LiDAR, a big disappointment, a sigh goes through the audience. What are we doing?
Ed: One thing widely received this year was a blog post Peter Kontschieder wrote about NeRFs — neural radiance fields — a way to create highly immersive 3D scenes from images. A 3D point cloud is just isolated points in space; a NeRF is much better for the visual experience. It can also create “novel synthesis” — new views — filling in bits you missed using AI and keeping the lighting consistent across the scene. That’s exciting for Meta, because it’s all about letting people experience a scene remotely with a headset — teleporting to the Great Ocean Road or Sydney Harbour Bridge. When people visit places they know, they want it to be realistic — not buildings missing or a wall leaning at 45 degrees.
Daniel: What about Ray-Ban glasses as a collection device, or drones?
Ed: We’ve tried the glasses. There are limitations — capture started at about 30 seconds, now around three minutes, but it drains the battery quickly; you need to rely on your phone’s GPS and sync the video frames to it; and when you wear glasses you turn your head, whereas for 3D reconstruction it’s much better to keep a consistent gaze angle. So they’d be good for limited bursts — walk down the street and grab a 30-second video of something that interests you. Drones we generally discourage — you’d really need a different platform, filtering by altitude, and the 3D model gets harder when some views are at ground level and some are 100 metres up. We have had success with low-level drone imagery, though — flying slightly higher down a European street gives a better perspective past parked cars and bicycles.
The Hardest Problem: Mapping Places
Daniel: Is there an object you’d love to be able to detect that you can’t today?
Ed: Places is the big one. On any map database — OpenStreetMap, Google Maps, Overture — places are the hardest thing to keep up to date, and Mapillary can play a better role there: discerning the name of a place, what type of business it is, where the entrance is located. Front-facing imagery from a car made that challenging — maybe only part of the name is visible — but with more 360 imagery it’s becoming feasible. Reading the name is hard because of different typography and languages, and working out which part is actually the name. Then there’s the entrance — a large building or shopping mall might have separate customer, pickup, and delivery entrances. And then conflating it with the existing 2D map: which polygon does it relate to, is there already a place there, has it changed name?
Daniel: Is there anything you want to leave the listeners with?
Ed: Mapillary is what it is because of everyone contributing to it over the last 10 years — last year was the official 10-year anniversary, and we had a nice celebration in southern Sweden. I’m at a stage now where I’m learning more from the people using Mapillary, like the City of Detroit and B Group, who keep finding new ways to push the boundaries of what’s possible with street-level imagery. I’d encourage people to use the data — it’s much more open than a lot of other street-level imagery platforms — look at the licensing and terms, and build things with it. And we’d love feedback; the team is always learning from our users, and that’s a big part of how we’ve improved.
