In this episode, the discussion is about how to make base maps that are more flexible and problem specific. Currently, the base maps that are commonly used are commercial, one-size-fits-all solutions that may not be a great fit for specific use cases. Protomaps is a project that makes it possible to build a single planet-scale tiled map, which does not need a server to put on the web. It can simply be hosted on cheaper options such as Amazon S3 object storage.
About The Guest
Brandon Liu is the founder and creator of Protomaps. He came from a computer science background, and first encountered geospatial when making a biking app for his city. Since then, he learned about OSM, web mapping, and GIS. While working as a developer he recognized the gap between the data that we are accustomed to with Google or Apple maps and what an individual developer can accomplish. This led to the birth of Protomaps, a project that seeks to make it possible for more people to make custom base maps that best serve their needs.
What Is The Difference Between A Server System And Serverless System?
A server is a computer or program that manages the access of other computers (clients) to a centralized resource in a network. In the geospatial context, this could be a GeoServer which can be used to share GeoJSON files.
On the other hand, a serverless system uses a cloud-native development model that allows developers to run their applications without getting into the complexities of running servers. This simplifies the process for organizations that need to serve geospatial data, especially since most of them are not software companies, but could span from local government agencies to non-profit organizations.
The Scarcity of Basemaps
Traditionally, being able to create and also deploy a global base map has been the capability of very few multi-billion companies such as Google, Apple, Microsoft, and Facebook. Typically, base maps are only accessible to such a small group of organizations. Consequently, what a developer can do with base maps is limited by the data that is available from a map API such as Google. The lack of flexibility in using external APIs as well as the pricing (which may be unfavorable for some users) makes base maps scarce when seeking a problem-specific solution. The long-term goal for Protomaps is to enable people to combine their imagination with any data that is available to them — whether that is from OpenStreetMap, open data from a government portal, or their own data — to build global-scale tiled maps.
What Are PMTiles?
PMTiles is Protomaps’ tile format. It provides a way to create and host a tiled map without the complexity of servers. Developers can simply drop the tiled map in a storage bucket such as S3 and users can access, zoom, and pan around the map – all on a single S3 file.
As opposed to an X, Y, Z tiling system that could mean having hundreds of millions of individual tiles at higher zoom levels with slower uploads, PMTiles is a single file that contains all the data, which can be accessed dynamically through HTTP range requests.
What Is The Difference Between PMTiles And MBTiles?
There are many similarities between the concept of MBTiles and PMTiles. However, the main difference is that MBTiles is an approach built on a SQLite database and also requires a server. Using MBTiles involves renting a server such as a Linux server and running a program that translates MBTiles requests into an HTTP request. On the other hand, PMTiles is an adaptation of the same overall concept but without the use of servers.
CDN (Content Delivery Network)
A CDN helps to solve potential performance issues related to posting PMTiles on a storage system like S3, especially when many users are trying to access the map at one time. If there is a huge spike of traffic to your website, a CDN can scale the handling of public traffic without the need to rent more servers for that spike. Using a CDN also makes the user experience more efficient since it pulls data requested by the user from the closest geographical node.
How to Create PMTiles
PMTiles can be created using a tool called Tippecanoe, which was originally developed by Mapbox. Tippecanoe can convert GeoJSON datasets into PMTiles. If you have data in formats other than GeoJSON, the first step would be to convert it to GeoJSON before using Tippecanoe to create PMTiles. It is also possible to combine two or more PMTiles into a single PMTiles using the Tile-join tool in Tippecanoe.
How Will A Future With PMTiles Look Like?
A future with PMTiles could mean a lot more diversity of the cartography done on the web. Having the capability to serve any kind of geospatial data instead of relying on a few SaaS API providers could provide an environment that promotes creativity. As an example, the creation of a watercolor map by a company called Stamen showed the possibility of doing very creative cartography with OpenStreetMap data. And we could have more kinds of these creative projects.
How PMTiles Is Serving the Underserved
Today, most base maps are created for a commercial audience. Companies that make base maps offer a one-size-fits-all solution to serve a larger section of the audience. For instance, since most people drive cars, the commercial priority for many base map publishers is to make sure the map shows information that is relevant to car drivers. For specific use cases such as a developer trying to make an application that is not for car drivers, they are largely limited by the unavailability of the base maps that fit their purpose.
Thankfully, with the development of PMTiles, it is becoming possible to create more kinds of maps with different value systems. This is a relief for audiences such as nonprofits, journalists, and the public sector who are often underserved by external map APIs that come with vendor restrictions or unfavorable price points.
Relevant podcast episodes
In Conversation
A Backwards Path Into Web Mapping
Daniel: Brandon, welcome to the podcast. You’re the founder and creator of Protomaps. How did you get involved in web mapping?
Brandon: I’ve been doing web mapping for about 10 years, since I graduated college, and I have a backwards path into this world. I studied computer science, and the first thing I ever did that was geo-related was make a biking map of the city I was living in. I knew nothing about GIS — I didn’t know what a shapefile or Mercator was. Through that I learned about OpenStreetMap, from OSM I learned about web mapping, and from web mapping I learned about GIS. A lot of folks come through a geography degree and then learn programming; for me it was the reverse — I wanted to apply my software engineering knowledge of data structures and algorithms to GIS problems.
Daniel: What was the biggest barrier to entry for you?
Brandon: I’ve always lived in the web mapping world, where everything is just Web Mercator and that’s good enough. The advanced GIS concepts — state planes, advanced projections — are still quite alien to me. The biggest challenge for me was the data side: the gap between open data from government portals and commercial data. Every single person uses Google or Apple Maps and sees that the data is so great, but there’s no real way to reproduce that yourself. There’s a huge gulf between the data we’re accustomed to in our daily lives and what an individual developer can accomplish.
The Scarcity of Base Maps
Daniel: The Protomaps website calls it “a serverless system for planet-scale maps.” What is the opposite of that?
Brandon: The opposite of a serverless system is the status quo of how we deploy geospatial software. When we talk about a server, we often mean something like a GeoServer — a piece of software you rent out on a Linux machine. A lot of the barriers to entry in web mapping are just the complexity of running servers and maintaining all that software over years, often by organizations that aren’t software companies — local government agencies, nonprofits, hobby projects.
Daniel: But there’s a little company called Google — three lines of JavaScript and I’ve got a map. Why do I need this?
Brandon: A lot of the motivation is being able to expose the guts of that. Google Maps as an API is completely opaque — it’s a black box. You can’t scrape the data or cache it; that’s against their terms of service. I like the word “scarcity” — when you want to use Google Maps, they charge you for every 1,000 people who visit your page, and not all projects have that budget. Traditionally, being able to create and deploy a global base map has been the capability of very few multi-billion-dollar companies — Google, Apple, Microsoft, Facebook. The goal of Protomaps is to make that available to anyone — a hobbyist, a one-person company, a small company that wants a more flexible solution.
What PMTiles Is
Daniel: Can we draw a comparison between PMTiles and Cloud Optimized GeoTIFFs?
Brandon: It is exactly the same thing — “serverless” is almost synonymous with “cloud optimized.” If you’re a fan of Cloud Optimized GeoTIFF, you should also be a fan of PMTiles, because PMTiles is a generalization of the same concepts to any kind of data, especially vector tiles. You have a single archive — a PMTiles file — and you can access individual tiles through HTTP range requests. The traditional ZXY addressing, with folders of tiles, doesn’t scale well: a typical base map at 15 zoom levels is in the hundreds of millions of individual tiles, and it’s almost impossible to upload 200 million files. PMTiles is a single file containing all the data. The technology for range requests was actually built for video — if you post an MP4 on S3, instead of downloading the entire movie you access the frames through range requests. PMTiles is that same idea, but moving in space instead of time.
PMTiles vs MBTiles
Daniel: How is this different from MBTiles?
Brandon: MBTiles is the same concept, but built on top of SQLite — the most deployed database in the world. But it’s an approach that requires a server: people rent a Linux server and run a program that translates MBTiles requests into HTTP requests. PMTiles is an adaptation of the same overall concept but removes that server part. I intentionally named PMTiles to sound similar to MBTiles — it helps communicate what the file format does. My focus for now is on vector tiles; for a lot of raster use cases, Cloud Optimized GeoTIFF might be a better choice.
CDNs and Performance
Daniel: Are there restrictions on how many users can access a single PMTiles file?
Brandon: There can be potential performance issues hosting it on S3, and one solution is putting a CDN — a content delivery network — in front of it. A CDN is a global network of servers run by Amazon or Cloudflare, where the data comes from the closest geographical node to the user. If I have a user in Denmark accessing data in California, it might take 400 milliseconds; with a CDN node in northern Europe, maybe 50 milliseconds. It’s also a load balancer — if your web map suddenly gets super popular and is on the front page of all the geo blogs, you shouldn’t have to rent more servers for that one occasion.
Creating PMTiles with Tippecanoe
Daniel: How would I make a PMTiles file — say from a shapefile of country outlines and one of the global road network?
Brandon: I’ve integrated PMTiles into a tool called Tippecanoe, originally developed by Mapbox and now under continued development by Felt. Tippecanoe can convert GeoJSON datasets into PMTiles, so you’d first convert your shapefiles to GeoJSON, then run them through Tippecanoe. What Tippecanoe is really good at is generalization — figuring out the right level of data to drop at each zoom level so the tiles aren’t too big. You can also merge two or more PMTiles files into one with a tool called tile-join. To consume them, I’ve focused on two libraries — Leaflet and MapLibre GL JS — with a plugin you install that lets you point your JavaScript at a PMTiles URL in your public S3 bucket.
The Vision: Diversity in Cartography
Daniel: Where would you like this project to go?
Brandon: I’d like to see a lot more diversity in cartography on the web. If we can commoditize the serving of any kind of geodata tile, then instead of only having Google Maps or a couple of SaaS API providers, we should have an explosion of possibility. One of the things that inspires me most is Stamen Design — more than a decade ago they created maps like Toner and Watercolor, which showed people the possibility of really creative web cartography with OpenStreetMap data. In a world where maps are scarce, all web maps have to be built for the lowest common denominator — usually a commercial audience. Most people drive cars, so the priority for a map publisher is to show highways and gas stations. If you’re making an application where you don’t care about car drivers, you’re limited. I’m hoping that controlling more of the cartographic toolchain will make more kinds of maps, with different value systems, possible.
Daniel: What signals tell you you’re on the right track?
Brandon: I’m a solo developer on the project, and where I’ve seen feedback I’m really happy with is the kinds of organizations adopting PMTiles — nonprofits, journalism, the public sector. Those are audiences often underserved by a Google Maps or another map API, with vendor restrictions and price points that aren’t great for what they want to do. It’s a really good signal for my goals if the markets that have been underserved by map technology are adopting Protomaps.
