Cloud-native geospatial, range requests, chunks, COGs and COPCs … [ insert confusing acronym here ]

Sometimes It feels like we need to learn a whole new vocabulary and if you have been doing #geo for a while you might be wondering how much of this is actually going to impact me. What bits of this are the ones that I need to know about?

I don’t think that anyone is going to be talking about cloud native in 10 years, in the same way, no one talks about digital cartography or computer analysis because where else would you do your cartography? And how else would you do your analysis?

Maybe the names won’t be as important but the concepts will and while this episode is focused on Geoparquet it does so within the context of cloud-native geospatial – and this concept is not going away!

You can connect with Kyle Barron here https://x.com/kylebarron2 or here https://kylebarron.dev/

If you want to learn more about cloud-native geospatial I can highly recommend these episodes

• https://mapscaping.com/podcast/cloud-optimized-point-clouds/
• https://mapscaping.com/podcast/introduction-to-cloud-native-geospatial/
• https://mapscaping.com/podcast/planet-scale-tiled-maps-without-a-server/
• https://mapscaping.com/podcast/what-is-modern-gis/
• https://mapscaping.com/podcast/the-planetary-computer/

I am working on a new project called https://quickmaptools.com/ like the name suggests it is a bunch of browser-based map tools. So far we have created several different conversation tools and will continue to add more to the list. Check it and let me know what you think!

In Conversation

Cloud Engineering at Development Seed

Daniel: Who are you, and what is cloud engineering? There’s no “geo” in that title, but you work for a geospatial company.

Kyle: I’m a cloud engineer at Development Seed — it’s a bit of a catch-all title; I don’t deploy as many services to the cloud as some people with that title. I work with a lot of open-source projects to help our clients; we’re a geospatial services firm that works with NASA and other impact-focused clients. Some of my work is enabling adoption of cloud-native geospatial file formats and interoperability with open-source projects, helping scientists do more processing on large datasets. At Development Seed we think the cloud is really the future of working with large datasets. In older ways of working you’d download a satellite scene or a large vector file to a local computer; now that file sizes are much larger, it’s easier to use resources hosted on cloud providers, which lets people scale up workloads far better than going out and buying their own computer.

What Is GeoParquet?

Daniel: What is GeoParquet?

Kyle: GeoParquet is a new file format for storing geospatial vector data that builds on the success of an existing format called Parquet. Parquet, in the non-spatial sense, has been around for about a decade, steadily gaining steam. It stores tabular data — think of Excel, a table that’s a collection of columns, some numeric, some holding text. Extending that to a geospatial context, you have a column of geospatial features where each geometry is associated with one or more properties. So GeoParquet isn’t really a new file format — it uses the Parquet format and standardizes how to store geospatial metadata in it. It’s very similar to how cloud-optimized GeoTIFFs and GeoTIFFs were both built on the success of the TIFF format. By building on top of Parquet, several of GeoParquet’s best characteristics we get for free — Parquet is itself a cloud-native file format.

Spatial Indexing: Why Vector Is Harder Than Raster

Daniel: Cloud-optimized GeoTIFFs solved this for raster. Why can’t we just import that same knowledge to vector data?

Kyle: There’s a real difference between indexing raster and vector data. Indexing raster can be relatively easy because the pixels are small compared to the extent of the dataset and they’re regularly gridded — you know all the pixels are in a row in a specific order. Vector data is much harder because it can be irregular. If you have long rivers or flight paths that stretch over half the world, it’s really hard to organize those geometries so all the nearby data is sorted close to each other.

Daniel: What about vector tiles?

Kyle: Vector tiles are a really great technology for viewing data, but they make it harder to work with analytical data. With vector tiles you clip geometries to a tiling grid, so if you look at one geometry in one tile you don’t know its full extent. For a coastline near New York City, the full extent includes many more tiles, and it’s hard to connect which other tiles you’d need to load to reconstruct the full original geometry.

Grouping vs. Indexing Every Geometry

Daniel: Other formats — flat geobuf, GeoPackage — have implemented spatial indexing. What’s different about GeoParquet’s approach?

Kyle: Flat geobuf — a really good format for its own use cases — indexes every geometry individually. That has the downside of making the spatial index very large if you have many geometries, and it prevents effective compression, because the index has to reference every geometry individually. GeoParquet instead implements grouping, like cloud-optimized GeoTIFF does — it indexes a collection of geometries at a time. That scales more effectively to larger sizes, and each group can be compressed as one. GeoParquet enforces the concept of groups, but users choose how many rows per group and how to sort the data within them. The upcoming GeoParquet 1.1 specification defines a way to store the spatial extent of each group, but it doesn’t prescribe a specific sorting method — which leaves room for future innovation. On the reading side there’s no extra complexity: from just the metadata you fetch at the beginning, you see the bounding box of each group and can implement spatial queries the same way regardless of how the data was sorted.

The Overture Use Case

Daniel: Let’s put this in a use case — the Overture dataset.

Kyle: Overture builds on top of OpenStreetMap. Their buildings layer combines OSM data with AI-generated sources — a single layer with 2.3 billion rows spanning the entire globe, hundreds of gigabytes. In flat geobuf it would be really tough to index all of that in a single file; loading the spatial index would take a lot of time. Overture saves all their data in GeoParquet and sorts it — they actually use quad keys to organize it, which still works with the specification. I’m working with members of the Overture team on a website to access their data that reads Parquet directly, without a server. Passing in a bounding box around Boston, we read straight from cloud storage to the web browser in about 15 seconds — it read the metadata to see which regions of data were in which files and locations, then figured out which portions to fetch. It’s common in GeoParquet to partition across multiple files as well as within a single file — multiple machines can write to separate files independently, which makes it faster to handle very large data.

Compression and Why It Matters

Daniel: Why is compression an interesting thing with GeoParquet?

Kyle: GeoParquet includes really efficient compression, which makes it much faster to move data between systems over a network. There are use cases where you don’t care about spatial partitioning at all and GeoParquet is still the best format because it’s the most compact to download. I work on a geospatial data visualization library in Python called Lonboard that moves large quantities of data from Python to JavaScript for interactive browser maps. A common example: three million points in GeoJSON is about 500 megabytes uncompressed, whereas as GeoParquet it’s only 30 to 40 megabytes for the same data. Part of that is because Parquet is a columnar format — all the data in a single column is saved next to each other, and data tends to be more similar within a column than across a row, so it compresses far better. GeoParquet tends to be about half the size of flat geobuf even when flat geobuf is compressed. There is a trade-off: there’s internal chunking, so you don’t have to download the whole file to access a chunk — but if all you want is a single row, the compression advantage is outweighed by the fact that you didn’t want much data.

GeoParquet and GeoArrow

Daniel: How does GeoParquet relate to something called GeoArrow?

Kyle: GeoParquet has a really symbiotic relationship with Arrow and its geospatial counterpart, GeoArrow. GeoParquet is a file format, and when you read a file format you have to read it into something — most Parquet implementations read and write data into Arrow. The strength of Arrow is that it lets programs share data efficiently at a binary level instead of a text level. Historically, if you were a Python data scientist with data in GeoPandas and wanted to read and write with GDAL, GeoPandas had no way to share data with GDAL at a binary level — it would create a GeoJSON feature for each row, converting from binary to text and back into a binary representation. In GDAL 3.6 and later, GDAL added support for reading and writing data from its drivers into the Arrow format. With recent changes to GeoPandas you can now move data as binary directly — reading GeoParquet into GeoPandas via GDAL became 33 times faster, and reading flat geobuf and GeoPackage 23 times faster, all by agreeing on this binary format.

Where GeoParquet Fits — and Where It Doesn’t

Daniel: Is GeoParquet a good choice for transactional use cases — returning one or very few items from a query?

Kyle: GeoParquet isn’t a silver bullet. It’s really good for analytical use cases, but it isn’t designed for transactional ones — getting a single record, or updating and modifying data. There’s work being done to improve updating Parquet files, but in general it’s easier to use existing tools like PostgreSQL and PostGIS for those. GeoParquet is really focused on supporting use cases that haven’t been as well supported until now — large datasets, this cloud-native architecture — and it complements a lot of existing tools instead of replacing them. Data providers definitely see it as a pro: looking at Overture, whose buildings dataset is massive, it simplifies their storage a lot because they don’t need servers or databases maintaining the data. The trade-off is that some users have to learn new tools, so ideally the friction is reduced as much as possible — by integrating GeoParquet support into GDAL and building helper tools, like the new Overture command-line interface. I’m still really excited about a future where data providers can share large datasets without servers, and where you can decouple how you store data efficiently from how you represent it in memory efficiently.