In this episode, the discussion revolves around cloud-optimized point clouds. Our guest is Martin Dobias, the CTO at Lutra Consulting. Coming from a background in computer science, and a passion for geospatial, Martin has been part of a team that has done a ton of interesting work in the open source geospatial world. Today, he shares about their latest, state of the art developments in working with point clouds on the web; Cloud Optimized Point Clouds (COPC).
What Does Cloud Optimized Point Clouds Even Mean?
Point clouds are sets of individual points plotted in 3D space. They are typically very large datasets, as they must capture a real space in great detail. For instance, point clouds for a whole country may easily be several trillions of points, and many terabytes of data. Handling these large datasets on the web requires a lot of bandwidth to download and process.
The files for Cloud Optimized Point Clouds are structured with indexes for each part of the dataset. The index structure makes it possible to stream only the parts of the data that are required, without having to download the entire dataset.
What is Point Cloud Indexing?
Point cloud indexing structures a point cloud file, making it possible to find any particular point of interest in the file without having to scan through the entire dataset. COPC files are internally indexed using a 3D structure of cubes called octrees. One part of the file is the data itself, and the other part contains the hierarchical information of where to find each cube.
At the root level of the hierarchy is a single cube, which at the next level is split into eight smaller cubes. The splitting continues subsequently up to the highest hierarchical level. As the cubes get smaller, they contain a smaller amount of data, which saves bandwidth if a user is only interested in a small part of the data. It is similar to traditional tiling, but with the added 3D context.
How COPC Files Are Accessed Using HTTP Range Requests
Range requesting is a feature of the HTTP protocol used to access information more efficiently from web servers. Instead of a server sending an entire file to a browser, the range request feature allows the browser (client) to define a specific part of the data that the user is interested in. Subsequently, only the requested part is sent by the server.
For cloud-optimized point clouds, the server will simply go through the hierarchy and find the cube or multiple cubes that satisfy the request, and return these. This process is much faster than having to search through the entire dataset. Moreover, only the point cloud cubes that satisfy the browser request are sent by the server, which reduces the bandwidth used.
Converting LAS Files to COPC
LAS is a standard open source format for point cloud data interchange. However, it is less efficient to work with on the cloud since the data in it is not indexed. This means the whole LAS file must be loaded before it can be queried, sidestepping the efficiency we see with COPC indexing.
Conversion of LAS to COPC can be done in QGIS using Entwine. Practically, when a point cloud file is loaded to QGIS, it is automatically converted to COPC. QGIS structures the data automatically in order to make operations more efficient as opposed to working with unorganized datasets.
When point cloud files are converted to COPC in QGIS, the new COPC file contains all the information in the original dataset. Unlike other software that may discard some information when processing a file, for COPC nothing is discarded. This makes the COPC format great not only for visualization purposes, but for analysis as well.
What Infrastructure is Required to Serve COPC Data?
Serving cloud-optimized point clouds does not require any special infrastructure between the server and the client. It is easy to host and get the data to the client without complex infrastructure, i.e. there is no need for something like GeoServer, MapServer, or QGIS Server. Just having the COPC data in blob storage somewhere is all the infrastructure that may be needed.
Compatibility of COPC Format
A compressed LAS file is called a LAZ file. COPC is much like a LAZ file. This means that applications that accept these formats will also be able to work with COPC files without having to implement special support. The only difference is that they will not be able to use the extra features of internal indexing in the COPC file.
Where Can You View COPC data?
QGIS offers support for viewing COPC – both stored locally on your device, or remotely in the cloud. Using a link that points to a server containing COPC data, QGIS will load the data on demand according to the queried range. The data is further cached in QGIS, which makes subsequent data loads and views much faster.
There are also a couple of projects coming to life that explicitly support cloud-optimized point clouds. An example is the web viewer built by Hobu. With the link to a COPC data server, the web viewer will fetch the relevant COPC files and render them in the browser.
The PDAL Library
PDAL (Point Data Abstraction Library) is a library that contains a set of tools for working with point cloud data. In the QGIS environment, many users are familiar with PDAL’s feature for the data access of point clouds. Many of the library’s other functionalities are unknown to a lot of users. It contains a dozen features to classify, filter, export, convert point cloud to raster or meshes, amongst others.
The main reason why many functionalities in PDAL are not popular among ordinary users is due to the complexity in using it. The PDAL library uses pipelines that need to be crafted manually when working with point cloud data. While this may work well for advanced users, ordinary users find it a bit too complicated.
After a successful crowdfunding campaign, Lutra Consulting and several other partners are working to reduce this complexity, and make the functionalities in PDAL more user friendly. The project seeks to build a simple integrated toolbox within QGIS for point cloud data processing. The same way there are integrated toolboxes in QGIS for working with vector or raster data, there will also be one for working with point cloud data. These developments may be expected across the next two QGIS releases; in February and June of 2023.
What is the STAC Protocol?
STAC (SpatioTemporal Asset Catalog) is a protocol for easy access to spatial and temporal data. It makes it easier to index, discover, and work with geospatial information. STAC is commonly used with satellite imagery but recently it is increasingly being used for distribution of point clouds as well.
How Were Point Clouds Streamed Before COPC?
No doubt, before COPC there were some existing formats for streaming point cloud data. In the open source world, one of them is the EPT format, built by Hobu Inc. The EPT format closely compares to raster tiles, but for point clouds. It is structured in a big directory with individual files (tiles). COPC files have an advantage over EPT format, as opposed to having thousands or even millions of files in a folder structure, COPC is just a single file – which is much easier to work with. In the proprietary world there are a couple of formats as well, one being the I3S format from ESRI, which supports point cloud data and other 3D data. There is no doubt we will continue to see explosive growth supporting point cloud data management. Stay tuned with the MapScaping Podcast to make sure you stay current on the latest and greatest developments!
PDAL – Point Data Abstraction Library
In Conversation
From Quantum GIS to Lutra Consulting
Daniel: Martin, welcome to the podcast. Would you introduce yourself and let us know how you got involved in geospatial?
Martin: Thanks for having me. My background is in computer science, but I always liked hiking and maps in general. Within my father’s small company I helped him do some programming with GPS receivers and smartphones — that was around 2005. Later, during my university studies, I had a project where I wanted to do something with maps, and my advisor pointed me to a very early version of QGIS — I think 0.4 or so — and said, rather than doing a project from scratch, why not try to improve something there. It was fun, and since then I’ve been in geospatial doing programming. Since 2013 I’ve worked with Lutra Consulting, where we do lots of geospatial work.
Daniel: Was that back when it was called Quantum GIS? And what feature did you add?
Martin: Exactly — Quantum GIS. It’s been a long time, but I think it was something like making the application able to accept command line arguments with X and Y minimum and maximum, so when you open the application it would be zoomed into that area.
What “Cloud Optimized” Means
Daniel: When you hear “cloud optimized point cloud,” what does the optimization actually mean?
Martin: The optimization for the cloud is mainly about the support of streaming. With point clouds, one important thing is that these are typically really large datasets — one square kilometre can easily be many millions of points, and for a whole country we’re talking about trillions of points and many terabytes. With that size of data, downloading the whole dataset would take a load of bandwidth. Streaming is the great thing — you just download whatever you need from the dataset, skipping all the rest. That’s what the cloud optimized point cloud format is all about.
Range Requests and the Octree Index
Daniel: Can you explain what streaming means and how range requests make it possible?
Martin: Streaming, as I think about it, is like streaming audio or video — out of one large file you skip to the piece of data you’re interested in and download just a small amount. Range requests are a feature of the HTTP protocol used on web servers: if the server supports it, you can tell it, rather than giving me the whole file, give me a subset starting at this point and this many bytes long. The complication with geospatial data is that we’re dealing with 2D or 3D data, while audio or video is really just one stream in time — so that needs to be addressed by a smart representation in the file.
Daniel: So this brings us to the indexing of the file itself.
Martin: Indexing means giving structure to the file. Standards used for interchanging point clouds, like LAS or LAZ, you can think of as one big bag of points — there’s no way of telling where to find any particular point; you may need to scan the whole file. With indexing, things are different. We use a structure internally called an octree — a three-dimensional structure you can imagine as a cube with different levels. At the root level there’s one cube; at the next level it splits into eight smaller cubes, and so on. When the client needs a small amount of data, we go through this hierarchy, find the cube or cubes that satisfy the request, and send only a small amount of data. These cubes are already built into the file — one part is the data itself, the other is the hierarchy information.
Daniel: If I converted a LAS file to COPC, would I see a huge change in file size?
Martin: If you worked with a compressed LAZ file, you’d see a slight increase — maybe 15 to 30%. But if you started with an uncompressed data file, the actual file size would be much smaller, thanks to the compression.
Almost No Infrastructure Needed
Daniel: One magical thing about these streamable formats is they need very little infrastructure — I could put a COPC on a web server, take the URL, and it would just work in QGIS without a GeoServer or MapServer in between?
Martin: Exactly — that’s the beauty. It removes a lot of the burden from the data host. They don’t need to run a huge expensive server in the background just to serve out data — it could just be blob storage somewhere, and that’s all the infrastructure they need. There’s one more feature: backwards compatibility. A cloud optimized point cloud is essentially a LAZ file, which is already an industry standard, so even after you convert your data, loads of applications still work with the file — they just wouldn’t be able to use the extra features of the internal indexing.
Daniel: So is there still a use case where you’d choose a plain LAS file over a COPC?
Martin: There are still use cases for the non-indexed LAS format, especially when preparing point cloud analysis workflows where you don’t need optimized access — if you know you’ll be processing all the points anyway, maybe in multiple stages, you can save some processing time by not doing the indexing into COPC.
Reading and Writing COPC
Daniel: What reads and writes cloud optimized point clouds right now?
Martin: You can use PDAL to read COPC files, specifying the spatial resolution and area of interest — useful for data analysis. QGIS now has support for loading cloud optimized point clouds, so you can load and view those files, and more and more projects support them, including a web viewer built by Hobu where you point it at a URL and it visualizes the point cloud in the browser. For converting, QGIS internally uses a small command line tool called Entwine, also built by Hobu — so when you load any point cloud file in QGIS, it does the indexing to COPC automatically, because to show point cloud data efficiently in 2D or 3D it needs the indexing anyway. And the COPC file includes everything that was in your original dataset — the classification, intensity, number of returns, return number — so in theory you don’t even need to keep the original LAS file. It’s great for both viewing and analysis.
Daniel: How would I read a COPC from a link in QGIS?
Martin: You open QGIS, open the data source manager, get to the point clouds tab, and instead of pointing it to a local file you switch to the mode where you set the URL of the COPC on a web server. QGIS queries the server, finds out if it can deal with range queries, and if it’s a valid COPC file it starts streaming and loads the data on demand. Internally QGIS caches the data, so subsequent views are faster. As with cloud optimized GeoTIFFs, this is read-only access.
Bringing PDAL’s Power into QGIS
Daniel: Lutra Consulting just finished a successful crowdfunding campaign. What functionality are you going to build?
Martin: Right now QGIS can visualize the point cloud, do a cross-section profile, some measurements and simple exporting — but for most people point clouds are more of a raw data type, and they want to do processing. We already use the PDAL library for data access, but that’s maybe just 5% of what the library offers. PDAL is a state-of-the-art open source library with dozens and dozens of filters — filtering, classification, exporting derived data like rasters or even meshes. Relatively few people know about it because PDAL uses pipelines, small JSON files you craft manually, which is fine for advanced users but too complicated for ordinary users. We want to take all that functionality and build a simple integrated point cloud toolbox within QGIS — so next to the existing algorithms for vectors and rasters, you’d have a point cloud toolbox where you can remove noise, do classification, spatial filtering, and create raster digital elevation models.
Daniel: You also mentioned working with large datasets more easily — virtual point clouds, like GDAL virtual files?
Martin: Absolutely. Point clouds are typically delivered in tiles — maybe one square kilometre each — and for a larger area you get hundreds or thousands of files. It’s impractical to load those one by one into QGIS. If you can create a virtual point cloud referenced from one single file, the user experience is much better for both visualization and processing. It’s a kind of meta hierarchy over the whole big dataset. The viewer initially shows the extents of individual tiles, and as you zoom closer it loads the real data from the COPC files on the server. We want to support combinations — a remote virtual point cloud referencing remote files, a local file referencing remote files, or a local file referencing other local files. And we’d like to stay close to STAC — the SpatioTemporal Asset Catalog — so you could easily export the equivalent definition of a virtual point cloud in the STAC protocol.
Cloud Optimized Vectors and What’s Next
Daniel: Could we copy the same idea and make every vector format cloud optimized?
Martin: That’s a hard topic. Nowadays there are lots of discussions about what the cloud optimized format for vectors would be. Having to deal with lines and polygons really complicates things. The complication is mainly with the structure of the geometry — with point clouds, if you need to lower the spatial resolution you just ignore some points and use a subset. It’s not clear how to do that with lines or polygons; there are generalization algorithms to lower the number of vertices, but they all have limitations, and it really depends on the use case how you’d want the simplification to work. So it’s much harder than with rasters or point clouds, and I expect a lot will go on in that area.
Daniel: When can people expect this new QGIS functionality?
Martin: Our plan is to deliver it across the next two QGIS releases — 3.30 and 3.32 — coming out in February 2023 and June 2023. So within roughly half a year we’d like to deliver the new analytical tools and the virtual point cloud support. To learn more, you can find us on Twitter, and I’d also suggest following Howard Butler, the head of Hobu, and North Road, who are doing the crowdfunding work with us.
