Metadata Management in Unstructured Data

Our guest this week is Kirk Marple, the founder of Unstruk data. He began his career in geospatial with his first job, making maps and performing various geospatial analyses. He later transitioned into software development in the media space, but still doubled back to geospatial from time to time. Currently, Kirk is most focused on dealing with geospatial data at Unstruk Data.

What is Unstructured Data?

Unstructured data is everything from imagery to audio, 3D point clouds, documents, emails, and many other digital files. The term unstructured is a bit of a misnomer, since there is always a known schema or file format for all these file types. The term unstructured may refer more to how people view data since a lot of people do not think about the bits on the disk when viewing data, but are more concerned with the contents.

Metadata in Unstructured Data

The metadata of unstructured data provide a starting point for working with unstructured data. They can be classified into three levels:

First Order Metadata is the data in the header of a file. It is the bare minimum of metadata that one can get out of a file i.e., you can read the EXIF data of an image, but if you are unable to read the image, you will not know what was actually captured.

Second Order Metadata is the data that helps in reading the file and identifying its contents. In the case of images, models are used to detect objects and identify what was captured. Bounding boxes and their tags, often used in training machine learning models, are perfect examples of second-order metadata in images.

Third Order Metadata is data pulled from making inferences across a bunch of related data and linked databases. This data provides a framework for contextualization that creates edges, like in a knowledge graph, that connect something to something else. This can be thought of as the spider web that grows bigger as more edges are created, as more inferences are pulled.

Is Geospatial Data Unstructured Data?

Geospatial data is less unstructured largely because it has that extra bit of context- geographical location. It is common to have other information besides the location data in the EXIF metadata. For a phone or drone image you might get the speed, acceleration, camera angle, and a lot more information that gives geospatial data more structure. This makes it possible to get a tonne of information from a single file, just from the metadata.

Knowledge Graphs

Knowledge graphs follow the same principle as a relational database. While in a database there are tables with keys that link one table to another via related records, knowledge graphs morph and pivot on edges created from inferencing a bunch of data. They are more dynamic, and do not have the same challenge of constantly updating the schema like you would need to accomplish the same in a database. Knowledge graphs provide the flexibility of inventing new edges on the go and pivoting on any entity in the system to find all the things that they relate to.

How Far Out Should a Knowledge Graph Grow?

Knowledge graphs grow bigger and bigger when new connections are established from new data. The knowledge graph’s ‘spider web’ is theoretically infinite, since data enrichment is recursive. The spidering can continue on and on as long as more context can be found from an edge or link. There is a danger in having a never-ending spread of data. 

The risk itself lies in the concept of data enrichment being almost boundless. As such, excessive resources might be tied up to do enrichment of data that may never be needed by the customer. This places the customer’s needs at a critical position as the determining factor of how far out to grow a knowledge graph. 

A good general rule is to start cutting off the spider if more data is coming in, but no new changes are apparent on the graph.

Dark Data

Dark data is data in the archives that is no longer being used. Unstructured data has a tendency to go dark fast. There is a growing stream of new unstructured data from drones, robots, and even mobile phones that make it easy to toss old data away in preference of the new data that is supplied. A company that does aerial surveys may do analysis on a day’s data, or a week’s data, but when the data ages out a little bit, it goes dark, and becomes obsolete. Structuring this data in a knowledge graph provides a way to look across years of data and begin to see trends and commonalities. It bridges the gaps between daily workflows, and historical analytics.

Machine Learning Models

Models are an essential part of building knowledge graphs. They are the backbone for building edges and links upon which further inferences can be made. In order to develop these models, a human has to be put in the loop to train the model, review, and then validate its results. 

Model training is a continuous process to make the model more accurate in detecting objects. If the model gets it wrong the inferences made on that result would be inaccurate as well.

Practically, there is no one all-purpose model. Models are trained for specific applications, and then used alongside each other in an ontology to detect objects as each has been trained. 

Today, there are third party vendors that provide models which are trained generically. These can be used as a starting point to filter out results, but may not be useful in identifying more specific things. 

For instance, if a generic model identifies a feature as a building, another model that is trained to differentiate buildings can be run to identify whether the building is a shed or garage. More specific models for features like windows or doors can further be strung together to produce even more specific results.

Layering models in this way, defined in a parent child relationship, is helpful in cost management. For instance, instead of running a window model on a whole image where the buildings cover only 10% of the area, filtering out will remove the 90% area that has no buildings, creating less data to be processed by the next, more intensive model. This carving out data optimizes cost and performance.

Taking Metadata Management to Edge Computing

Edge computing is a concept of pushing computing closer to the data. In satellite platforms, it is used to reduce the amount of data being sent back to Earth by doing computations out in space, removing the bad pieces, and keeping and sending only the useful data. 

In IoT, more and more data are being collected from a variety of sensors. Currently, some companies are starting to use edge computing to reduce the bulk of data sent from these sensors when data is captured. This is mainly by processing metadata and moving through first order to second order metadata at the source.

There is a danger in the shift of the kind of metadata that is being created at the source. 

If something is not correct right at the source, it might be impossible to come back to it since it is now about metadata management and not file management, which is much more fickle. This is why continuous training of the models is invaluable to dial in accuracy. A human review, and approval process is a critical element of the loop.

New Possibilities with Unstructured data

Machine learning models and knowledge graphs provide a gateway to new products that can be built on processing unstructured data. Having a semantic search is one of the possibilities. 

This can prove to be very useful for industries like oil and gas, or real estate. A real estate inspector can use photos of rental apartments to pull a tonne of information about the facilities i.e., results of the last inspection, crime data, any reports and much more.

A web catalogue service that is searchable on the web can help to bring these capabilities to the public. 

This might be an open, public API that serves as a crowdsource catalogue for publishing data into a knowledge graph, which then is made accessible to the public. This catalogue can be a source of useful links that expose unobvious analytics. By making it public, the knowledge itself becomes open, and enables the next person down the line to make use of it. 

Recommended Podcast Episodes 

In Conversation

From Media Files to Unstructured Data

Daniel: Kirk, welcome to the podcast. You’re the founder and CEO of Unstruk Data. Before we dive in, can you introduce yourself and tell us how you got involved in geospatial?

Kirk: I founded Unstruk Data, and I’ve been a long-time software developer. I actually remembered yesterday that I was dealing with geospatial data even in my first job — working with maps on laserdiscs. That’s how far back it goes. I’ve been more in the media software space, but dabbled in geospatial from time to time, and now I’m a bit more focused on it.

Daniel: What is unstructured data for you?

Kirk: Everything — imagery, audio, 3D geometry, point clouds, documents, emails. A broad set of file-based data. Coming from the video and media space we just called them files — file-based workflows.

Daniel: Every file has a well-defined structure. Why do you call it unstructured?

Kirk: It’s partly a marketing thing to differentiate it from the structured modern data stack world. It is a bit of a misnomer — there’s always a schema or known file format in these file types. But for a lot of people, when they see a document or an image they’re looking at the content, not thinking about the bits on disk.

First, Second, and Third Order Metadata

Daniel: You talk about first, second, and third order metadata — can you walk us through that?

Kirk: This is my own terminology to structure how we look at the data. First order metadata is what you can get just by opening a file — file headers, EXIF, XMP. The bare minimum, without doing much else. Second order metadata is what you get from actually reading the data inside the file: running object detection on an image and getting tagged bounding boxes, or pulling terms out of a document. Third order metadata is inference: you see a conveyor belt in a maintenance photo, then you link it to the conveyor belt record in someone’s SAP database. That contextualisation across data sources — that’s the third order, and that’s when you really get into machine learning and more complex inference. We think in knowledge graphs — everything is an edge connecting something to something.

Daniel: Is there any limit to how far you can spider out? Could this just carry on indefinitely?

Kirk: It’s not that different from what Google does with their knowledge graph for the web. Theoretically you could build a spider. I was doing this for podcasts — an RSS feed has MP3 files, transcribed terms become second order metadata, the show notes are full of links, and you spider out from there. Data enrichment is recursive — as long as you can find a link to something, it’s theoretically infinite.

Daniel: So you could use geospatial as another anchor — if I detect pipes in a drone image, I now know roughly where they are in the world and can cross-reference with my asset database.

Kirk: Exactly — that’s something we’re working on. From the drone metadata you know the camera frustum, so you can project that onto the real world, get a geofence, and then cross-reference against a database. “Is this Pipe A or Pipe B?” If you can automate that link, you can pivot on any asset — show me all images of this pipe in the last 30 days — without humans having to make every connection by hand.

Commodity Computer Vision and Model Ensembles

Daniel: Tell me about Azure feature detection and the commodity computer vision services.

Kirk: Computer vision has commoditised a lot over the last several years. You can get models off the shelf — Azure Cognitive Services Vision, AWS, third-party vendors — where it’s just an API: send an image, get metadata back. There are also no-code services where you provide a few hundred annotated images and they handle training and deployment for you. If you have a data science team you can go right to the metal and write Python yourself. There’s a wide range from easy to hard.

Daniel: So I send images to Azure and it tells me “pipe, bucket, house” — labelled?

Kirk: Yes — but the off-the-shelf models are trained generically. Run Azure Cognitive Services on a drone image and it’ll say “aerial, outdoor, building.” It’s useful for filtering, but not for identifying the specific things you care about. That’s why we look at ensembles — a rough-cut model says “this is a building,” then you run a more tuned model to differentiate sheds from garages. You layer models. There’s not one model to serve all.

Daniel: So you found a house, now run the window model and the door model.

Kirk: Exactly — and it helps with cost management. Each model run costs a few cents. If only 10% of a drone flight has buildings in it, you don’t want to run the window detector on the other 90%. I always say “smaller haystacks from bigger haystacks.” Carve down the data you’re running each model on and optimise cost and performance together.

Knowledge Graphs and the Network Effect

Daniel: When you talk about this, it sounds like the network effect for data.

Kirk: Exactly. I got into knowledge graphs heavily about five years ago. I’d done a lot of database work — tables, foreign keys, schema migrations as the biggest pain point. Knowledge graphs are dynamic — you can invent new edges on the fly. You don’t need a schema migration to say “this entity is related to this other entity in a new way.” You can pivot on any entity and any edge, see what relates to it, and learn something. The ability to represent your data this way is what’s key for us.

Daniel: Can you do this with documents and audio too? Sentiment, topics, things like that?

Kirk: Yes — that’s where it gets really interesting. It’s about contextualising what you capture to real-world entities: people, places, things. The project that became my company started by taking podcast feeds, analysing them for entities — topics, people, organisations — and creating those links. The show notes for podcasts that have good ones become a huge source of value through linked entities. You can use it for discovery: “find me other podcasts that discussed this topic with this guest.” We had to build a web spider because it just keeps pulling in data.

Edge Computing: Metadata Without Files

Daniel: What does edge computing mean for first/second/third order metadata?

Kirk: Interesting question. We’ve talked to companies running live video capture on-site — little video boxes connected to a camera doing analysis right there. With them you almost jump past first order metadata because there isn’t really a file — it’s a stream. You jump straight into second order: they run ML on the device and send us back object detections. So we still do things with it, but we don’t have the original files. Edge computing ends up being more metadata management than file management. There’s a danger though: continuous training matters. You’re not training on the edge typically, so you need some data flowing down to validate “is my model even good?” We talked to a chicken processing plant — how do you know the video saw that a hanger was broken? They can send an alert, but you have to close the loop on whether that decision was right.

Dark Data and the Value of the Archive

Daniel: Who typically comes to you and says “I have all this unstructured data, structure it for us”?

Kirk: We’re early — just launched about a month ago — so we’re going out and finding people more often than they find us. Trade shows have been useful. One interesting case in geospatial was an aerial survey company. They’re savvy on photogrammetry and capture, but poor at data management — keeping data on SharePoint, not cloud native, no search. The common thread we hear is people look at their data with blinders on. They look at today’s data, this week’s data — but once it starts to age out, it goes dark. That’s the dark data concept, and it’s becoming an industry term. They’ve captured it but they’re not making good use of it. We provide a way to look across years of data and start to see trends, commonality — bridging daily workflows to historical analytics.

Daniel: So a lot of the value is in the archive — making connections between past and present.

Kirk: Everything is indexed geospatially and temporally and via a tagging taxonomy. We generalise tags into what we call observations — a person, a place, a real-world asset, or just a generic word. Observations come from document analysis, audio transcription, computer vision. Once you have them, you can see “I’ve been seeing these observations more this month” — trend analysis, alerting. People usually want one of two things: data discovery (user-directed) or data triage and alerting (automated). I uploaded a few thousand drone images last night and didn’t realise some were taken in Europe — looking at DJI files named DJI_1234 in a folder, there’s no obvious metadata. But once you process it through our system you see when it was taken, related captures around that time and place, clustering by content. All of that is non-obvious from a folder on S3.

Geospatial Integration and Data Enrichment

Daniel: Could you do this against a live API rather than a bucket of files — point it at a service and just keep pulling?

Kirk: That’s how it all started. We have a concept of a feed, based on RSS. Any API we can read becomes a feed — RSS, Spotify new releases, even a SQL database polled for new rows. We just added email support recently — drop in an .msg file or even a .pst, we crack it open and do document analysis. We could easily listen to Gmail or Microsoft Graph. Geospatial APIs with a polygon and a bounding box would fit right in.

Kirk: We also have a concept of places. Drop in an ESRI shapefile, we convert it to GeoJSON internally, get a geofence, and promote it to a place entity in the graph. Then we do data enrichment — call the Google Places API, layer in metadata. We’ve talked to Nearmap and a couple of satellite services about enrichment. Because of our eventing model and webhooks, anyone can build their own data enrichment: when an entity is created or a tag is added, we call a webhook, they call some other API, then push the result back into our graph.

Daniel: How do you know where to stop with all this spidering? You could keep enriching forever.

Kirk: That logic is the tricky part — I’ve created bugs that were infinite loops of spidering. We tend to cut off when enrichment stops producing changes — when you keep getting back the same data that’s already there. Data enrichment is almost boundless, so really it comes down to the customer’s actual question.

Who It’s For — and Who It’s Not

Daniel: Who is this not for?

Kirk: There’s a difference between the technology and the company. The technology is broad — it started as podcast discovery and could be used for media, entertainment, lots of things. But as a company we’ve focused on extracting insights from unstructured data that’s about real-world assets — there’s a geospatial element. We’ve had interest from medical research, scanning X-rays — there’s overlap, but no geospatial angle, so it’s not our sweet spot. We’re a small company; we can’t boil the ocean.

Daniel: Aren’t you worried Google might do this — their whole mission is indexing the world?

Kirk: What we do is a lot like Google’s knowledge graph, but they’re consumer-focused. I’d be more concerned about Palantir, C3, Cognite — companies focused on real-world things. But there are unique angles to us by leaning into the unstructured side. We want ISVs and companies to build on our platform — more of a Snowflake or Databricks pattern, a data platform people build vertical applications around. Property inspection is a great example: iPhone photos of rental apartments, automatically tracking “when did this sink break? was it broken at the last inspection?” — pulling in the emails, the prior reports, everything around it.

Daniel: Where can people go to learn more?

Kirk: We’re on the Azure Marketplace. The website is unstruk.com (a better one’s coming), and LinkedIn is the best place to follow the company and connect with me. I love talking to people about the data they have and the problems they’re seeing.