Collecting and validating geospatial data for every commercial location in the USA and Canada is not an easy task. It requires aggregation of data from multiple sources and formats. This data then needs to be validated and decisions need to be made about which data sources represent the truth in the case of conflicting data. Safegraph does this weighing datasets based on certain criteria and using a voting system.
- Safegraph Data Blog — Cool GIS data stories from SafeGraph
- SafeGraph Data Bar — If you want to try SafeGraph data, you can use the code Mapscaping for $500 free data, no credit card needed.
- Safegraph Docs Page is our extensive user-centric documentation and releases notes, data dictionaries, etc. so you can ramp up quickly when working with SafeGraph data.
- SafeGraph is on Twitter
- Auren Hoffman, CEO at SafeGraph Auren’s Twitter
- Ryan Fox Squire, Data Scientist at SafeGraph, Ryan’s Quora, Ryan’s Twitter
This episode is sponsored by HiveMapper
A platform that takes video and creates 3D mapping layers based on that data. The video can be from a variety of different sensors, does not need to be vertically looking down on the geography and each 3D output is georeferenced!
You are more than welcome to reach out to me on social media, I would love to hear from you!
In Conversation
From Neuroscience to Points of Interest
Daniel: Hi Ryan, welcome to the show, thanks so much for taking the time to do this interview with me. We’re in different time zones so I really appreciate that we’ve managed to work it out. You work for a company called SafeGraph, and unless I’m mistaken, you make ground truth datasets of the physical world. So this sounds like it’s got a lot to do with geospatial. But before we dive into all that, can you give myself and the listeners a brief overview of your background and how you got involved in this work?
Ryan: Thanks Daniel, and pleasure to be here. I have a science background — I went to graduate school, did a PhD in neuroscience actually. During my time at grad school I realized that what I enjoyed the most was working with data and communicating about data, and there were so many great opportunities to do that sort of stuff outside of academia. So I’ve been working in data science and product roles at technology startups for the last seven years or so. And the reason I got into geospatial data was because of SafeGraph. Like you said, SafeGraph is a geospatial data company with a key focus on building datasets about places and the physical world, and that was my doorway into the GIS world.
What Is a Ground Truth Dataset?
Daniel: So we’re building datasets about the physical world, and at the start I mentioned something about ground truth datasets. Can you put a few more words around that — what is a ground truth dataset?
Ryan: Absolutely. The concept of ground truth sort of comes from the machine learning world. If you want to build models, if you want to build artificial intelligence systems, all those systems need to be powered by access to highly accurate, highly clean, quote unquote ground truth data. If you’re going to train the model off of lots of examples, you want those examples to be accurate and correct. And so that’s broadly the problem that SafeGraph is trying to solve, which is: how do I get good data about the physical world?
Where the Data Comes From
Daniel: Maybe we could talk about how we get data of the physical world — that’d be a great place to start. I’m assuming this is not something that already exists; you go and make it out of presumably a whole bunch of other datasets, maybe you derive some data based on these other datasets that you’re gathering together. Can you give us a little bit of a sense of how this is made, and some of the data sources that you use or integrate?
Ryan: Absolutely. The GIS data world is obviously a big world, and so SafeGraph is really focused specifically on understanding places, or points of interest, in the physical world where people can spend time or spend money. In particular we’re focused on these commercial points of interest in the United States and in Canada — so these are all of the retail places, all of the restaurants, movie theaters, hotels, golf courses, all these places where consumers might go and spend time or spend money.
Ryan: And so then the question is, okay, where do we get data about those places? And the answer is a lot of different places. Half the battle is figuring out the sourcing of this data, and where can we get information about these places that we think will be valuable to our end users. For example, we do a lot of crawling and scraping of data off of the internet. These days all these businesses are putting information about themselves up on their own web pages, and so a big portion of that data comes from intelligently trying to crawl and explore the internet to find data about places. That’s a multi-faceted problem with a lot of complexity around how to do it in a way that’s scalable. It’s easy to build a crawler that will crawl a single website for a single restaurant; it’s much harder to build a system that will crawl hundreds of thousands of websites.
Daniel: I can imagine something like Google Places would be a very structured place to begin to find some of these points of interest, but not everyone is on Google Places. So there must be a lot of aggregation of data that happens in the background from websites, and I’m assuming councils or other local government organizations are also providing a bit of data to help solve these problems.
Ryan: That’s definitely right. Government sources are another major source of data. At the city level, at the county level, at the state and federal level, all these different organizations keep records about licensed businesses, and a lot of data about those businesses — geospatial information about those places, category information about those places. So that’s another big source of data that we use to put together these datasets.
Daniel: Maybe you could give us a rundown of what this dataset looks like. I’m assuming there’s a polygon, if we’re representing a building for example. So let’s start with that — we have a polygon, we have a shape. Where does that come from?
Ryan: From the SafeGraph point of view we actually start with the concept of a POI. Abstractly we’re trying to find what are all these points of interest that exist out there, and to start, those things are defined with basic metadata like the name of that place, the address, things like that. Then a big part of that dataset is understanding the core geospatial coordinates and geospatial shape of those businesses in the physical world. We often get those two different parts of the picture from different sources, and a big part of what we do is match those datasets together.
Ryan: So for example we might get information about a place’s name and street address from scraping their website on the internet, but their website doesn’t necessarily tell you any geospatial information about that place other than the street address. So we also have many other sources that we’re collecting to get that geospatial polygon data — as we mentioned, the city government might keep records of the building layouts and property layouts of different structures that we can get. Sometimes those data are correctly labelled with the right addresses and the right business names, sometimes they’re not at all. And so a big part of what we do is join these datasets together to try to give you a complete picture of: okay, here’s this business, here’s its name, here’s its address, and here are its geospatial coordinates.
Voting and Weighting Conflicting Sources
Daniel: I think we all understand now that we’re aggregating data from a whole bunch of different sources together to make this dataset. The big question for me is: what do you do, or how do you solve the problem of — let’s say we’ve got two different datasets and they’re saying different things. How do I know which is right, how do I validate that, and how do I validate things like that at scale?
Ryan: That’s definitely one of the hard problems that we are excited to be solving for our clients. The value proposition to our clients is: yes, you could go spend a lot of time and money to build this dataset yourself, and in many cases these datasets ultimately come from public and open sources — but it’s actually quite a large effort to put these things together, to join these things together, and to validate. When you get conflicting information, how do you know which is correct? We have source A that tells us that this coffee shop is at 555 Main Street, and another source says it’s 557 Main Street. So how do we decide what’s correct?
Ryan: There are a couple of different ways that we approach that. One way that I think makes sense is that often we’re getting data from not just two sources but maybe five or ten or a dozen sources. And if it’s the case that eleven out of those twelve sources say that it’s 555 and one of them says it’s 557, we think that’s good evidence that the address is probably 555. So you can sort of imagine a voting system in which we’re looking at many different sources and trying to trust what is the consensus. That’s one approach.
Ryan: Another approach is that we essentially have come to understand that certain types of sources are more accurate than others. So if we have conflicting sources, and one of those sources is that restaurant’s personal business website, and another source is some public review forum, we’re going to trust the restaurant’s personal business website over that public review forum, if those are the only two pieces of data that we have.
Daniel: It sounds like you’re starting to solve some of the same problems that Google has when they’re trying to give the best search result. It sounds like you’re using a voting system, where it quickly becomes a numbers game — if 20 say yes and one says no, then that’s a good reason to believe that it’s yes. And you’re also using a weighting system, giving more weight to a personal website, something that’s more directly connected to the object.
Three Products: Core Places, Geometries and Patterns
Daniel: Perhaps you could tell me about some of the attributes that are inside this dataset. We have a polygon and we have a whole bunch of other things — can you talk us through that?
Ryan: We think about our product in three categories. We offer three different types of products about points of interest, and we call those products Core Places, Geometries and Patterns. Each of those different products are all keyed on the concept of a place — the primary key is a unique place identifier in each dataset — but each dataset has different columns that are relevant. You can easily join those datasets together based on this primary key, but often, depending on your use case, you may or may not want all of those different products.
Ryan: The Core Places product is your essential metadata about that place: its name, its address, city, state, zip. We have category information about that place based on the North American Industry Classification System, the NAICS code, and we have some other metadata like the operating hours of that business. So there’s not necessarily a lot of geospatial data in that dataset, but maybe you don’t care about that for your use case.
Ryan: Then we have the Geometry dataset, which includes all the core geospatial data about that place. We have both a latitude longitude point of where we think that business is, and then we have the polygon, which is the full building shape that we believe that business resides in. And then we have some other information to help make that data more useful — these are getting kind of into the weeds, but for example we have flags that tell you things like, is this business inside another building, is this a standalone place, does this business potentially share a polygon with another business in our dataset? Depending on your use case you might care a lot about that, so there are some other flags that we include to try to make that geospatial data as useful and as context-rich as possible.
Ryan: And then the third dataset, which is what we call Patterns, is a little bit different. It’s a summary of human movement dynamics around these places — essentially a dataset about foot traffic in and around these businesses. That adds a whole other rich dimension to your picture of this business, because now you can see things like, in aggregate, how many people are coming to visit this business, what times of day do people visit, what days of the week do people visit, when people visit how far are they travelling to get there, what other places do they go to. So that gives you a rich picture about consumer behavior around these places.
Daniel: It sounds like an absolutely amazing dataset, and it’s really interesting that you’ve broken it up into those three different categories. Could you give us a use case? Let’s go with the geometry — what are typical users doing with this data? I come to you and I’d like to have all the business geometries.
Ryan: The main customers of our geometry data tend to be in one of two categories. The first category is we work with a lot of geospatial analytics software companies, including some very large companies. For example, one of our deep partners that we’ve worked with a lot is Esri. Esri is this incredibly powerful software suite that lets you do all these amazing geospatial analytics, and in many cases Esri has done an incredible job sourcing and pulling together their own datasets to make available to you inside the Esri platform. But we’ve partnered with Esri to take their POI datasets to the next level, so that when you’re working in Esri and maybe you want to have a base map that shows where all the points of interest are, SafeGraph data can now power that inside Esri, so that data is natively available to all Esri users. So that’s a class of customers that we have — these geospatial mapping software companies that want to be able to show POI on a map to their end users, and they can use our data for that, either with points or as polygons.
Daniel: Typically when people think about a company like yours collecting and aggregating this kind of data, I think the fear, or perhaps the assumption, is: someone’s tracking me again, and they’re going to use this data to track my movements against this dataset. Do you see a lot of the use cases around your data used in the ad tech industry?
Ryan: Ad tech is definitely another large segment that we sell to. As you mentioned, there’s a lot of interest these days — both a lot of innovation and a lot of concern and controversy — happening around location-based advertising. One of the ways that that works is, let’s say you’re a brand, let’s say you’re Target or you’re Starbucks, and you have an app that your users use, like the Starbucks app. It might be useful for Starbucks inside that app to be able to send you notifications or advertisements or coupons based on whether you’re close to a Starbucks or close to a competitor. So those kinds of use cases are very common, and definitely another large area that we sell to. The common use case in that context is, for whatever reason, the advertiser has access to some location data about their users and they want to deliver some location context-based advertisement to that user.
Daniel: I can completely understand why they want to do this, but I think the fear on the consumer side is this idea of being tracked all the time, and perhaps being overwhelmed with push notifications. Although I am hopeful — I don’t see a way around this being tracked, and I don’t want to dwell too much on the problem, but I’m hoping that people take datasets like what you’re creating and create some really positive experiences for users out there, instead of just constantly hammering them with advertising. So I really look forward to seeing the different use cases which come out of this, especially once the industry starts to mature and look beyond the borders of advertising.
Malls: Polygons Inside Polygons
Daniel: When we’re talking about geometries and the relationships between geometries, we talked a little bit about spatial hierarchy. An example of this might be having a shopping center and then having individual retail spaces in the shopping center. That sounds like a really difficult relationship to define when we’re talking about geometries. Can you talk a little bit about how you solve that problem?
Ryan: Definitely, it is a hard problem. A lot of customers of SafeGraph data work in retail analytics or retail marketing, and so they care a lot about malls and they care a lot about all these major brands that often exist in malls. So it’s definitely an area that our customers care a lot about and that we’ve spent a lot of time trying to work on. It’s a multi-faceted solution.
Ryan: One thing is that we can do a variety of bottom-up detections to try to understand whether places are part of malls or part of strip malls — things that give you clues, when a lot of businesses share addresses or share top level addresses. Another thing that we’ve done is we’ve manually gone out to figure out, okay, what are all of the major and minor malls in America and in Canada, and let’s just go spend some manual time curating all that data. So we’ve essentially hand drawn accurate polygons for all those malls, and then we have steps in our pipeline when we’re processing data to try to detect: okay, I know that this particular business is located at this particular point — is that point inside one of our malls that we’ve gone and drawn and curated? And if it is, then that’s probably the case that that POI belongs to that mall.
Ryan: And then a final thing that we’ve also done is that almost all malls, especially major malls, have their own web page and their own directory listing of places. And if you’re lucky, often they even will have maps of those places inside their mall. So we’ve also done a lot of work to try to capture that information from all those mall websites and reproduce that in a way that’s easy to use inside our dataset, so that we have the best picture we can have about what are malls, what are not malls, and if it is a mall, what’s going on inside that mall.
Daniel: One of the reasons why I wanted to give you the opportunity to put some more words around that is because I think this must be very critical for a lot of these use cases — having those accurate polygons. If I’m a business and I’m looking to advertise to people based on their location, then I don’t want to do it just based on a proximity to a centroid, I want to know that they’re actually inside the business. I can see that being a really critical piece of these use cases.
Ryan: It is an interesting challenge, and there are also interesting challenges on the end user side as well. Because if you want to take SafeGraph polygons and combine it with, for example, anonymized GPS data from smartphones, you’re also going to be running up against limits to the GPS technology. Intuitively we know that when you go inside a big indoor mall your GPS data on your smartphone is going to get a little noisier, it’s going to be harder to pinpoint that location. There are a variety of efforts around this to try to improve that with things like beacons and wifi signalling — if you connect to the wifi in a store, maybe that gives you some information about where that device is.
Ryan: So our end users are fighting this from two ends. We at SafeGraph are trying to make the polygons as accurate as we can to help those end users with those problems, and similarly the smartphones are trying to make GPS as accurate as possible and solve these indoor location problems. But indoor location is a different beast, and in general I think the SafeGraph dataset has been designed and built towards GPS style data, which is more for understanding these macro things — are you going into a building or not — but not, once you’re inside the mall, which exactly which store did you go to.
Daniel: But it sounds like you’ll be ready for that when it comes — when we get that accuracy in our cell phones, when we can accurately geolocate inside buildings easily and intuitively and it just happens behind the scenes. It sounds like your data will be ready for that. Is that correct, or is there other data cleanup, data maintenance that needs to be done before you could really take advantage of that?
Ryan: I think that’s definitely our goal, to have accurate data at that level. And I think one thing it’ll be interesting to see is exactly how those technologies come around, because that might dictate different strategies for what we should do at SafeGraph to make the product more useful. If it just turns out to be some sort of super hyper accurate coordinates inside locations, then SafeGraph’s well positioned already to serve those needs. If we end up using other types of signals, then SafeGraph might want to look to try to add other types of attributes to the dataset. For example, something we’ve thought about in the past is, should we try to understand what the different IP addresses are for these different businesses? Because if you go into a store and connect to their wifi, maybe the end user advertiser would have access to anonymized internet traffic data, but they wouldn’t necessarily know the GPS coordinate — and so how do you join that to something like SafeGraph data to make it useful? So there are product things like that that we’ve thought about that we potentially could do in the future.
Adding Rows and Columns
Daniel: This is a really good point in the conversation to start talking about the future. What are your plans? What kind of rows and columns would you, or are you, considering adding to your database? Or is it just enough the way it is?
Ryan: Great question. We’re always thinking about how we can add rows and columns to the database, and that is definitely the two ways we think about it. When we think about adding more rows we think about what are the places that we want to catalog and report on that we aren’t already. And when we think about columns it’s, what are the attributes we want to add to these places? So we think about both of those as our long-term product strategy.
Ryan: I think the big picture vision for SafeGraph is we want to be the definitive place to go to get information about every physical place in the world. That’s a big task, it’s a big challenge, and there are a lot of steps to go from where we are now to get there. As I mentioned we’ve started by focusing on places in the United States and Canada where consumers go and consumers spend money, and we think we have a pretty good handle on those places. Today the dataset’s about six million points of interest, so we have six million rows, and we think we’re doing a pretty good job of capturing not only the major retail brands like Walmart and Starbucks, but also the small businesses — individually owned restaurants or individually owned convenience stores.
Ryan: In terms of the types of attributes we want to add to those places, we’re thinking a lot about what are other dimensions of consumer behavior that would be interesting for our end users to work with. From our Patterns dataset we have summaries about things like what times of day people are going to the stores and how those people are travelling to get to that store in aggregate. And you can also tie demographic data to that based on census data. One of the things that we report in the dataset is, on average, what are the different census block groups that people are coming from to visit the store. Again, we don’t have any individual data about any of these individual devices — that’s fully anonymized and then aggregated when we get it — but we can attach demographic data to those census block groups on average and say, for example, people going to this particular Starbucks on average are coming from these census block groups that on average have these types of demographics. So that’s also pretty interesting. There are other types of things that we’ve thought about doing — for example, can we get into more data about transaction level things that are happening at that store? Do people make lots of small purchases at this place, do people make fewer big purchases? Things like that would also be interesting to our clients to add.
Daniel: I’m imagining that if you were looking to add more things to the database you’d be looking at what is the industry asking for, what can we provide, and what can we repeat. Is there any weighting on those three categories, if you can split them up like that? Is any one of them more important than the other?
Ryan: That’s definitely the way we think about it, and we care about all three of those dimensions. I think the most important thing that drives our thinking is: is this a product that, if we made it and we did a good job with it, people would actually want it, and would our clients use it? We have a number of really outstanding customers that give us a lot of great feedback, and if we have ideas or if we’re thinking about exploring new ideas, we will go talk to them and ask them, if we were able to provide this type of data, is that something you would want to use? Okay, so if we structured it this way would that be useful, or would you rather have it structured this way?
Ryan: It’s important to understand that we’re really just a data company. We don’t build solutions for people, we don’t build analytic services. We’re really just building a dataset that our end users take to build cool things with. And so we want to make that data both useful and valuable, but also easy to work with and generalizable enough so that different types of customers will all be able to use it and run with it. So those factors all affect the strategy a lot.
Specialists, Not Jacks of All Trades
Daniel: What are the really big opportunities that you see in this space in the future?
Ryan: I think there are a couple of big trends happening right now that are going to be exciting for the future. The first is a general macro trend around data science and machine learning, which is that as data’s becoming more and more easy to collect, easy to store and easy to work with, and compute’s getting cheaper and cheaper, we’ve seen this explosion of machine learning and artificial intelligence applications. I think that’s going to continue, and that’s very exciting.
Ryan: That then intersects with the GIS world, where I think that as companies in general are getting better at working with data, companies are starting to realize more and more that there’s this whole rich GIS world that exists to help them think about location. I think we’re going to continue to see more and more GIS users in the future. In some ways GIS is this interesting niche within data science, but I think in the future it’s going to become much more integrated with the rest of data science, because the tools are getting so good and the needs are becoming so much more clear. Location and GIS can be a part of almost any type of retail or marketing or supply chain logistics — so many things in the real world are location context dependent. So to me what’s really exciting is to see these different niches of data science being more integrated and coming together more, so that whereas in the past maybe someone was a specialist in NLP or GIS, in the future we’re going to have tools to make these things much more integrated. And that’s just going to mean more and more people using GIS and understanding how powerful GIS can be.
Daniel: You said it yourself a lot of times there — you talked about people using GIS, and that’s exactly the way I think of it. When I think of GIS I think of a person sitting there doing some kind of spatial analysis or working with geographical data. But right at the top of the interview we talked about this being a truth set of the physical world. What we’re talking about there is a truth set for people, of course, as a set of data that they can use — but also for machines. If we have a way of validating what machines are doing, are they doing it correctly, can we hold it up against the truth set and say yes, 80 or 90% is correct? I think that’s a whole different way of thinking about GIS and geospatial in general. We’re making data for people, but we’re also doing it for machines.
Ryan: I think that’s right, and I think there’s an interesting dynamic in this regard in the GIS world. Again, I’m relatively new to the GIS world, I’ve only been working in it the last three and a half years or so, so you can tell me what you think, Daniel, if this is the right impression. But I think that in the past — especially going way back, but in general — people that worked in GIS, if you made maps, if you were a mapping person, that was often very synonymous with collecting and generating the data yourself. To make the map you had to go out and get the data about the map.
Ryan: And often when we talk to people who work in GIS, especially people who have been working a long time, sometimes they’re a little confused about our value proposition. Because SafeGraph is saying, hey, our value proposition is we’ve gone out and done all this work to get this data together so that you can just take it and make maps and build applications and answer questions. And sometimes people are thinking, well, I thought my job was to get the data. So I think it’ll be interesting to see — we think that there are enough resources now where you can segment these different types of skills, and in some ways I think they are different types of skill sets. The whole idea is that SafeGraph thinks that we can focus just on doing this hard thing of getting the data together, and you can focus on what you do best, which is GIS analytics and analysis and making maps that tell good stories and communicate effectively. I think that’s a really exciting version of the future.
Daniel: I would agree a hundred percent with what you said there. In the same way we see companies like SafeGraph saying, hey, we’re not a jack of all trades, we do this one thing but we do it really well. You used the word segmenting before, and I think that’s a great way of thinking about it — we’re very targeted, we’re doing this one thing and we’re amazing at it. In terms of geospatial, at least my impression of the industry up until now has been that the typical GIS person has been that jack of all trades. They’ve been doing a lot of different things and never really had the opportunity to focus on one thing in particular. It’s difficult to find a person who is a really good spatial programmer, at the same time understands the IT infrastructure, and can produce incredible cartographic output — but that has been the expectation in the past. So I think there’s a real shift happening now where people are saying, okay, we need specialists in these areas.
Ryan: I think that’s right, and it’s the same sort of trend that you see in the broader technology world, where in the beginning you had people that were full stack engineers that would do everything from the infrastructure and managing the hardware all the way to building the web apps. And as that technology has grown, and as we’ve had companies that offer services to help segment and compartmentalize those different skills, we’ve also been able to have people focus their expertise on different parts of that value chain. Ultimately that just lets people double down on what they do best and what they enjoy the most, and I think that’s a good thing.
Daniel: I agree, but I think it’s always interesting to hear if people understand this as an opportunity or as a setback. I find it really interesting how people approach this — what I would call an opportunity, anyway. Hey Ryan, it’s been an absolute pleasure talking to you, I’ve really enjoyed the conversation. Before I let you go, I just want to hear where we can go to learn more about you and follow along with the company.
Ryan: It’s been great talking with you Daniel. If you want to learn more about SafeGraph we have a couple of great resources on our web page, safegraph.com, that links to our blog. We do a lot of writing about our data, use cases, and things you can do with our data, so if you’re interested to learn more about that I definitely encourage you to check that out. We also have the SafeGraph Data Bar — if you want to try some SafeGraph data yourself you can go there, no credit card needed, you can download some data. I think we’re going to offer a coupon code: with the code Mapscaping you get $500 worth of free data from our shop, if you want to download some of the data and play with it. You’re strongly encouraged to do that. We’re also on Twitter at @SafeGraph. If you like the space I definitely encourage you to follow our CEO and founder Auren Hoffman on Twitter, that’s A-U-R-E-N. And you can also follow me — I’m on Quora and on Twitter at @RyanFoxSquire.
Daniel: Thanks very much Ryan, I’ll make sure to put these links in the show notes so people can follow up there. And I just want to say to the listeners out there, I’ve been to the Data Bar, I’ve tried it out, it’s amazing. It’s not often I get really excited about this kind of data, but I have to say it’s structured in a really beautiful way. It was really intuitive, easy to find my way around and to see what I was getting. I was deeply impressed.
Ryan: Thank you so much, great to hear that.
