Episode #43 —Drones are changing the frequency, resolution, and scale of geospatial data collection. SkyCatch is applying drone-based data collection to the mining industry which might just be the first step on the journey to an autonomous workplace.
This episode is sponsored by HiveMapper
A platform that takes video and creates 3D mapping layers based on that data. The video can be from a variety of different sensors, does not need to be vertically looking down on the geography and each 3D output is georeferenced!
You are more than welcome to reach out to me on social media, I would love to hear from you!
In Conversation
From Wind Turbines to Mining
Daniel: Welcome to the podcast Kieran, thank you so much for taking the time to come along and talk to us today. I discovered during the pre-interview that your title is Fleet Commander, and you work for a company called Skycatch — so you’ve got something to do with drones. I know that it’s related to mining, and we’re going to be talking a little bit about how drones can help us automate workplaces. But before we dive into that, how did you get involved in drone mapping?
Kieran: About three years ago I graduated from Santa Clara University, and from there started a small startup doing wind turbine inspections using drones. That was mostly around automating the capture, the camera firing, and then taking those images and presenting them for an inspection. I worked there for about a year and a half, and typical startup fashion, they ran out of money. I kind of looked around at other drone companies and found Skycatch. So I’ve been here for two years — started out in the QA process, and now working in the customer success team, so working with trainings, implementation, supporting customers in the field.
Daniel: Before we go on to talk about Skycatch, could you explain a little about what you were doing with these wind turbines? Was it just a visual inspection, or were you flying other sensors around them and scanning them? How did it work?
Kieran: It was really just kind of cobbling together — three years ago is a long time in the drone industry. So I was building a platform using a Pixhawk flight controller and then mounting a Sony A7 camera on there to take really super high resolution photos. Because with wind turbine inspection what you’re trying to do is zoom in really closely to the blade, so you’re flying fairly close, within about five meters, and you’re just scanning up and down. The wind turbine is stopped and the pilot is manually flying up and down those blades. Then with those images you can actually zoom in to a sub-centimeter level, and you’re looking for very thin hairline fractures, whether it’s a lightning strike, bird strike, oil leakage — just really getting that sensor close to something that is otherwise hard to reach.
Kieran: Actually the person who started that company, before that he worked on rappelling. They would climb up to the top of the turbine and then set up a bunch of ropes, rotate one blade so it’s hanging down, and then just go down with a checklist and a pocket camera and mark things off. So with drones you can capture the whole turbine very quickly and you have a much richer set of data to work with later.
Daniel: That speaks a lot to the idea of starting to automate workplaces, which is what we’re going to be talking about, just in a slightly different setting. So that was your past — Skycatch is your future. What is it that you’re doing with Skycatch and drone mapping?
Kieran: Working with people to enable them to use drones to capture their job sites, whether it’s a mine or construction site or a forest. Really enabling all these different use cases of drones, and training people so that they can use the drones themselves and they’re not having to enter into a pit, for example. With traditional surveying you have to get very close to things to measure it — with a GPS rover or a laser scanner — and traditionally the closer you are the more accurate you’re going to be in that measurement. With drones you can increase that standoff distance significantly, and you can capture a very large area. Because it’s an aerial vehicle it can fly tens or hundreds of acres and capture a whole pit wall in 30 minutes, what would traditionally take days of planning and then a day or two in the field to set up all those different capture methods. So drones are just a really awesome way to capture all types of new data. So it’s both training people how to use this system, and then also learning with them how to make use of this data in new ways that they haven’t before.
Why You Fly a Pit Wall
Daniel: I think we need to add a little bit of context here. We’re talking about a mining environment, and you talked about a pit wall for example. Can you tell us why it’s important to capture that, why is it important to measure it?
Kieran: A pit wall is that benching that you might be familiar with in a mine. What you’re doing with that benching is you’re digging as steep as possible while minimizing the risk of that wall collapsing. If you dug this straight down that wall would collapse at some point, and if you’re in a mine, or anywhere, that’s a bad thing. So a lot of what mine planning is, is figuring out what’s that optimal angle that that pit wall is going down. And sometimes you’re basically right on the edge of what’s possible.
Kieran: So measuring that wall is very important for geological surveys, geotechnical surveys, and just determining, hey, is this wall moving? Oftentimes that wall is actually moving very slowly, so it’s important to be able to characterize what that collapse looks like. And then actually looking into the fissures and the actual rock itself to determine, hey, if this sedimentary layer is here, I can also combine that with other types of geospatial data like drill hole data or surface data to determine, is there another mineral vein over here that we can go look at?
Daniel: This sounds like something that needs to be repeated often. Are we looking at the same kind of time savings that you talked about earlier in terms of surveying the wind turbines? Are we talking about the difference between days of human effort and a couple of minutes of drone effort?
Kieran: The time savings are pretty significant — hours, not days. But really, more importantly, you can capture a whole wall in a short period of time. Traditionally you wouldn’t do that. You would just capture one area that you’ve identified as a high risk area, and then maybe once a year or every couple of years get a much larger area. So with drones you can capture a huge area on a very repeatable basis and get very consistent data.
Terrain-Following Flight
Daniel: Presumably you’re flying over these in some kind of planned way, and you’re using some kind of sensor to catch the data. Is it a normal visual camera, or are you using other kinds of sensors on your drones?
Kieran: Pretty much just RGB cameras. You’ve got an iPad, you basically draw where you want the drone to fly, and then it’ll do a lawnmower pattern over that area. And actually it’ll load in a DSM or surface model of that pit wall, so it’s flying the same height above those benches. So it’s not just a two-dimensional lawnmower pattern in the air, it’s actually going up and down scanning the surfaces. It’s really cool to see. So you’re flying over that pit, taking a couple hundred photos, and then we do a process called photogrammetry which combines that RGB information into a three-dimensional point cloud and other data types.
Daniel: That’s really interesting — I didn’t realize that drones could ingest a 3D model like that and fly so they’re always at the same relative height over the surface of the earth. I didn’t know that was possible.
Kieran: And that’s kind of — I would say we’re at early stages in that. It’s not just a pit wall. Say if you have a refinery, you can actually load in a model of that refinery and then say, hey, these are the smoke stacks, I want you to go inspect them. And then the drone would plan itself and know where the obstacles are. So it’s using a combination of onboard sensors to detect things in the air, but then also using that pre-built map of the world to go identify what areas it should be inspecting. That’s still in the future a little bit, but we’re starting to see an evolution from very simple lawnmower patterns, orbits, 360 photos, to more complex flight patterns that the drone is figuring out itself, and humans aren’t having to meticulously plan every flight pattern.
Toward Autonomous Drones
Daniel: How far off do you think we are from drones working autonomously — knowing when to take off, knowing what to look for based on what they’ve seen before, and landing by themselves? Where you just have a drone on a landing pad and from time to time, when it feels the moment is right, it flies off, does some surveying and comes back.
Kieran: Oh man, there have been a lot of wrong predictions about the drone industry, so I don’t want to get myself in hot water. But I would say a couple of years. That vision of a fully autonomous drone that’s in a box, it flies around, and basically there’s just a person in the office saying, hey, go inspect these things — we’ll get there. I think it’s within five to ten years where the human is basically removed from the physical operation of it.
Kieran: But right now what I think the industry is really focused on is just making that very easy to use. So a human pilot is still fully present, and feeding the pilot all the right type of information. You’re always going to have to have a person monitoring that system, wherever they are — whether it’s in the field, in the pit, or in an office. You’re always going to have to be presenting the right telemetry information and the right sensor and health data for safe operations. So right now it’s a person next to the pickup truck surveying the whole area. I think you’ll see an evolution of where that pilot is located, but there’s still a lot of work to be done for sure.
Reconciliation: Tracking Material Movement
Daniel: So at the moment we’ve realized that drones are a great choice if we’re going to survey these large open areas, and do it in a really detailed fashion and in a very repeatable manner. We’re collecting data of the surface of the world and combining it with other geodata sets we have of fissures and rocks and different layers in the earth. What other kinds of things are we using drones for in a mining situation today?
Kieran: On a daily basis I would say reconciliation. At the end of every month you’ve got to count up — for example if you’re at a coal mine, your clean coal pile is the concentrate, so that’s stuff that’s been through the mill, it’s been processed, it’s high-grade and ready to be shipped out. So that’s a very important area to be capturing on a regular basis, and at month end. And then your spoils — the stuff that is not leaving the mine but is actually being dumped somewhere else. These are big mounds.
Kieran: So what surveyors are doing, with whatever equipment they’re using — drones, laser scanners, GPS rovers — is they are trying to track the material movement. If I’ve got x amount of material in the pit, and y of it goes to the trains to be shipped out, and half of it goes to my spoils pile, surveyors are responsible for tracking that material movement. And then what they’re doing is combining that with sensor information from the trucks. The trucks are more or less aware of how much weight is in them, so they’re basically trying to reconcile those two — call them data flows. They’re really material flows, but the data is measured in two different ways: on the truck, and then with a drone. So you’re trying to match up and make sure that you know where all your material is going.
Daniel: Just to try and summarize really quickly — the trucks know how much material, or the weight of the material, that they’ve moved around the place, and presumably the trucks also know where they’ve been, where they’ve dropped this material off. So you’re trying to match that against flying drones over these piles of material that you have around the place, and I’m assuming they’re collecting some kind of volume data. Is that correct?
Kieran: Yeah, so off of that three-dimensional point cloud that the drone is producing, you can do a measurement — either a cut and fill report or a stockpile report — and figure out how much material was there. And then if I fly the next day I can see what’s the difference between those two surfaces.
Blast Heave and BMT Balls
Daniel: In the pre-interview you talked a little bit about how drones were used to map something called heave. Can you tell me a little bit about that?
Kieran: The heave is the pile that’s there after a blast. If you’re about to blast an area in the mine you drill a bunch of drill holes and you fill those up with explosives, and then you blast it, and that loosens up all the material and gets it ready so the excavators can come in, load it onto those trucks and then take them to the spoils or the mill or whatever. So that heave pile is the active area — it’s where a lot of the activity is happening.
Kieran: An interesting project that we worked on is these things called BMT balls. Along with those explosives that get dropped into the drill holes, you also drop this kind of steel cylinder ball that’s about the size of a soccer ball. You drop a bunch of those in with the explosives and then you blow it all up, and so those steel balls will move according to the density of the material. So if some of the heave is directed in this direction, well, I know the density of the material is slightly higher over here. And then based on what I know about the geological conditions, I can determine, hey, this part of the heave actually has more mineral in it, so we can go and extract that, and we can be a bit smarter about what material we’re sending to the mill versus what material we’re just dumping somewhere else.
Daniel: I’m assuming here that you’re flying drones over there and mapping where these balls are landing, and based on that you’re saying, okay, this part of the heave we’re going to use for one certain thing and the rest we’re going to take somewhere else. Is that the way it works?
Kieran: Yeah. Before, it was kind of a guessing game based on a lot of other more subjective forms of data. You would start digging places and remove it — oftentimes it was just visually, the operator is looking and going, hey, this is a little bit sparkly, I know that there’s more zinc in that one, so we’re going to take that to the mill and process it. With these BMT balls, once they get blasted and the heave is there, then the drones will fly over, map out where that is, and then the excavators can go in and start digging right away.
The Automated Mine
Daniel: Right at the start of the interview we mentioned this idea of automated workplaces, and so far we’ve talked about automating a few different parts of the workplace. But where is this going? Are we building these maps, are we using these drones so we have better information, with the thinking that one day these trucks, this heavy machinery, will be moving around autonomously in the mine? Is that the end goal?
Kieran: That’s certainly the end goal for a lot of these large mining corporations. It’s a very dull, dangerous and dirty job — you’ve got coal dust everywhere and very long shifts. So drones are I think the first type of robots that you’re seeing, because the environment they’re operating in is relatively unobstructed. You’re operating in the airspace, so there’s not much there, and to navigate that is pretty simple, and with quadcopter drones we figured that out pretty well.
Kieran: But now I think the next is starting to automate some of the machinery in there. So we’re seeing a lot of autonomous haul trucks — new mines that are coming online these days are basically looking at fully automated autonomous haul trucks. These are huge 300, 400 ton dump trucks that are carrying that much material, and those are going to be the first ones automated on the ground. And then you’ll start to see excavators, water trucks, really everything start to be automated.
Kieran: If you want to start working on a mine, often what they’ll do is they’ll hire people with no driving experience — because these haul trucks, you’re sitting basically on a two-story building, super high up in the air, and the way they’re laid out is so that the driver is on the right-hand side, which is not normal for most people. So if you don’t have any driving experience these are actually one of the first entry-level positions. And I think we’re going to see that job kind of go away in the near future.
Daniel: When that happens, when these kinds of jobs go away and we’re looking at a completely automated mine, would you expect that these different pieces of machinery that are moving around will be creating maps on their own as well? Or will there be something like a more advanced version of what you’re talking about with drones, creating one sort of global truth and sharing that information out to the rest of the autonomous vehicles in the pit?
Kieran: That’s really interesting. I think drones are going to create that base level map, and then all these other types of robots that are operating are going to be feeding into that map as well. So you’re getting geospatial data, maps, alerts, from all these different robots, and then you’ll have one central control center where you can see a very up-to-date, almost live feed of the map — whether that imagery is coming from a drone or a laser scanner on a truck or a person out there. There are many different types of data streams that will feed into this very cool looking map, probably.
Beyond Mining: Construction, Forestry and What’s Next
Daniel: Up until now we’ve been talking about the use of drones in a mining situation, and it seems to me that this is almost a perfect use case for drones — there’s a clear view of the ground, it’s a large open area, and not too many obstacles to fly around in the air. Could you imagine drones being used in construction situations?
Kieran: We do some work in construction as well. Actually the most value that drones in construction see is during the early stages, where you’re moving a bunch of dirt around, no building has been constructed yet, and you’re basically just trying to grade and shape the earth for that eventual pouring of concrete, creating those pads for the buildings to go on. So that’s drones in the early stages — civil construction, heavy earthworks — that’s where we’re seeing the most value.
Kieran: And then as soon as the buildings go up it gets a lot harder, because the stuff that you’re interested in is very tiny. You’re looking for manholes or staking or tie-ins, so the objects are smaller. And then also it’s just a very complicated site — you have roofs that are going up so drones can’t see into that, and most of construction actually happens indoors. Once those roofs go up it’s pretty hard to use drones.
Daniel: Are there any other really great use cases out there for drones? I can think of a few, but is there anything out there where you think, wow, once we start doing this, this will be the end game for drones?
Kieran: There’s a lot. Everything from forestry, where you fly a drone over and create a very high density point cloud of that forest, and you can determine how many trees there are and how tall those trees are, and based off of that height you can estimate how much wood mass is in that tree. So for forestry, we have all these managed forests — that’s a really good use case. What I thought about when I was on the road yesterday is traffic drones. If you have an accident up ahead, a drone can get to the scene very quickly to determine the conditions and then also start to direct traffic. We’ve seen Intel has all those light shows, I think it was on the Super Bowl, and I think you’ll start to see some of that. And anywhere where the surface of the earth is changing on a daily basis, drones are just a very good way to measure that change.
Daniel: You’ve got a lot of experience with drones and drone mapping. When you look out into the future — and we talked a little bit before about how quickly the landscape is changing in terms of drone technology — what are the most exciting developments you see coming towards us?
Kieran: It’s not so much the drones themselves. Every drone company says they’re not a drone company, they’re a data company, and I guess in some way it’s true that the drones are just the camera. They’re recording photo information and then GPS information, and then really the processing and the analysis on that data is the most interesting part. Drones are a very good way to capture a lot of that information.
Kieran: But once those base maps are built, combining it with an excavator — so you can tell an excavator, here’s the map and here’s my design file, and then you can start to automate that shovel head and tell you exactly where to dig. We’ve seen RTK GPS technology really improve, and the quality has gone up and the cost has come down really low. So I think combining that with the sensor information from machines, we’ll get very rich models of the world. Drones are the first layer of that, but once we start to build other layers on top by combining excavators and bulldozers and trucks with that data, I think they’ll all tie in to that drone data initially, and then start to automate some of those tasks that other machines are doing.
Daniel: It sounds like a really exciting future that we’re going towards. I really want to thank you for taking the time to talk to me today, I’ve really enjoyed the conversation. Just before I let you go, where can we go to learn more about you, and if we have any questions who can we reach out to?
Kieran: Skycatch.com is my work. I’ve got Twitter, I guess it’s @kieranwskelly. And anybody who’s working on this type of stuff, or is interested, or is even thinking about getting into the field, I encourage them to reach out — because it’s an exciting future. There are a lot of hard problems to work out, both on the technical side and then also I think the social side as well. Really great talking to you, thank you.
