Geo-tagged audio - another way of augmenting reality

May 03, 2019 16 min read

Geo-tagged audio - another way of augmenting reality

 LISTEN HERE OR WHEREVER YOU GET YOUR PODCASTS

 In this episode you will learn what geolocated audio is, how it relates to 3D audio experiences and how geolocating audio can be used to augment reality. Enjoy!

Josh:
Yes exactly. The areas which you use to trigger the sound files, geo fences. We allow you to attach information to those geo fences as well so you can add a picture and a bit of text to those areas, to those zones, but really the main focus is the audio. It's tagging a location with audio.

Daniel:
Hello and welcome to another episode of the MapScaping podcast. My name is Daniel and this is the podcast for the geospatial community. Today I'm talking to Josh from the company called Echoes and he's going to be telling us all about the geolocation of audio files. I hope you enjoy the interview.

Daniel:
Welcome, Josh. I'm so pleased that you could take the time to talk to me today. I think from the little tiny piece that I know about your work. It sounds really interesting, but before we jump into it, maybe you could just give us a quick introduction about who you are.

Josh:
Sure, my name's Josh Kopecek. I'm the director of Echos, which is a platform for creating locative audio experiences.

Daniel:
Right. That begs the question, what is locative audio?

Josh:
It's a large topic really. Perhaps it would be good for me to give a little bit of background about where my history in locative audio came from. Personally, my background is as a musician and I studied electroacoustic music and classical music and it wasn't until I finished my studies that I was introduced to locative audio.

Josh:
A few friends and I were asked to create a walking audio experience. At the time I had no idea what it was. We were thinking about the experiences to create. At the time I was living in Vietnam and we had collected sounds from around the city and we were thinking about placing them in various situations. We used an APP which was available at the time to create a map of the sounds around the city.

Josh:
As you walked around the sounds were then triggered by your location. The APP we used at the time was great, but we came up with a load of problems which we wanted to solve. We ended up eventually creating our own platform for making these walking audio experiences. Through the process of creating that platform, we discovered that there was a really great interest in creating these experiences.

Josh:
A lot of people who were involved in that original walking audio tour were then interested in creating their own experiences. We opened up our platform for people to be able to publish their own experiences. That's really where Echoes has come from. The whole concept around locative audio is that you have a piece of audio or sound. Could be anything, somebody speaking. A narrative is very popular. It could be a field recording, something that you've taken a recording of outside.

Josh:
It could be something you've recorded in a studio, it could be a piece of music, and then you fix a location to that GPS coordinates. Then when the user gets close to that sound, then it's triggered for them and then it starts playing in their headphones. This, normally requires a smart phone to be able to play it back with GPS. But that's all you need.

Josh:
A lot of people nowadays have a smart phone. I think smart phone penetration in the UK is around about 90% so basically anybody could do it. Then you need a decent pair of headphones as well if you want to really immerse experience but just to get started, you need a standard pair of headphones. That's the basic underlying concept behind locative audio and a little bit about our history.

Daniel:
So, if I had to put this in layman's terms, it sounds like geo tagging sound files. I know that's probably a really rough way of saying it, but is that what we're talking about here?

Josh:
Yes, exactly. The areas which you use to trigger the sound files, are geo fences. We allow you to attach information to those Geo fences as well. You can add a picture and a bit of tax to those areas, to those zones. Really they are... The main focus is the audio. It's tagging a location with audio, yes.

Daniel:
In the pre-interview talk we had, you talked about that you could change the nuances of this audio depending on where you were in the zone. It wasn't just, okay, here's the point. When you get within five meters at this point this things happened. The audio starts playing, but you could fade in and fade out and trigger other sounds, but all in the same area. Is that correct?

Josh:
Yes, that's right. You have control over some basic playback properties. For example, if you want a looping sound, you can set a sound to loop when you're inside a zone which can create a continuous ambient background sound for you, which can then be a backdrop for your narrative or whatever if you're creating a guided tour.

Josh:
You also have the ability to... Especially which is good for guided tours, be able to play a sound file right to the end of it rather than if you leave a zone, then it'll stop. Then your also able to control the volume of the sound based on how close you are to the center of that zone.

Josh:
The further away you get the quieter the sound gets and the closer to the center you get, the louder it gets, which enables you to create these layers, these lattices or collages of sound so you can create a moving texture, moving sound texture as you walk around in that sound map which creates a whole gamma of possibilities for making interactive audio.

Josh:
That's really the part which excites me, which is the creative possibilities behind locative audio. That's just scratching the surface of it really. Because through the Echoes platform, we just offer the most basic parameters for changing the sound. Of course there are other ways of doing it.

Josh:
You could potentially use all sorts of other different factors apart from location. You could use the time of day or the temperature. There's all sorts of sensors on smart phones, which you could use to change the sound as well. But the core of what we do is working with location. Using the spacial coordinates to trigger sound and then control as playback.

Daniel:
In the GIS and mapping world, we're quite used to attaching information to geometries, to things in space to certain areas. For example in GIS, geographic information system, for example, you might have polygons representing buildings and each polygon might have an attribute which would be... It could be a number, it could be the amount of people living in that building. It could be the amount of parking spaces attached to that building. It could be something like that.

Daniel:
This is the first time that I've ever heard of people attaching a sound file to a geometry. I think it's a really interesting concept because it's another way of augmenting that data and it sounds like what we're talking about in reality is augmented reality, but maybe just not with the glasses this time. Maybe not in visual way, but in an audio way. Would that be a correct way of looking at this?

Josh:
Yes, exactly. I describe it as locative audio, which of course is you could say quite antiquated term now. Locative audio as a concept has been around for a long time, maybe 10, nearly 20 years I think. The new concept is to use augmented reality. We tend to call it audio augmented reality these days, which is like you say, attaching sounds to locations and really we allow you, through the Echo's platform to be able to add circles and polygons.

Josh:
Like you were describing. You could add a polygon to a building and then when you get within the vicinity of that building than the sound will be would be triggered. This really raises the possibility of creating augmented reality experiences by just using audio. This is probably the most exciting part for me, which is how you can create an immersive experience using audio. Because when you remove the visual component, you are in a sense reintroducing the real world and the only sense you're taking away his sound.

Josh:
It becomes like a soundtrack to your life as you walk around. That sound then merges with the real environment and probably, what I'll get into in a minute, is 3D audio, but that 3D audio then offers this next generation of immersive capability.

Daniel:
I think it's really interesting. Your approach to this clearly as an artist, someone who's wanting to create experiences. I come from a GIS, geospatial mapping background, and I can see this as a way of delivering information. Like I said before, another way of augmenting reality and another interface. Maybe just a less intrusive interface where you could walk around with your smart phone, your headphones on and receive information passively from these different sources depending on where you were in relation to them.

Daniel:
You mentioned this before 3D audio. What is that?

Josh:
Right. Well, It's interesting you say... I just want to go back a second because you mentioned passive. It's important to remember. I think that this isn't just a passive experience, the listener is really controlling the experience as well. As a medium locative audio is giving the listener the control back over the kind of experience they're creating.

Josh:
Of course the base components, the sounds which are attached to each location are created by the composer or the person who's made the audio tour themselves. But the listener has control over which sounds they play back, where they go, how long they remain in a particular location. Which offers a very interesting [inaudible 00:11:55] de-constructivists approach to audio experiences and which is not something which is really possible with traditional audio or listening experiences.

Josh:
Okay, now I'll get onto the 3D sound aspect. 3D sound is a another layer on top of locative audio. 3D sound allows you to create what is in essence a surround sound experience over headphones. If you imagine when you go to the cinema, you're watching a screen in front of you, but you might be hearing a helicopter, classic Dolby experience, you hear a helicopter flying overhead.

Josh:
It sounds as if it's over... It sounds as if it's traveling over your head and it's quite realistic. You can achieve the same thing with headphones. This technology has been around for again more than 20 years. I think it was in fact possibly developed in the late eighties. The possibility to specialize a sound. It appears as if it's coming from a particular direction around you just using a pair of headphones.

Josh:
This has traditionally been used in games so you can hear a character who's in front of you or an enemy coming from behind you, something like this. What we want to do is use that spacial audio to create more immersive experiences. As you walk around, you would hear sounds coming from a particular building or you would hear a narrator who is directly in front of you and you could approach them and walk around them.

Josh:
You could hear clues coming from particular locations, you could change the audio environment of a particular space. For example, you walk into a factory building and it sounds like a jungle. That's a really basic way of explaining it but the possibilities are really endless.

Josh:
With the new technologies and hardware which are coming out, we're now able to offer head tracking, which enables you to basically create an experience where as you move around and as you move your head, the audio appears to stay in the same place. You can have a static piece of audio. You imagine listening to a bouncing ball in front of you, you can approach that bouncing ball. You can move away from it. As you turn your head away from it, it stays in the same place and that's all possible with head tracking and 3D audio. It's a very exciting sphere.

Daniel:
It sounds like it's really going to personalize things. It sounds like that you're going to have even more control over your experience.

Josh:
Exactly, yes. You are creating a personal listening experience. Like you say, there's a lot of possibility for creating personalized experiences as you walk around and also interactive experiences. If you imagine the kind of massive multiplayer online games which are available at the moment, such as fortnight, you can imagine playing those in a real world environment where the other people are creating sounds around you in the virtual world and then you are able to hear those sounds as well. So yes, it's really exciting.

Daniel:
You gave some examples before of use cases and one of them was that the classic walking tour, and I'm assuming this would be through a city for example. If you had this 3D sound, this immersive environment that would... I guess that would require quite a detailed map of that city environment, wouldn't it? I mean you'd have to know where the buildings were and how they responded to the different sound if you were going to recreate that environment in a realistic way.

Josh:
Exactly, yes. This is one of the questions and we talked a little bit before about the need to map environments in terms of having 3D modeling for environments. This is very important in audio because for example, a cello playing in a concert hall, sounds very different to a cello playing outside. If we were to record a cello playing in the concert hall and then play it back over the headphones while you're outside, it creates a sense of oral discontinuously.

Josh:
By using 3D modeling, we can actually try to recreate the audio scene using that modeling data and then replace that audio in the environment. For example, if you hear somebody speaking and you pass by a building, the actual occlusion of that sound would then be masked by the 3D model itself. The voice of the person actually appears to be coming in the real space. There are all sorts of interesting questions, very difficult questions, I think, which are, for example, how does sounds behave differently in different contexts.

Josh:
For example, in a park, the behavior of sound is very different to that in a concrete playground or a concrete car park. We have a lot of research to do and a lot of interesting questions to answer with regards to that.

Daniel:
Absolutely. Could you give me some use cases for this? I think you've done a really good job of painting the picture of what locative audio is and the future of it, this 3D audio. We've talked a little bit about that sort of walking tour idea. Have you got any other use cases, may be business use case where you think, "Wow, this is going to be a big deal?"

Josh:
Well, yes, I think one of the most exciting things for me, which I thought of in the past was... Which I'd start with a real passion for doing is creating, I guess historic soundscapes. You imagine going into a museum and being able to experience or traveling back in time using sound. Imagine going into the Neolithic period and being able to experience the sounds of the Neolithic period. Hearing the animals running past, hearing people working with basic tools or calling in their own language, things like this and then placing that in the museum context.

Josh:
Then taking that audio and being able to walk through it in a museum and then switch between different contexts based on where you are in the museum or even changing the soundscape yourself in the APP as you walk through the space. I think there are a huge number of different possibilities for this. Imagine if you could go back to feudal Japan just with a pair of headphones and walking around in the museum space. I think it'd be an incredible experience and I think that's where the real potential lies for spacial audio and 3D sound or audio augmented reality.

Daniel:
Absolutely. That sound like really interesting examples. Like I said before, sounds like... If they're going to be a reality, then it's going to require some pretty hefty 3D models. Updated 3D models of these environments. We want to experience and move around and listen to. I think that the mapping industry has got a lot of work to do there.

Josh:
Yes, exactly. There are some shortcuts we can take like pre rendering sounds in particular environments. We work with something called Amazonic audio, which is a prerecorded audio, like 360 audio. It's very similar to 360 video it is the sound format that they use for 360 video, which enables you to create a pre-rendered audio experience within a particular space. You could then pre-render the space for that particular piece of audio.

Josh:
There are lots of different shortcuts you can do, but there's also a huge amount of research if we were to create those kinds of Pokemon go style experiences.

Daniel:
In the pre interview, you gave this great example of recording a 3D experience of a Gospel Choir and you had a lot of challenges there, especially with the indoor positioning. Could you say a little bit about that?

Josh:
Yes, sure. A project we did a couple of years ago worked with indoor location and spacial audio. We created in a warehouse space a walking audio experience, and I should say this, the content was created by an artist that we collaborated with in the US, he wanted to recreate the sound of a gospel choir.

Josh:
What he did was he multi tracked the audio, so he had the individual sound of each singer, but as a whole it created the gospel choir. Then we specialized the voice of each gospel singer within that space and place them where they would have been standing. What actually happened was that you could walk through the space and walk between the gospel singers and hear each individual singer as you approach them. It would be like standing within a real gospel choir, but it being a virtual gospel choir.

Josh:
This was a very challenging experience because we had to work with high accuracy indoor location, although it was a warehouse and the space was quite large, we still had to have pretty good positioning. We worked with an indoor location provider. This was before some of the newer indoor location solutions came out. We were able to get it down to I think below a meter accuracy. I can't remember what the exact accuracy was, but we needed that level of accuracy in order to give this immersive experience, the playback.

Josh:
Then the other half of the experience was that people were able to contribute their own sounds to the experience. They were able to go to a specific point in the space and record their reactions to the concept of the installation, which was climate change in a particular location in the US and leave that as a sound bite in that space. When other listeners walked into those spots, they could hear that particular feedback. This was really, pushing the possibilities of locative audio to the Max. That was a really cool project.

Daniel:
That sounds really interesting. It reminds me of a conversation I had with a company recently and what they were doing is they were were dragging a whole bunch of data out of Facebook, Instagram, Twitter, anything that was text based and Geo located. Then they were putting it in... Grouping it into this broad categories of political or stay at home moms or whatever it was.

Daniel:
Their idea was that if they could do that enough, collect enough of this data, then they could get a feel for what the pulse of the community was like at any one time and then watch that change. Watch the feelings, the emotions of the community change. Then they could use it for things like we should... A company might say, "Hey, we need to position a shop where we should it be, what community would this suit best?"

Daniel:
They would use this data as another layer to that analysis. What you're talking about before just reminded me of that idea of locating information and in that context they were locating texts. There was text messages but it had a location. What you're talking about is locating sound and people could contribute their own sound. I wonder if this is going to be the next thing, geo tagging sound in the same way we do with photos, texts and videos on social media for example.

Josh:
I certainly hope so, yes. Sound has always been something which has been somewhat left by the wayside but we talked before about the fact that podcasts are also extremely popular these days. I really think that sound has now reached that level of maturity in terms of technology that everyone has a pair of headphones and everyone has a smart phone in the same way that a few years ago it was... the ubiquity of cameras in smartphones suddenly enabled everybody to be able to take a picture of whatever was happening around them.

Josh:
Then that laid the ground for Instagram and other startups to take advantage of that. I think now with the rise of podcasts and the interest in audio generally, people have got a pair of headphones with them most of the time. So, the possibilities for that are huge now and I think platforms hopefully like echoes are going to take advantage of that and the fact that our platform is free and open and free to use, free to publish content on, anybody can start taking advantage of it now. This could lay the ground for things like locative podcasts or locative news, all sorts of new ideas, new mediums through linking location with sound.

Daniel:
Sounds like we're going towards a very exciting future in terms of locative audio. Hey, I've just got a couple of questions left for you before we say goodbye and one of them is, why is location and audio, why are they so good together?

Josh:
Well, what excited me about location and audio in the past and what still excites me in fact is the fact that it's a different medium. I touched on this a little bit before, but the possibility of linking location with audio changes how the content is presented. Instead of creating a linear experience which you would be doing when composing a piece of music or creating a podcast, which starts at the beginning and finishes at the end, and you listen to all the parts in between in a specified order.

Josh:
What you're doing with locative audio is actually deconstructing that audio, taking it apart. Even the background sound and the foreground sound, if you're creating a narrative with [inaudible 00:27:03] sound, you can separate those layers and then trigger them in different ways. For example, the ambient sound could loop for five minutes before your narrative comes in, but that's dependent on how quickly the the listener moves.

Josh:
By creating these layers of sound and placing them in different places, you're creating this level of serendipity in the creation of the audio experience. Also, the listener is free to construct the listening experience as they choose. If you're creating something in a park or in a public space, the listener may not be able to get to all of the parts of it, parts of your experience, so you don't have control over how much they listen to.

Josh:
It's not like they can just press pause and resume it later. They have to be in that space and experiencing it right there and right then in order to be able to get the full experience. That's really the crux of what really still interests me about locative audio is this... the kind of magical serendipity of it.

Daniel:
Josh, I really want to thank you for taking the time to talk to me today and introduce me to this idea of locative audio. I've definitely learned a lot and I'm sure the audience have appreciated your story and hearing about this as well. But just before we say goodbye to each other, where can we go to learn more about what you do in your work?

Josh:
If you head to echos.xyz there's lots of information about what we do. You can sign up for Echoes and create your own content online right now. It's free to upload for you to publish content. We have IOS and android apps ready to download, so get creating now.

Daniel:
Excellent. Thank you so much.

Josh:
Thanks a lot Daniel.

Daniel:
That's it for another episode of the MapScaping podcast. My name is Daniel. I'm the host of the show and I want to thank you so much for listening all the way to the end. I really appreciate it. If you want to reach out to me for whatever reason, you can contact me on social media. We are MapScaping on Facebook and Twitter and [inaudible 00:29:13] on Instagram. Hope to hear from you soon and see next week. Bye.