In this episode, the discussion is aimed at an increased understanding of the differences between computer vision and the AI that is used in the Earth Observation world.

About the Guest

Jordi Inglada is the Senior Expert Scientist for Machine Learning and Earth Observation at the French Space Agency (CNES). His main responsibility involves developing algorithms to extract useful information from satellite image time series for applications such as land cover mapping and geophysical variable extraction, among others.

Jordi has a background in signal processing, image processing, and physical modeling. Coupled with over 20 years’ work experience, this uniquely equips him to apply machine learning in Earth observation.

Computer Vision Vs. AI in Earth Observation

Computer vision and AI in Earth observation might seem similar, but they carry essential differences. In computer vision, the goal is semantic interpretation – recognizing faces or objects in an image using machine learning algorithms. The accurate location of the pixels in space or data in time is not essential. Conversely, AI in Earth observation hinges on physical measures accurately located in space and time.

In essence, computer vision is often data-driven, leaning heavily on learning from the data to interpret and understand images. On the other hand, Earth observation is driven by a deep understanding of physical characteristics. For instance, Earth observation uses established models and equations describing light interaction with vegetation or light’s travel through the atmosphere. This knowledge, combined with an understanding of the sensor’s operations, allows researchers to extract accurate information from the images of observed surfaces.

There is also a difference in reflectance measurement in computer vision and in Earth observation. While RGB imagery in computer vision involves a form of light reflectance measurement, the focus is not on the accurate measurement of the level of light across the three channels. On the contrary, Earth observation requires more accurate light measurements across various spectral bands. Remote sensing satellites like Landsat or Sentinel-2 provide precise measurements not only in the RGB spectrum but in the infrared as well. Obtaining accurate measures from these bands is necessary for the interpretation of physical phenomena occurring on the Earth’s surface.

Is It Possible to Use Computer Vision Algorithms in Earth Observation?

Although AI application in Earth observation borrows from computer vision technology, the transition from computer vision to Earth observation is not as straightforward as it might seem. While some applications might find success in employing traditional computer vision algorithms – for instance, identifying cars in a parking lot or mapping buildings in a city – they might not serve the more intricate applications of remote sensing.

In remote sensing, the focus is often on monitoring physical magnitudes that are not directly observable, such as forest biomass for climate studies. These physical magnitudes cannot be observed directly but rather through reflectance in the infrared spectrum or via radar images in the microwave spectrum. The application of physical models is required to make the connection between the target magnitude and what the sensors observe.

It is also difficult to apply computer vision algorithms to remote sensing due to a difference in time series. In computer vision, time series data are typically in the form of video content, rich with numerous timestamps. In contrast, remote sensing operates on a different, less dense timescale, with data capture events typically happening daily, weekly, or perhaps every ten days. This difference in scale necessitates the development of bespoke algorithms to effectively handle remote sensing data.

Key differences between traditional computer vision and computer vision applied to Earth observation data:

Super Resolution Vs GANs

Super-resolution is a technique used in remote sensing to enhance image quality. It draws from correlations within the data, spatial or temporal, to generate high-resolution content. However, while super-resolution algorithms can create highly detailed, realistic images, the goal is not to generate information that the sensor has not captured.

This emphasizes a fundamental difference with computer vision applications, where the goal of techniques such as GANs (Generative Adversarial Networks) is to generate new, realistic images. In remote sensing, the objective is to faithfully represent a specific geographical location at a particular point in time, not to generate new, plausible images.

Key differences between Super Resolution and Generative Adversarial Networks (GANs):

Deep Learning: Promise Vs Practicality

Deep learning has been hailed as the golden solution for turning unstructured data into meaningful knowledge. But despite this, many operational systems in Earth Observation today rely more on traditional machine learning for tasks such as land cover classifications. A major reason arises from the high costs associated with deep learning.

Training deep learning algorithms is computationally expensive and requires large volumes of data. In many instances, the resources needed to deploy such algorithms often outweigh their benefits – especially when compared to less resource-intensive alternatives such as random forests.

Synthetic Data as Training Data

One application of GANs is the creation of synthetic data to train deep learning algorithms in remote sensing. Given the vast amounts of data required to train these algorithms, synthetic data generation can be a helpful approach.

Synthetic data may help fill gaps for phenomena that are not fully understood or lack physical models. Here, GANs can produce realistic images that help train and calibrate models. But in cases where extensive prior knowledge and physical models exist, such as vegetation growth or deforestation, synthetic data may not be necessary. Remote sensing has long employed physical models to generate images and understand sensor behavior. If the physics is known and the necessary information is available, synthetic images may not hold significant additional value.

Vectors in Machine Learning

In machine learning, vectors of data are produced by transforming spectral bands of the input data into a different representation – which contains meaningful information from the original images. As a summarized dataset, vectors can easily be used by downstream machine learning models.

In many ways, feature embeddings can be likened to Principal Component Analysis (PCA), which simplifies data by projecting it into a different space, with each component being orthogonal to the others and holding maximum variance. However, feature embeddings go beyond PCA by taking into account additional correlations and complex information in the data – which PCA might miss.

Is Coding Knowledge Necessary in Earth Observation?

A future where professionals in Earth Observation cannot create their own software seems unlikely. The landscape of data science can be visualized as a Venn diagram with three crucial components: computer science (coding, software engineering), applied math (statistics, AI), and domain expertise. To succeed in Earth observation, it’s beneficial to have knowledge in each of these areas.

Even if one’s primary expertise lies in geosciences, it is beneficial to learn coding to implement things. Conversely, if someone comes from computer vision and understands statistics and AI, they should also engage with the physics of the phenomena they are observing. For those starting out, Python is highly recommended as the language of choice. With a multitude of libraries available for remote sensing, as well as AI and statistics, Python serves as a versatile and powerful tool in the field.

Unexplored Potential: Higher Resolution Time Series

Higher resolution time series, despite their immense wealth of information, are not fully exploited. Programs like Landsat and Sentinels provide extensive data, sometimes at a frequency of every five or six days. However, often only a few data points are used for applications like mapping, detection, or variable estimation.

This is often due to the sheer size of the datasets. For instance, monitoring agricultural fields over an agricultural season with high-frequency, high-resolution data can quickly amass massive datasets. The resources required to download, process, and extract valuable information from such data can be extensive and expensive. Luckily, platforms like Google Earth Engine, Microsoft’s Planetary Computer, or Sentinel Hub offer interesting possibilities, providing access to these massive datasets along with the computational resources to process them.

Sponsored by Sinergise, as part of Copernicus Data Space Ecosystem knowledge sharing. dataspace.copernicus.eu

Related podcast episodes

• Super Resolution
• Fake Satellite Imagery
• Sentinel Hub
• Google Earth Engine
• Microsoft’s Planetary Computer

In Conversation

The Senior Expert Scientist

Daniel: Jordi, welcome to the podcast. Would you introduce yourself and explain what your title means?

Jordi: I’m Senior Expert Scientist in Machine Learning for Earth Observation at CNES, the French space agency. I do research at a lab called CESBIO in Toulouse, in southern France. My job is to develop algorithms for information extraction from satellite image time series, with applications to land cover mapping and geophysical variable extraction. My background is in signal processing, image processing, and then physical modelling — a blend of applied math with signal processing and the physics point of view of remote sensing.

Computer Vision vs AI in Earth Observation

Daniel: What are the key differences between classic computer vision algorithms and the algorithms we use in Earth observation?

Jordi: In computer vision we mostly work with RGB images, and we’re not interested in the accurate location of pixels in space or data in time — if you’re recognizing faces or objects, you’re interested in the semantic information, not the accurate location. In remote sensing, we’re looking at physical measures which are located in space — latitude, longitude — very accurately, and in time, with the exact timestamp. We also know our sensors very well: their noise levels, their resolutions, the viewing angle, where the sun is. With all that we can extract very accurate information about the things we observe.

Daniel: When I think of RGB imagery, isn’t that also a measurement of reflectance?

Jordi: It is, but you only have three bands — blue, red, green — and the sensors used in computer vision are not very well calibrated. You’re not interested in the accurate measure of the level of light. In remote sensing, satellites like Landsat or Sentinel-2 have something like 10 spectral bands — narrow bands on the infrared, the red edge, the shortwave infrared — and you’re interested in very accurate measures because you’re going to relate these levels of measured light to physical phenomena. We have equations for how light interacts with vegetation canopy, how light travels through the atmosphere as a function of aerosols and water vapor, and how the sensor works.

Can You Use Computer Vision Algorithms in Earth Observation?

Daniel: Can we just take computer vision algorithms and poke them into an Earth observation stack?

Jordi: For some applications you can. If you’re working with very high-resolution optical imagery — say 50-centimeter images — and you want to count cars in a parking lot or delineate buildings, you can use out-of-the-box computer vision algorithms. But that’s maybe not the most interesting application of remote sensing, where we’re interested in monitoring land surfaces through time — vegetation growth, climate studies, deforestation. There we’re interested in physical magnitudes we don’t directly observe, like forest biomass. We observe it through reflectance in the infrared spectrum or through radar, and we need physical models to link that target magnitude to what the sensors observe. Time series in computer vision means video, with many timestamps; in remote sensing the highest revisit rate is daily, usually weekly. So you have to develop custom algorithms.

Why Machine Learning, Not Deep Learning

Daniel: Why don’t operational systems in Earth observation focus more on deep learning?

Jordi: Deep learning is just a subset of machine learning. The specific thing about deep learning algorithms is that they’re very costly to train — both computationally and in terms of the amount of data needed — and their application can also be more expensive than classical algorithms like random forest. That’s why most operational applications today are not yet using deep learning. With deep learning you could improve land cover mapping accuracy by maybe 5 to 10%, but the cost is that you need much more data — not only images, but the ground-truth labels needed to train the algorithms.

AI-Ready Data and Embeddings

Daniel: Could you explain the idea of AI-ready data?

Jordi: Since deep learning models are costly to train, one thing we can do is pre-train them using self-supervised learning — training algorithms without labels, using the huge amounts of data in the Landsat and Sentinel archives. These pre-trained models produce a different representation of the data — embeddings, or features — that can then be used by downstream tasks for which you need fewer training data. Instead of getting raw reflectances from the data providers, you would get these features, and every downstream application would be tailored to the user’s needs.

Daniel: Is there a comparison between these embeddings and principal component analysis?

Jordi: When you use these embeddings, it’s some kind of glorified PCA. PCA projects the data into a different space where every component is orthogonal to the others and has maximum variance. Generic embeddings go beyond that — they’re able to take into account other correlations and more complex information. But it’s a very good analogy.

Super Resolution and the Hallucination Problem

Daniel: If I told you I’m creating 1-meter resolution data from Sentinel-2 imagery using super resolution, what would you say?

Jordi: Going from 10-meter resolution to 1-meter resolution is very difficult, and I think it’s even impossible. Super resolution — we call it deconvolution or image restoration — can improve a 70-centimeter image to 60 or 50 centimeters by increasing the high-frequency content and reducing noise. But there’s a physical limit. At 10-meter resolution you won’t see small cars or a bench on a sidewalk. There are techniques, often using GANs, that generate plausible, realistic 1-meter images from 10-meter ones — but a GAN could create cars that weren’t present in the real scene. If your application is counting the exact number of cars, you don’t want a super-resolution algorithm that hallucinates information. In remote sensing you want the reality that was present at that point in space and time, not new, plausible images.

Synthetic Data and Physics-Informed Models

Daniel: What are the pros and cons of using synthetic data as training data?

Jordi: To train deep learning algorithms you need lots of labeled data, and one way to get it is to simulate it. For most remote sensing applications we have lots of prior physical knowledge — models of vegetation growth, of deforestation, of the sensors — so we’ve been using physical models to generate images for many years. But there are things we don’t fully understand, or for which we don’t have physical models. For those, generative models like GANs could be very useful to generate realistic data. There’s also another place machine learning is useful: when a physical model is very complex and costly to run, you can replace part of the physics with a simplified, trained model — what we call physics-informed neural networks.

Skills and the Untapped Potential of Time Series

Daniel: What would you say to someone just starting out in Earth observation?

Jordi: You can come to remote sensing from both ends — from the geosciences and physical modelling side, or from the computer vision and AI side — but you need to be interested in both. Think of the data science Venn diagram: computer science and coding, applied math and statistics, and domain expertise. You don’t need to be excellent at all three, but you need some knowledge of each. I don’t think there’s a future for people in Earth observation who cannot code, and I’d recommend Python — there are lots of libraries: rasterio, GeoPandas, Fiona, plus PyTorch, TensorFlow, scikit-learn.

Daniel: Is there a part of Earth observation you feel isn’t getting the attention it deserves?

Jordi: We are not fully exploiting the richness of high-resolution time series. The Landsat program has been running 30 or 40 years, and the Sentinels provide data every five or six days over the whole Earth — yet for most applications we use just a couple of dates, or even a single date, to produce maps. The time-resolution aspect is not fully exploited. The problem is that with time series you very quickly deal with huge amounts of data, which requires expensive computational resources. Platforms like Google Earth Engine and Microsoft’s Planetary Computer open the possibility — it’s partly just inertia in the remote sensing community, where for many years we’ve been using just one or two images.