[MUSCATE news] MAJA upgraded to version 2

Yesterday, the MUSCATE ground segment started delivering Sentinel-2A L2A products using the MAJA V2 processor, while MAJA V1 was used until now. MAJA V2 corrects for a few bugs (such as the pixels flagged as cloud or cloud shadow on the edge of the images) and adds three new features :

  • We implemented a correction for directional effects for a better estimate of Aerosol optical thickness. As you probably know, MAJA uses multi-temporal criteria to detect clouds and estimate aerosol content. These criteria suppose that surface reflectance does not change much from one date to the next one. Over a given tile, we can combine data coming from two adjacent orbits, with slightly different viewing angles, and the directional effect can induce some variations in the surface reflectance of successive acquisitions, which would be interpreted as atmospheric effects. We are now using a simple directional correction to reduce this noise source (D.Roy et al). As this directional correction is not perfectly accurate, it is not applied to the surface reflectance we deliver at the end of MAJA processing.

Elsa Bourgeois, from Cap Gemini, compared the performances of processing with or without directional correction. In both cases (S2A, top, S2B, bottom), there is only a very slight improvement of performances.

  • We implemented a cirrus correction, that comes from DLR works. This correction is not in production yet, as it needs more validation on a large data set. We simply parametrise MAJA with "cirrus_correction= false". A coming post will address this subject.
  • We have started using the new high cloud threshold that was refined in this post. It will result in more cloud cover, with a large difference over mountains.

 

All these small improvements are now implemented in version 1.6. They do not justify a reprocessing of the Sentinel-2 archive, although the accumulation of successive slight improvement could justify it. We intend to start a reprocessing when MAJA V3 is put in production.

Speed-up downloads from PEPS S2 mirror site with peps_download.py

The French Sentinel mirror site, PEPS, has a very clever data management facility. All the products are stored on tapes, with a capacity of several PB, and there is some sort of cache made of disks. The products accessed recently are on disks, while the other products stay on tapes. The storage costs and also power consumption are therefore largely optimized.

 

The drawback is that before accessing a file on tape, some time is needed to get the tape, and read the file on tapes. This can take something like 2 to 10 minutes. My little tool, peps_download.py was designed when most of the products were on disks, and it was quite slow to download products on tapes. As I am not a patient person, I have tried to speed it up, and it works well, thanks to good advise from CNES peps  colleagues (Christophe Taillan and Erwann Poupart).

 

The previous version was working like that :

Make catalog request

For all product in the request result :

- while product is not downloaded

 - try to download the product

 - if still on tape, wait for 2 minutes

As a result, for each product on tape, it was necessary to wait for 2 to 10 minutes.

 
Now, it works like that

Make catalog request

For all products on tape in the request result

- ask to read it on disks

While (still some products to download):

- Redo catalog request

- Download products on disk

- If some products are not on disk yet

 - wait for 2 minutes

 
On my computer, it used to take more that 12 hours to download 2 years of Sentinel-2 data for a given tile. It has now been reduced to less that 3 hours (but my computer is on CNES network). I hope you will have similar results !

Validation systématique des produits Sentinel-2 de Theia

=>

Camille Desjardins (du CNES (DSO/SI/MO)), qui s'occupe de la validation des produits de niveau 2A distribués par THEIA, a très récement mis en place une validation systématique des produits fournis par MAJA, avec l'aide d'un service d'exploitation du CNES (DNO.OT/PE). Le travail est effectué par Bruno Besson avec l'aide de Nicolas Guilleminot (de Thales Services), en utilisant des outils développés par Aurélie Courtois, elle aussi de Thales).

 

Systématiquement, tous les mois, les valeurs d'épaisseurs optiques d'aérosols et de vapeur d'eau, déterminées par MAJA, sont comparées avec celles du réseau Aeronet, à chaque fois qu'un produit om THEIA observe l'un de ses sites.

 

Les deux figures ci-dessous montrent les résultats obtenus pour le mois de Février, pour l'épaisseur optique, à gauche, et pour la vapeur d'eau (à droite). Les points bleus correspondent à des validations effectuées dans les conditions idéales pour une bonne comparaison (peu de nuages, pas de données interpolées, données Aeronet qualifiées au niveau 2 ). Les points rouges correspondent à des conditions dégradées, et pour la plupart d'entre eux, c'est dû au fait que les données Aeronet qualifiées ne sont pas encore disponibles.  Dans ce cas, il est possible qu'elles soient dégradées, en raison d'une dérive de leur étalonnage... ou de la présence d'une araignée dans le tube du collimateur.

 

Validation des épaisseurs optiques d'aérosols des produits Sentinel-2 N2A de Theia  pour toutes les observations conjointes avec Aeronet en février 2018 Validation de la vapeur d'eau des produits Sentinel-2 N2A de Theia  pour toutes les observations conjointes avec Aeronet en février 2018

 

Continue reading

Systematic validation of Sentinel-2 THEIA L2A products

=>

Very recently, Camille Desjardins (from CNES), who is handling the validation of the L2A products generated by THEIA, has set up a systematic validation of the products delivered by MAJA, with the help of an operational service from CNES (OT/PE) (Bruno Besson, and Nicolas Guilleminot from Thales Services, using tools developed by Aurélie Courtois, also from Thales)

 

Systematically, a comparison of AOT and water vapour is made for every Sentinel-2 L2A product from THEIA which observes one of the sites of the Aeronet network.

 

Both plots below show the results obtained during the month of February, for the Aerosol Optical Thickness (left), and for the water vapour content (right). Blue dots correspond to validations in ideal conditions (low cloud amount, no gap filling, and quality assured Aeronet data (Level 2.0). The red dots allow degraded conditions, and most of them correspond to the unavailability, yet, of version 2.0 Aeronet data. As data are processed in near real time, and level 2.0 data are made available a few months later, these plots rely mainly on Level 1.5 data, which are more prone to errors (such as a calibration drift... or the presence of a spider in the instrument tubes).

 

Aerosol optical thickness validation of Sentinel-2 L2A for all Aeronet match-ups gathered in February 2018 Water vapour validation of Sentinel-2 L2A for all Aeronet match-ups gathered in February 2018 (in g/cm2)

 

Continue reading

Sentinel-2 goes global

Great news ! As announced in Sentinel-2 Mission status, laser links to geostationary relay satellites are now working, both for S2B (since last October) and S2B (since a few days ago). Sentinel-2 5 days repetitivity is now nominal above all lands, and that's cool ! A big thank you to ESA, Copernicus and all the engineers who strived to achieve that !

 

Map of S2A and S2B acquisition segments on the 28th of February. Almost all segments over continents were acquires, and are available on https://peps.cnes.fr

Une grand nouvelle ! Comme annoncé dans le "Sentinel-2 Mission status", les liaisons par laser vers un satellite relais géo-stationnaire fonctionnent, à la fois pour S2B (depuis Octobre 2017) et S2A (depuis quelques jours). Les deux Sentinel-2 observent les 5 continents avec la répétitivité nominale de 5 jours, et c'est chouette !  Un grand merci à l'ESA, Copernicus et tous les ingénieurs qui ont permis cette réussite !

[MUSCATE News] Retraitement des données Sentinel-2B acquises de Juillet à Octobre 2017

=>

MUSCATE est en grande forme ces jours-ci, grâce aux efforts de l'équipe de développement (CNES et CAP GEMINI), qui ont résolu de nombreux problèmes récemment. Le compteur des produits de niveau 2A de Sentinel-2 vient de dépasser 70 000, juste un mois après avoir passé 60 000. Si on fait la somme de tous les produits distribués par  THEIA, on arrive à plus de 99000. Theia distribue aussi des masques de neige sur les montagnes à partir de Sentinel-2, and des données Spot World Heritage (les anciennes données SPOT retraitées, ortho-rectifiées et disponibles gratuitement).

 

Cette grande forme nous permet d'augmenter les rythmes de production. Nous avons commencé à traiter les données Sentinel-2B acquises entre Juillet et Octobre 2017, car nous ne traitions S2B que depuis Novembre dernier. Mais comme MAJA est une méthode multi-temporelle, c'est un véritable retraitement que nous venons de lancer, en incluant à la fois S2A et S2B. La qualité des données S2A devrait elle aussi bénéficier de l'amélioration de la répétitivité des observations.

 

Ce retraitement va prendre quelques semaines. Il commence avec la France (ROM COM inclus), puis va s'étendre à nos voisins Européens. Puis viendra le tour du site sur le Maghreb, le reste des sites Africains, et enfin les autres sites répartis dans le monde.

 

Si vous avez que votre site soit traité de manière plus rapide, n'hésitez pas à demander ! (Bien sûr, cela ne s'applique qu'aux tuiles déjà présentes sur notre liste).

 

 

 

[MUSCATE News] Reprocessing of Sentinel-2B data aquired from July to October 2017

=>

MUSCATE is in a good shape these days thznkd to the continuous efforts of the development team (CNES and CAP GEMINI) who solved several issues. The counter of Sentinel-2 Level 2A products reached 70 000 products this night, just one month after reaching 60000. If we sum all the products delivered by MUSCATE, we are reaching 99 000 images. MUSCATE also distributes Sentinel-2 Snow masks over mountains, and Spot World Heritage data (old SPOT data reprocessed after ortho-rectification.and made available for free).

 

This good shape allows us to increase our production rhythm. We have started processing the Sentinel-2B data acquired between July and October 2017, as we had started processing S2B in November 2017 only. But since MAJA is a multi-temporal processor, we are in fact starting a complete reprocessing of the data, including S2A and S2B. The quality of S2A products should therefore also benefit from the improved repetitivity of observations.

 

This reprocessing will last several weeks. We are starting with data from France and will go on with our neighbouring European countries, then sites in Maghreb, the remaining sites of Africa, and finally, the rest of the sites in the world.

 

In case you have an urgent need for some tiles, please ask ! (of course, it is only applicable to the tiles already in our list)

 

 

 

Improvement of high cloud detection in MAJA for Sentinel-2 images.

=>

The research to improve the performances of our MAJA atmospheric correction processor relies on a very small team within CESBIO, with two persons, Bastien Rouquié, who is working at improving aerosol retrievals, and myself, who spend most of the time in meetings, or writing blog posts, and sometimes doing both simultaneously ;) . This is why improvements take time.

 

Atmospheric absorption : in blue, the surface reflectance of a vegetation pixel, as a function of wavelength. In red, the reflectance of the same pixel at the top of atmosphere. For a wavelength of 1.38 µm, water vapour totally absorbs the light that comes from the earth surface at sea level.

Anyway, I recently found some time to work on improving cloud masks above water. And these last days, i worked at improving the detection of high cloud using the cirrus band, which is available on Sentinel-2 as well as Landsat 8. As explained in this post, the "cirrus" band is locates in a strong absorption band of water vapour, strong enough to prevent photons from going from the sun to the satellite via the earth surface at sea level. As most of the water vapour lies in the low layers of the atmosphere, the clouds situated higher in the atmosphere have more chances to reflect light to the satellite. As a result, this channel allows us to detect the high clouds.

 

The detection method looks simple : just use a threshold on the cirrus band. Well that's not that easy !

Continue reading

Amélioration de la détection des nuages hauts dans MAJA pour les images de Sentinel-2

=>

La recherche pour l'amélioration des performances des méthodes de MAJA repose actuellement sur une toute petite équipe au CESBIO, composée de deux personnes, Bastien Rouquié, qui travaille sur l'amélioration des estimations d'aérosols, et moi, qui passe presque tout mon temps en réunions, ou a écrire des articles de blog (et parfois les deux en même temps). C'est pour cette raison que de nombreux points que nous aimerions améliorer sur MAJA ne s'améliorent que lentement.

 

Absorption atmosphérique. En bleu, la réflectance de surface pour un pixel couvert de végétation, en fonction de la longueur d'onde, en rouge la réflectance au sommet de l'atmosphère pour ce même pixel. A 1.38 µm, la vapeur d'eau absorbe totalement la lumière provenant de la surface au niveau de la mer.

J'ai pu récemment consacrer un peu de temps à l'amélioration de la détection des nuages, notamment au dessus de l'eau, et plus récemment, à améliorer la détection des nuages hauts, en utilisant la  bande "cirrus" située à 1.38 µm, qui est disponible sur Sentinel-2 et Landsat 8. Comme précisé dans ce post, la bande "cirrus" est située dans une forte bande d'absorption de la vapeur d'eau, telle qu'au niveau de la mer, les photons ont très peu de chances de faire le parcours du soleil au satellite en passant par la surface de la terre sans être absorbés. Comme la majorité de la vapeur d'eau est située dans les basses couches de l'atmosphère, les nuages situés plus haut dans l'atmosphère ont davantage de chances de renvoyer de la lumière jusqu'au satellite. C'est donc un moyen de détecter les nuages hauts.

 

La méthode de détection parait donc très simple, il suffit de faire un seuillage sur la réflectance dans la bande 1.38 µm.  Malheureusement, c'est plus compliqué que ça.

Continue reading

Another validation of CESBIO's 2016 France land-cover map

In this post, a validation of the land-cover map of France produced by CESBIO for the 2016 period was presented. This validation used independent data (that is data collected by different teams and using different procedures than the data used for the classifier training), but the validation procedure consisted in applying classical machine learning metrics which, as described in this other post, have some limitations.

A fully independent validation following a sound protocol is costly and needs skills and expertise that are very specific. SIRS is a company which is specialised in the production of geographic data from satellite or aerial images. Among other things, they are the producers of Corine Land Cover for France and they are also responsible for quality control and validation of other Copernicus Land products.

SIRS has recently performed a validation of the 2016 France land-cover map. The executive summary of the report reads as follows:

This report provides the evaluation results of the CESBIO OSO 2016 10m layer and the CESBIO OSO 2016 20m layer.

The thematic accuracy assessment was conducted in a two-stage process:

  1. An initial blind interpretation in which the validation team did not have knowledge of the product’s thematic classes.
  2. A plausibility analysis was performed on all sample units in disagreement with the production data to consider the following cases:
  • Uncertain code, both producer and operator codes are plausible. Final validation code used is producer code.
  • Error from first validation interpretation. Final validation used is producer code
  • Error from producer. Final validation code used is from first validation interpretation
  • Producer and operator are both wrong. Final Validation code used is a new code from this second interpretation.

Resulting to this two-stage approach, it should be noticed that the plausibility analysis exhibit better results than the blind analysis.

The thematic accuracy assessment was carried out over 1,428 sample units covering France and Corsica.
The final results show that the CESBIO OSO product meet the usually accepted thematic validation requirement, i.e. 85 % in both blind interpretation and plausibility analysis. Indeed, the overall accuracies obtained are 81.4 +/- 3.68% for the blind analysis and 91.7 +/- 1.25% for the plausibility analysis on the CESBIO OSO 10m layer. The analysis on the 20m layer shows us that the overall accuracy for the blind approach is 81.1 +/-3.65% and 88.2 +/-3.15% for the plausibility approach.
Quality checks of the validation points have been made by French experts. It should be noticed that for the blind analysis, the methodology of control was based mostly on Google Earth imagery, no additional thematic source of information that could provide further context was used such as forest stand maps, peatland maps, etc.

These results are very good news for us and for our users. The report also contains interesting recommendations that will help us to improve our algorithms. The full report is available for download.