ESA is making plans for a global Sentinel2 reprocessing in 2019 to enhance multi-temporal registration

A good piece of news, directly from ESA's S2 project manager: ESA is now making plans for a global reprocessing of Sentinel-2 archive in 2019. As explained here, Sentinel-2 data multi-temporal registration isn't perfect yet. It should be improved in the first quarter of 2019, thanks to the use of ground control points obtained via automatic matching, but the issue with the reprocessing of Sentinel-2 archive had not been addressed. It is now, but of course not before 2019.

 

I let you imagine the amount of processing required to do so for the complete 3.5 years archive of Sentinel-2 at that time, so it will be costly and require hard work, but yet it is indispensable. Let's thank ESA and Copernicus for considering this and letting us know !

 

[MUSCATE news] Early Sentinel-2B L2A images over France are available

=>

MUSCATE sentinel-2 L2A counter just reached 80000 products, but a lot more products were processed these past weeks. Indeed, the MUSCATE team started processing the early Sentinel-2B images acquired from July to October 2017, which were missing on our catalog. Indeed. We only started Level 2A treatments in October 2017, in real time (a bit late, but before the ESA;)). We take advantage of this processing to reprocess Sentinel-2A data, since the quality of the products from MAJA improves with the repetitiveness of the observations. And since we replace old versions with new ones (v 1.7), the product counter is not affected by the reprocessing of S2A data.

 

The reprocessing of data on France and its overseas territories is finished, and the treatment of the other European zones is in progress. We will continue soon with the African zones then those of the rest of the world. If you are using the Sentinel-2 data acquired in the second half of 2017, we encourage you to download them again. We took advantage of this reprocessing to also produce snow products in the France area.

 

 

 

 

 

[Nouvelles de MUSCATE] Le traitement au niveau 2A des anciennes données Sentinel-2B sur la France est terminé

=>
Le compteur de produits de Niveau 2A de Sentinel-2 de MUSCATE vient de franchir les 80000 produits, mais en fait, MUSCATE en a produit beaucoup plus. En effet, l'équipe d'exploitation (Joel, Raquel, Sylvie) a commencé à traiter les données de Sentinel-2B acquises de juillet à octobre 2017, qui manquaient dans notre catalogue. Nous n'avions démarré les traitements de niveau 2A qu'en octobre 2017, en temps réel (avec un peu de retard, mais avant l'ESA ;) ). Nous profitons de ce traitement pour retraiter les données Sentinel-2A, puisque la qualité des produits issus de MAJA s'améliore avec la répétitivité des observations. Et comme nous remplaçons les anciennes versions par les nouvelles (v 1.7), le compteur de produits n'est pas affecté par le retraitement des données S2A.

 

 

Le retraitement des données sur la France et ses territoires d'outre-mer est terminé, et le traitement des autres zones Européennes est en cours. Nous poursuivrons prochainement avec les zones Africaines puis celles du reste du monde. Si vous utilisez les données Sentinel-2 acquises au deuxième semestre 2017, nous vous encourageons donc a les télécharger à nouveau. Nous avons profité de ce retraitement pour produire également les produits neige sur la zone France.

 

Theia started the distribution of Venµs products in Near Real Time


Great news for our little Venµs satellite project: Theia has started distributing Venµs L1C data in near real time. The data acquired yesterday are already available on https://theia.cnes.fr

 

 

These L1C products are tagged with 0.9 version, which means that their quality is not as good as what we think we will be able to deliver in a couple of months. The multi-spectral and multi-temporal registration can still be improved, even if it is not too far from our expectations yet in most cases. The image quality teams are finishing an error budget of the current situation, and also preparing the improvement with our Israeli partner who manage the satellite.  We will tell you about that.

 

The L2A delivery will also start in a few weeks, but the L1C teams, who had much more work than expected with this satellite, handled us correct products very late, and we still need to tune a few parameters to provide good quality products.

 

On the Theia website, you will find several ways to download the tiles. My little download tool has also been updated. Once you have registered, and updated the config.cfg file, you will be able to download at once all the products in Australia with the following command line :

python theia_download.py -l 'Australia' -c VENUS -a config.cfg -d 2018-01-01

The data format is explained here. Its packaging with useless zips and tars is still provisional and will be simplified very soon.

 

 

Let's ask ESA to improve Sentinel-2 multi-temporal registration

Update from May 4th 2018 : ESA has started planning for a global reprocessing in 2019

 

The information gathered by Sentinel-2 system on Sentinel-2 orbit, attitude, date accuracy, and viewing directions of all detectors allows an excellent accuracy for the geolocation of all Sentinel-2 pixels. The overall geo-location accuracy is better than 11 or 12 meter, for about 97 % of the cases, which is about the size of one Sentinel-2 pixel. Such a performance is more or less equivalent to that obtained for Pleiades, but Pleiades has a resolution of 0.7m. It is therefore really an achievement, which is  to be credited to ESA, to the satellite and instrument manufacturer, and to the image quality teams (including my CNES colleagues).

 

But even if it is excellent, it is not enough. The standard need for multi-temporal registration errors is 0.3 pixels, and the current performances show that for more than 50% of the cases, the performance does not meet that requirement.

From Sentinel-2 data quality report

 

 

For many  users, I guess, these figures do not mean much, and it is not easy to figure out their impact on real life applications. The animation below (made by the twitter star Simon Gascoin) makes that much more concrete :

Time series of Sentinel-2 images of the construction of Nour solar power station near Ouarzazate in Morocco. (made by Simon Gascoin using Sentinelhub)

 

Continue reading

MAJA now corrects thin cirrus clouds on Landsat-8 and Sentinel-2 images

=>
Thanks to our collaboration with  DLR,, who developed the method [1], and with an ESA funding, we added within MAJA a correction for thin cirrus clouds that works for LANDSAT 8 and Sentinel-2. This correction uses the cirrus band  at 1.38 µm, which allows to estimate cirrus reflectance, which is then subtracted from the other bands with an empiric factor derived from the images.

 

Continue reading

Correction des cirrus sur les images de Sentinel-2 et Landsat 8

=>

Grâce à notre collaboration avec le DLR, qui a mis au point la méthode [1], nous avons mis en place dans MAJA une correction des cirrus fins pour LANDSAT 8 et Sentinel-2. Cette correction utilise la bande cirrus située à 1.38 µm, qui permet d'estimer la réflectance des cirrus qui est ensuite soustraite des autres bandes avec un facteur de proportionnalité calculé empiriquement.

 

Continue reading

[MUSCATE news] MAJA upgraded to version 2

Yesterday, the MUSCATE ground segment started delivering Sentinel-2A L2A products using the MAJA V2 processor, while MAJA V1 was used until now. MAJA V2 corrects for a few bugs (such as the pixels flagged as cloud or cloud shadow on the edge of the images) and adds three new features :

  • We implemented a correction for directional effects for a better estimate of Aerosol optical thickness. As you probably know, MAJA uses multi-temporal criteria to detect clouds and estimate aerosol content. These criteria suppose that surface reflectance does not change much from one date to the next one. Over a given tile, we can combine data coming from two adjacent orbits, with slightly different viewing angles, and the directional effect can induce some variations in the surface reflectance of successive acquisitions, which would be interpreted as atmospheric effects. We are now using a simple directional correction to reduce this noise source (D.Roy et al). As this directional correction is not perfectly accurate, it is not applied to the surface reflectance we deliver at the end of MAJA processing.

Elsa Bourgeois, from Cap Gemini, compared the performances of processing with or without directional correction. In both cases (S2A, top, S2B, bottom), there is only a very slight improvement of performances.

  • We implemented a cirrus correction, that comes from DLR works. This correction is not in production yet, as it needs more validation on a large data set. We simply parametrise MAJA with "cirrus_correction= false". A coming post will address this subject.
  • We have started using the new high cloud threshold that was refined in this post. It will result in more cloud cover, with a large difference over mountains.

 

All these small improvements are now implemented in version 1.6. They do not justify a reprocessing of the Sentinel-2 archive, although the accumulation of successive slight improvement could justify it. We intend to start a reprocessing when MAJA V3 is put in production.

Speed-up downloads from PEPS S2 mirror site with peps_download.py

The French Sentinel mirror site, PEPS, has a very clever data management facility. All the products are stored on tapes, with a capacity of several PB, and there is some sort of cache made of disks. The products accessed recently are on disks, while the other products stay on tapes. The storage costs and also power consumption are therefore largely optimized.

 

The drawback is that before accessing a file on tape, some time is needed to get the tape, and read the file on tapes. This can take something like 2 to 10 minutes. My little tool, peps_download.py was designed when most of the products were on disks, and it was quite slow to download products on tapes. As I am not a patient person, I have tried to speed it up, and it works well, thanks to good advise from CNES peps  colleagues (Christophe Taillan and Erwann Poupart).

 

The previous version was working like that :

Make catalog request

For all product in the request result :

- while product is not downloaded

 - try to download the product

 - if still on tape, wait for 2 minutes

As a result, for each product on tape, it was necessary to wait for 2 to 10 minutes.

 
Now, it works like that

Make catalog request

For all products on tape in the request result

- ask to read it on disks

While (still some products to download):

- Redo catalog request

- Download products on disk

- If some products are not on disk yet

 - wait for 2 minutes

 
On my computer, it used to take more that 12 hours to download 2 years of Sentinel-2 data for a given tile. It has now been reduced to less that 3 hours (but my computer is on CNES network). I hope you will have similar results !

Validation systématique des produits Sentinel-2 de Theia

=>

Camille Desjardins (du CNES (DSO/SI/MO)), qui s'occupe de la validation des produits de niveau 2A distribués par THEIA, a très récement mis en place une validation systématique des produits fournis par MAJA, avec l'aide d'un service d'exploitation du CNES (DNO.OT/PE). Le travail est effectué par Bruno Besson avec l'aide de Nicolas Guilleminot (de Thales Services), en utilisant des outils développés par Aurélie Courtois, elle aussi de Thales).

 

Systématiquement, tous les mois, les valeurs d'épaisseurs optiques d'aérosols et de vapeur d'eau, déterminées par MAJA, sont comparées avec celles du réseau Aeronet, à chaque fois qu'un produit om THEIA observe l'un de ses sites.

 

Les deux figures ci-dessous montrent les résultats obtenus pour le mois de Février, pour l'épaisseur optique, à gauche, et pour la vapeur d'eau (à droite). Les points bleus correspondent à des validations effectuées dans les conditions idéales pour une bonne comparaison (peu de nuages, pas de données interpolées, données Aeronet qualifiées au niveau 2 ). Les points rouges correspondent à des conditions dégradées, et pour la plupart d'entre eux, c'est dû au fait que les données Aeronet qualifiées ne sont pas encore disponibles.  Dans ce cas, il est possible qu'elles soient dégradées, en raison d'une dérive de leur étalonnage... ou de la présence d'une araignée dans le tube du collimateur.

 

Validation des épaisseurs optiques d'aérosols des produits Sentinel-2 N2A de Theia  pour toutes les observations conjointes avec Aeronet en février 2018 Validation de la vapeur d'eau des produits Sentinel-2 N2A de Theia  pour toutes les observations conjointes avec Aeronet en février 2018

 

Continue reading