THEIA's Sentinel-2 L2A processing on Sahel is progressing

=>

As we had announced in November, the MUSCATE production centre in Theia is gradually adding areas in the Sahel, which are shown in the image below. The data are processed from December 2016 onwards, which means that we have a large amount of data to process. So we started with the most westerly tiles, in Senegal on the UTM28 zone, then progressed from one zone to another towards the East.


Theia Sentinel-2 processing area on the Sahel

 

In red, the tiles available from Dec 2016 to NRT, in blue , the remaining tiles to be added.

In recent days, Theia has completed the processing of tiles in the UTM29 area, which mainly covers northern Guinea and western Mali, but also partially covers southern Mauritania and Sierra Leone and north-western Côte d'Ivoire. The treatment of the UTM 30 zone, which covers Burkina Faso and Mali, is also well advanced. The east of this area is finished, and the west is progressing well, as shown in the animation below. The UTM31 zone has also been brought into production. Feel free to take a look from time to time at the map of areas covered by MUSCATE. The blue tiles turn red as soon as we switch to run-of-river processing.

 

The data can be downloaded from here:

https://theia.cnes.fr

 

Animation in the region of the city of Mopti, Mali, with about one image per month in 2017. The displayed time series extends between two rainy seasons and covers the dry season. Many fire scars are visible during the dry season. Some shadows appear, which actually correspond to the shadows of cirrus clouds corrected by MAJA. Shadows and cirrus are marked in the products.

Le traitement des données Sentinel-2 L2A au Sahel progresse

=>

Comme nous l'avions annoncé en Novembre, le centre de production MUSCATE de Theia rajoute progressivement des zones au Sahel, qui sont affichées sur l'image ci-dessous. Les données sont traitées à partir de décembre 2016, ce qui nous fait une grande quantité de données à traiter. Nous avons donc commencé avec les tuiles les plus à l'ouest, au Sénegal sur la zone UTM28, puis de proche en proche vers l''Est.


Zone de traitement Sentinel-2 sur le Sahel proposée à Theia. En vert foncées les tuiles qui déjà traitées par Theia, en teintes claires, les tuiles que nous ajoutons progressivement.

 

Depuis quelques jours, Theia a terminé le traitement des tuiles de la zone UTM29 qui

En rouge, les zones déjà disponibles en temps réel et depuis fin 2016, en bleu, les zones qui le seront bientôt.

couvre principalement le Nord de la Guinée et l'Ouest du Mali, mais aussi partiellement le sud de la Mauritanie et la Sierra Leone et le Nord Ouest de la Côte d'Ivoire. Le traitement de la zone UTM 30 qui couvre le Burkina Faso et le Mali, est lui aussi bien avancé. L'est de cette zone est terminé, et l'ouest avance bien, comme le montre l'animation ci-dessous. La zone UTM31 a elle aussi été mise en production. N'hésitez pas à jeter un coup d’œil de temps en temps à la carte des zones couvertes de MUSCATE. Les tuiles en bleu deviennent rouges dès que l'on passe au traitement au fil de l'eau.

 

Les données peuvent être téléchargées ici :

https://theia.cnes.fr

 

Animation sur la région de la ville de Mopti, au Mali, avec environ une image par mois en 2017. La série temporelle s'étend entre deux saisons des pluies et couvre la saison sèche. De nombreuses cicatrices d'incendies sont visibles pendant la saison sèche. Quelques ombres apparaissent, qui correspondent en fait aux ombres des cirrus corrigées par MAJA. Les ombres sont marquées dans les produits.

Plus de 1000 téléchargements en deux ans pour le code de correction atmosphérique MAJA

=>

Nous avons commencé à distribuer MAJA gratuitement en juillet 2017, il y a moins de deux ans, et le logiciel a déjà été téléchargé plus de 1000 fois (1068 précisément aujourd'hui). Celà représente en moyenne plus de deux téléchargements par jour de semaine, et ces dernières semaines, nous avons atteint environ 4 téléchargements par jour. Bien sûr, c'est probablement beaucoup moins que Sen2cor, qui dispose d'un financement confortable pour le rendre facile à installer et à utiliser sur un ordinateur personnel, sous Windows, IOS ou Linux. Mais les deux logiciels ne jouent pas dans la même catégorie: Sen2cor a été conçu pour fonctionner du côté client, tandis que MAJA a été conçu pour être robuste et efficace dans les environnements de production.

 

MAJA ne fonctionne que sur linux, et ses algorithmes multi-temporels le rendent moins facile à utiliser. MAJA fait des calculs beaucoup plus complexes que Sen2cor, en deux fois moins de temps que Sen2cor grâce à une bonne parallélisation utilisant la bibliothèque Orfeo Toolbox C++ Library. Malgré cette complexité, beaucoup d'utilisateurs semblent avoir réussi à le faire fonctionner.


Comparaison de séries temporelles d'images de MAJA (Gauche) and Sen2cor (droite) sur la région de Naples. Les masques de nuages sont soulignés en vert, et les ombres en jaune. Cliquez sur l'image pour l'agrandir.

La version de MAJA que nous recommandons actuellement est la version 3.3 qui apporte de nombreuses améliorations.

Deux outils permettent de faire tourner MAJA efficacement :
- Sen2Agri, pour lequel MAJA (et son ancienne version MACCS) a été téléchargé environs 700 fois
- Start-MAJA, qui est un petit orchestrateur que j'ai d'abord développé pour mon usage personnel, puis publié en open source sur github. Il a été téléchargé 450 fois et a reçu 18 étoiles sur la plateforme github du CNES , et 45 sur mon dépôt github personnel, où il était disponible avant de basculer vers celui du CNES.

Les téléchargements de STart-MAJA sur github pendant la dernière quinzaine

 

Cependant, il n'est pas facile de savoir combien d'utilisateurs ont réussi à l'installer correctement, et combien ont échoué. Nous savons que certains utilisateurs parviennent à l'installer facilement. Nous recevons souvent des commentaires et des questions lorsque les utilisateurs ne réussissent pas immédiatement, et nous ne connaissons pas la proportion de ceux qui abandonnent. Nous serions heureux de recevoir plus de retours sur les difficultés rencontrées et sur le taux de succès. N'hésitez pas à nous informer !

MAJA est un logiciel développé par la société CS-SI pour le compte du CNES (service DNO/OT/IS, Imagerie Spatiale). Les méthodes ont été définies par le CESBIO avec des apports du DLR. D'autres services du CNES (DSO/SI/MO, physique de la mesure optique) et compagnies (Magellium, Cap Gemini, Thales-IS) on apporté leur concours à son développement et sa validation.

 

More than 1000 downloads of MAJA atmospheric correction software

=>

We started distributing MAJA as a free software in July 2017, less than 2 years ago, and it already reached 1000 downloads (1043, as a mater of fact).  MAJA was in fact downloaded twice per working day, and in the very last weeks, we had about 4 downloads a day ! Of course it is probably much less than Sen2cor, which is a software with a comfortable  funding to make it easy to install and use on a personal computer, under Windows, IOS or Linux systems. But both software are not in the same category, Sen2cor was designed to run on client side,  while MAJA was designed  to be robust and efficient in production environments.

 

MAJA only works on linux systems, and its multi-temporal features make it less easy to use. MAJA does much more complex computations than Sen2cor, about twice faster thanks to a good parallelization using the Orfeo Toolbox C++ Library. Still, despite this complexity, a lot of users seem to have managed to make it work and seem to be happy with it.

Comparison of MAJA (left) and Sen2cor (right) time series over Naples, Italy. Cloud masks are outlined in green, shadows in yellow. click on the image to enlarge.


We recommend the use of the latest version 3.3 which brings a lot of improvements.

 

Users can run MAJA through 2 main launchers:

  • Sen2Agri, for which MAJA (or its former MACCS version) was downloaded almost 700 times
  • Start-MAJA, which is a little scheduler I developed first for my own usage and then released as open source on github. It has been downloaded 450 times and has received 18 stars on the CNES github platform, and 45 on my github repository, where it was available before moving to CNES's.

Last fortnight clones of Start-MAJA scheduler

However, it is not easy to know how many users managed to install it properly, and how many failed. We know some users manage to install it easily. We often receive feedback and questions when users do not succeed at once, and we do not know the proportion who give-up. We would be happy to receive feedback emails and know if you succeeded, and if you are happy with the results.

 

 

Sen2Agri final version was just published

The Sen2Agri system was just released in version 2.0. As the project is now terminated and will not receive more funding from ESA, V2_0 is the final version, apart maybe for a few bug fixes.

 

Sen2Agri is a very complete ground segment that automates download and processing of Sentinel-2 data to obtain multi-temporal syntheses, crop/non crop, and crop type masks as well as biophysical variables and phenological indicators. It was designed to be able to process the Sentinel-2 data over whole countries.

 

The Sen2AGri system has been a huge success. I receive an email each time MACCS/MAJA is downloaded, and I just counted 651 downloads after sorting them out the double download,

The new version includes the most recent version of MAJA (3.3) for atmospheric correction and cloud detection (but without using the CAMS data, which are not always easy to access), an early version of WASP to make the composites, an early version of Iota2 to obtain crop types, a processor to obtain the crop/non Crop mask, and a processor to compute biophysical variables inspired by the works of F.Baret and M.Weiss at INRA.

 

The consortium, funded by ESA, was lead by Sophie Bontemps et Pierre Defourny at the Université Catholique de Louvain. The system was developed by CS-SI France and Romania, and most of the methods were designed in CESBIO.

 

 

 

MAJA 3.3 is available, with a LOT of improvements

What's new ?

Pfew ! It has been quite long, but MAJA 3.3 is available, and it improves a LOT of things !

  • Some bugs have been fixed, like the one which caused detection of cloud or cloud shadows on the edges of the images
  • It seems we have finally solved the bugs that plagued the CAMS option since we released MAJA V3.0. Since V3.0, this option uses the Copernicus Atmosphere aerosol forecasts to set the aerosol type before retrieving the aerosol optical thickness (AOT) from Sentinel-2 data
  • We now also use CAMS AOT as a default value, when it is not possible to estimate AOT using the images, for instance above a snow covered landscape or for small gaps in a large cloud cover. Before that, we used 0.1 everywhere as a default value. The default value is used in the cost function with a very low weight, it has no impact when conditions for AOT estimates are good, but a large impact in bad conditions.
  • The cirrus correction module was over correcting the impact of thick cirrus clouds, providing images with dark clouds. We have limited the correction in order to get more realistic values
  • We have improved the cloud detection, with a better compromise between false positives and false negatives. We also handle better the variation against altitude of the cirrus cloud detection with band 10 (1.38 µm). MAJA 3.3 is the version with which we obtained the results of our recently published article. This paper shows that MAJA has slightly better performances than FMask 4.0, and much better performances than Sen2Cor.

Moreover, when we validated the results, we figured out that one of the parameters in our settings had a wrong value (10 instead of 1). It is easy to make such errors, because there are about 150 parameters in MAJA, and it's easy to make an error. We have set up a version management of MAJA settings since 2017, but the erroneous value was already there before that. And this value has a big impact ! The standard deviation of errors in AOT estimates is reduced by 30 to 40% !!

The W_dark parameter controls the weight of the dark pixel method in the AOT estimation. This method is just supposed to be used as a safeguard in case the multi-temporal or multi-spectral methods provide wrong results. It should therefore have a low weight, but with a weight of 10, it was in fact the method which had the highest weight in our estimates. As this method provides a maximum value of the AOT, it tended to reduce the provide too low AOTs. This improvement is therefore a great piece of news, but it comes with some shame not to have found this error before.

 

AOT validation against AERONET for 10 sites with the wrong W_dark value AOT validation against AERONET for 10 sites with the correct W_dark value

The blue dots correspond to validation obtained in good conditions, while red dots correspond to less reliable validation points.

These are the results of comparison of version 3.3 with CAMS activated, changing only the W_dark parameter, but if we compare with the results of version 3.1, the improvement is even more impressive :

AOT validation against AERONET for 10 sites, version 3.1/td> AOT validation against AERONET for 10 sites version 3.3, with the correct W_dark value

How to access MAJA ?

Here is how to access MAJA 3.3 :

  • MAJA 3.3 is distributed as a free software for non commercial purposes from CNES free software site (select the 3.3 TM version in the download tab). If you need it for commercial purposes, you just have to ask me for a different licence, but it still will be free of charge.
  • The best way to use MAJA is to run it with Start_MAJA, which is a simple python code that runs MAJA for a whole time series for a given Sentinel-2 tile. The Strart MAJA readme also explains how to get the good settings, with the good W_dark value, how to prepare the DEM or how to get CAMS data.
  • PEPS on-demand processing facility will be updated soon, but it is still working with MAJA 3.2 so far
  • THEIA is also running MAJA 3.2. We will update first the wdark parameter, and then MAJA 3.3 and then start production with CAMS, hopefully before summer. If everything goes well, we will then start a reprocessing of all our data set. So, stay tuned on this information channel.

Spot the odd one out

=>

There is an odd image in this time series of L2A products of the 31TCJ Sentinel-2 tile (Toulouse region). Can you guess which one ?

 

Yes it is the last one, acquired on the 26th of February. But what's odd with it ?

  • The black South East corner ? No, just a different orbit
  • The date ? Close enough...
  • Two images separated by one day (25 th, 26 th of January) ? You're boiling !

Continue reading

Trouvez l'intrus

=>

Parmi cette série temporelle d'images de niveau 2A de la tuile 31TCJ (Toulouse), il y a une intruse. Saurez vous déviner laquelle ?

 

Oui, c'est la dernière,, acquise le 26 Février. Mais qu'a t'elle de spécial ?

  • Le coin noir au sud-est ? Oui, mais c'est juste le résultat d'une orbite diffférente. Je ne vais quand même pas écrire tout un article là dessus.
  • La date ? Oui, vous chauffez.
  • Deux images séparées par un seul jour (les 25 et 26 février ?) ? Ah, là, vous brulez !

Continue reading

WASP source and executable are now openly available

WASP (Weighted Average Synthesis Processor) is the tool we use to compute the nice (mostly) cloud free syntheses of Sentinel-2 surface reflectances, as shown in the images below. A full resolution viewer is also available in this post, or you may also download the products you can download from Theia. As promised (but it took a while to get the allowance), we have just released it as an open source software and we are also providing a compiled version for linux platforms.

The WASP method was developed at CESBIO by O.Hagolle, in 2007, during the preparation of the Venµs mission. It then evolved and improved with the help of several persons at CESBIO (V.Debaecker, M.Huc, D.Morin, M.Kadiri). Then an operational version was developed by CS Romania within the Sen2Agri consortium funded by ESA, which is distributed as open source. WASP was finally adapted to work in Theia context, and improved by P.Kettig. Peter also set up distribution of the software.

So finally, here is how you can download the software :

  • the source code is available within CNES github repository. You will also find there a forum (issues) and a readme file to compile and use the software.
  • but P.Kettig also compiled an executable version (which is tested on Redhat and Ubuntu), which is available from CNES software distribution server. Until now, only a couple of users have used it, so in case of problems, please open an issue on the github platform.

 

Sentinel-2 Level 3A products : syntheses or composites ?

=>

For the past few months, Copernicus has been distributing Level 3A products for the Sentinel-2 mission as part of the S2GM (Sentinel-2 Global Mosaics) .

 

This ambitious project aims to provide monthly, quarterly or annual Level 3A products worldwide. The user defines his zone and period of interest and orders the product. The web site seems to be hosted by SInergise, and as everything done by this company, is easy and very straightforward to use.

 

The call for tender, with two or three million euros, was launched by the Copernicus program of the European Union two years ago. It required the use of ESA Level 2A official products obtained with Sen2Cor. For this reason, we decided not to participate because our Level 3A method, which calculates a weighted average of non-cloudy observations, requires very good cloud masks, which is not quite the case of Sen2cor products.

 

The tender was won by a consortium of Brockman Consult, Geoville and SInergise companies. To compensate for the poor quality of cloud detection, the authors of the S2GM product had to use a BAP method: "Best Available Pixel". This method chooses for each pixel the best date according to certain criteria (no cloud or shadow detected, minimum reflectance in the blue, maximum NDVI ...). This method minimizes cloud disturbances when clouds are not detected correctly, but also has the disadvantage of suddenly changing the date from one pixel to another, which causes artifacts and noise. Outputs are therefore composite products , which assemble pieces of images acquired for the different dates available over the period.

 

Theia Level 3 products are not composites, but syntheses, which use all cloudless observations of a single pixel over the entire monthly observation period to find the value that best represents the surface reflectance we would have had at the central date of the product. Theia's syntheses use the WASP (Weighted Average Synthesis Processor) chain, which calculates a weighted average of surface reflectances over a month, after atmospheric correction and detection of clouds obtained from Level 2A products generated by our MAJA channel , of course. If the clouds are badly detected, they enter into the synthesis and disturb it.

Comparison of a synthesis obtained with WASP + MAJA, with a composite product from S2GM + Sen2cor, on the Toulouse region, in October 2018. (Click on image to enlarge)

The animation provided above shows a full resolution comparison over Toulouse region, of a synthesis of WASP and of the corresponding composite of S2GM obtained on the same date in October 2018. We see very quickly that the composite of S2GM is very noisy, much more than the synthesis from WASP. It is quite often possible to locate the areas where the synthesis tool has chosen to change the date in its composite. You will also notice the appearance of many white dots, which are in fact pixels without clouds, but quite bright that Sen2Cor systematically classifies as clouds.

 

In short, provided you have a good level 2A product, syntheses can provide much better results than composites.