Major changes for Venµs data processing

We are about to implement a major change in the ground segment of VENµS. Until now, L1C and L2A products were produced in the Venµs Image Processing (VIP) center, and imported into THEIA catalog. Next week, this will be changed.

 

L1C products will still be produced by the VIP, but the L2A will be generated by MUSCATE center within Theia. Moreover, it will soon be possible to generate L3A products which are bi-weekly composites of cloud free surface reflectances. And the cherry on the cake, all these products will be delivered at 5m resolution, while the L2A products were only available at 10m resolution.

 

This change comes with some drawbacks for users : the formats will slightly change to keep homogeneity between all the products distributed by MUSCATE.

  • the metadata format is different, the keywords have changed.
  • the images are still provided with TIFF format, but there is now one file per band. For L2A only, The scaling factor has changed, it is now 10000, as for Sentinel-2.
  • the L2A resolution is now 5m !
  • the grid used corresponds to that of Sentinel-2, you only have to account for the upper left origin of each site to obtain the registration of the images. No reprojection and no resampling is necessary.

 

These changes will be first applied to the real time processing, but the whole archive will be reprocessed later on.

 

A description of the new formats is provided in the following references :

 

During a few days, both versions will coexist in Theia distribution server. The new products contain the "_XS_" string in their filenames, so they are easy to recognise.

 

MAJA and LIS selected for the next Pan European high resolution snow and ice monitoring service

The European Environment Agency (EEA) has selected a consortium led by Magellium to implement the next Pan European high resolution snow and ice monitoring of the Copernicus Land Monitoring Service.

EEA 34 countries and corresponding Sentinel-2 tiles

This future service aims to monitor three variables at 20 m resolution over the tiles displayed above:

  1. Fractional Snow Cover (FSC)
  2. Permanent Snow Line (PSL)
  3. River and Lake Ice (RLI)

Continue reading

Le catalogue Theia/Muscate compte maintenant 200 000 produits Sentinel2 de niveau 2 et 3

=>

Le centre de production THEIA MUSCATE a franchi une nouvelle étape dans la nuit du 26 au 27 juin 2019. Il y a maintenant 200 000 produits Sentinel-2 N2A et N3A dans notre catalogue. Il a fallu deux ans et demi pour passer le cap des 100 000 produits en août dernier , mais nous avons doublé ce nombre en moins d'un an. C'est la preuve que notre système de production est maintenant mature et robuste, même s'il demande encore beaucoup de travail de la part de l'équipe d'exploitation, que nous remercions beaucoup !

 

Mais comme vous pouvez le voir ci-dessous, si le nombre de produits Sentinel-2 est le plus élevé, et aussi le plus téléchargé, le nombre de types de produits que nous fournissons aux utilisateurs augmente régulièrement. Nous vous parlerons bientôt du produit " qualité de l'eau" qui vient d'apparaître à la fin de la liste.

 

200 000 Sentinel-2 L2A and L3A products in Theia catalog

=>

THEIA MUSCATE production center passed a new milestone during the night of the 26th to 27th of June 2019. There are now 200 000 Sentinel-2 L2A and L3A products in our catalog generated with MAJA. It had taken two years and half to pass the 100000 products milestone last august, but it took less than a year to double that number. This is a proof that our production system is now mature and robust, even if it still requires a lot of work from the exploitation team, which we thank a lot for their hard work !

 

But as you can see below, if the  number of Sentinel-2 products is the highest, and also the most downloaded by far, the catalogue has grown a lot, and the number of products we provide to the users is steadily increasing. We will soon tell you about the water quality product that just appeared at the end of the list.

 

 

THEIA's Sentinel-2 L2A processing on Sahel is progressing

=>

As we had announced in November, the MUSCATE production centre in Theia is gradually adding areas in the Sahel, which are shown in the image below. The data are processed from December 2016 onwards, which means that we have a large amount of data to process. So we started with the most westerly tiles, in Senegal on the UTM28 zone, then progressed from one zone to another towards the East.


Theia Sentinel-2 processing area on the Sahel

 

In red, the tiles available from Dec 2016 to NRT, in blue , the remaining tiles to be added.

In recent days, Theia has completed the processing of tiles in the UTM29 area, which mainly covers northern Guinea and western Mali, but also partially covers southern Mauritania and Sierra Leone and north-western Côte d'Ivoire. The treatment of the UTM 30 zone, which covers Burkina Faso and Mali, is also well advanced. The east of this area is finished, and the west is progressing well, as shown in the animation below. The UTM31 zone has also been brought into production. Feel free to take a look from time to time at the map of areas covered by MUSCATE. The blue tiles turn red as soon as we switch to run-of-river processing.

 

The data can be downloaded from here:

https://theia.cnes.fr

 

Animation in the region of the city of Mopti, Mali, with about one image per month in 2017. The displayed time series extends between two rainy seasons and covers the dry season. Many fire scars are visible during the dry season. Some shadows appear, which actually correspond to the shadows of cirrus clouds corrected by MAJA. Shadows and cirrus are marked in the products.

Le traitement des données Sentinel-2 L2A au Sahel progresse

=>

Comme nous l'avions annoncé en Novembre, le centre de production MUSCATE de Theia rajoute progressivement des zones au Sahel, qui sont affichées sur l'image ci-dessous. Les données sont traitées à partir de décembre 2016, ce qui nous fait une grande quantité de données à traiter. Nous avons donc commencé avec les tuiles les plus à l'ouest, au Sénegal sur la zone UTM28, puis de proche en proche vers l''Est.


Zone de traitement Sentinel-2 sur le Sahel proposée à Theia. En vert foncées les tuiles qui déjà traitées par Theia, en teintes claires, les tuiles que nous ajoutons progressivement.

 

Depuis quelques jours, Theia a terminé le traitement des tuiles de la zone UTM29 qui

En rouge, les zones déjà disponibles en temps réel et depuis fin 2016, en bleu, les zones qui le seront bientôt.

couvre principalement le Nord de la Guinée et l'Ouest du Mali, mais aussi partiellement le sud de la Mauritanie et la Sierra Leone et le Nord Ouest de la Côte d'Ivoire. Le traitement de la zone UTM 30 qui couvre le Burkina Faso et le Mali, est lui aussi bien avancé. L'est de cette zone est terminé, et l'ouest avance bien, comme le montre l'animation ci-dessous. La zone UTM31 a elle aussi été mise en production. N'hésitez pas à jeter un coup d’œil de temps en temps à la carte des zones couvertes de MUSCATE. Les tuiles en bleu deviennent rouges dès que l'on passe au traitement au fil de l'eau.

 

Les données peuvent être téléchargées ici :

https://theia.cnes.fr

 

Animation sur la région de la ville de Mopti, au Mali, avec environ une image par mois en 2017. La série temporelle s'étend entre deux saisons des pluies et couvre la saison sèche. De nombreuses cicatrices d'incendies sont visibles pendant la saison sèche. Quelques ombres apparaissent, qui correspondent en fait aux ombres des cirrus corrigées par MAJA. Les ombres sont marquées dans les produits.

Plus de 1000 téléchargements en deux ans pour le code de correction atmosphérique MAJA

=>

Nous avons commencé à distribuer MAJA gratuitement en juillet 2017, il y a moins de deux ans, et le logiciel a déjà été téléchargé plus de 1000 fois (1068 précisément aujourd'hui). Celà représente en moyenne plus de deux téléchargements par jour de semaine, et ces dernières semaines, nous avons atteint environ 4 téléchargements par jour. Bien sûr, c'est probablement beaucoup moins que Sen2cor, qui dispose d'un financement confortable pour le rendre facile à installer et à utiliser sur un ordinateur personnel, sous Windows, IOS ou Linux. Mais les deux logiciels ne jouent pas dans la même catégorie: Sen2cor a été conçu pour fonctionner du côté client, tandis que MAJA a été conçu pour être robuste et efficace dans les environnements de production.

 

MAJA ne fonctionne que sur linux, et ses algorithmes multi-temporels le rendent moins facile à utiliser. MAJA fait des calculs beaucoup plus complexes que Sen2cor, en deux fois moins de temps que Sen2cor grâce à une bonne parallélisation utilisant la bibliothèque Orfeo Toolbox C++ Library. Malgré cette complexité, beaucoup d'utilisateurs semblent avoir réussi à le faire fonctionner.


Comparaison de séries temporelles d'images de MAJA (Gauche) and Sen2cor (droite) sur la région de Naples. Les masques de nuages sont soulignés en vert, et les ombres en jaune. Cliquez sur l'image pour l'agrandir.

La version de MAJA que nous recommandons actuellement est la version 3.3 qui apporte de nombreuses améliorations.

Deux outils permettent de faire tourner MAJA efficacement :
- Sen2Agri, pour lequel MAJA (et son ancienne version MACCS) a été téléchargé environs 700 fois
- Start-MAJA, qui est un petit orchestrateur que j'ai d'abord développé pour mon usage personnel, puis publié en open source sur github. Il a été téléchargé 450 fois et a reçu 18 étoiles sur la plateforme github du CNES , et 45 sur mon dépôt github personnel, où il était disponible avant de basculer vers celui du CNES.

Les téléchargements de STart-MAJA sur github pendant la dernière quinzaine

 

Cependant, il n'est pas facile de savoir combien d'utilisateurs ont réussi à l'installer correctement, et combien ont échoué. Nous savons que certains utilisateurs parviennent à l'installer facilement. Nous recevons souvent des commentaires et des questions lorsque les utilisateurs ne réussissent pas immédiatement, et nous ne connaissons pas la proportion de ceux qui abandonnent. Nous serions heureux de recevoir plus de retours sur les difficultés rencontrées et sur le taux de succès. N'hésitez pas à nous informer !

MAJA est un logiciel développé par la société CS-SI pour le compte du CNES (service DNO/OT/IS, Imagerie Spatiale). Les méthodes ont été définies par le CESBIO avec des apports du DLR. D'autres services du CNES (DSO/SI/MO, physique de la mesure optique) et compagnies (Magellium, Cap Gemini, Thales-IS) on apporté leur concours à son développement et sa validation.

 

More than 1000 downloads of MAJA atmospheric correction software

=>

We started distributing MAJA as a free software in July 2017, less than 2 years ago, and it already reached 1000 downloads (1043, as a mater of fact).  MAJA was in fact downloaded twice per working day, and in the very last weeks, we had about 4 downloads a day ! Of course it is probably much less than Sen2cor, which is a software with a comfortable  funding to make it easy to install and use on a personal computer, under Windows, IOS or Linux systems. But both software are not in the same category, Sen2cor was designed to run on client side,  while MAJA was designed  to be robust and efficient in production environments.

 

MAJA only works on linux systems, and its multi-temporal features make it less easy to use. MAJA does much more complex computations than Sen2cor, about twice faster thanks to a good parallelization using the Orfeo Toolbox C++ Library. Still, despite this complexity, a lot of users seem to have managed to make it work and seem to be happy with it.

Comparison of MAJA (left) and Sen2cor (right) time series over Naples, Italy. Cloud masks are outlined in green, shadows in yellow. click on the image to enlarge.


We recommend the use of the latest version 3.3 which brings a lot of improvements.

 

Users can run MAJA through 2 main launchers:

  • Sen2Agri, for which MAJA (or its former MACCS version) was downloaded almost 700 times
  • Start-MAJA, which is a little scheduler I developed first for my own usage and then released as open source on github. It has been downloaded 450 times and has received 18 stars on the CNES github platform, and 45 on my github repository, where it was available before moving to CNES's.

Last fortnight clones of Start-MAJA scheduler

However, it is not easy to know how many users managed to install it properly, and how many failed. We know some users manage to install it easily. We often receive feedback and questions when users do not succeed at once, and we do not know the proportion who give-up. We would be happy to receive feedback emails and know if you succeeded, and if you are happy with the results.

 

 

Sen2Agri final version was just published

The Sen2Agri system was just released in version 2.0. As the project is now terminated and will not receive more funding from ESA, V2_0 is the final version, apart maybe for a few bug fixes.

 

Sen2Agri is a very complete ground segment that automates download and processing of Sentinel-2 data to obtain multi-temporal syntheses, crop/non crop, and crop type masks as well as biophysical variables and phenological indicators. It was designed to be able to process the Sentinel-2 data over whole countries.

 

The Sen2AGri system has been a huge success. I receive an email each time MACCS/MAJA is downloaded, and I just counted 651 downloads after sorting them out the double download,

The new version includes the most recent version of MAJA (3.3) for atmospheric correction and cloud detection (but without using the CAMS data, which are not always easy to access), an early version of WASP to make the composites, an early version of Iota2 to obtain crop types, a processor to obtain the crop/non Crop mask, and a processor to compute biophysical variables inspired by the works of F.Baret and M.Weiss at INRA.

 

The consortium, funded by ESA, was lead by Sophie Bontemps et Pierre Defourny at the Université Catholique de Louvain. The system was developed by CS-SI France and Romania, and most of the methods were designed in CESBIO.

 

 

 

MAJA 3.3 is available, with a LOT of improvements

What's new ?

Pfew ! It has been quite long, but MAJA 3.3 is available, and it improves a LOT of things !

  • Some bugs have been fixed, like the one which caused detection of cloud or cloud shadows on the edges of the images
  • It seems we have finally solved the bugs that plagued the CAMS option since we released MAJA V3.0. Since V3.0, this option uses the Copernicus Atmosphere aerosol forecasts to set the aerosol type before retrieving the aerosol optical thickness (AOT) from Sentinel-2 data
  • We now also use CAMS AOT as a default value, when it is not possible to estimate AOT using the images, for instance above a snow covered landscape or for small gaps in a large cloud cover. Before that, we used 0.1 everywhere as a default value. The default value is used in the cost function with a very low weight, it has no impact when conditions for AOT estimates are good, but a large impact in bad conditions.
  • The cirrus correction module was over correcting the impact of thick cirrus clouds, providing images with dark clouds. We have limited the correction in order to get more realistic values
  • We have improved the cloud detection, with a better compromise between false positives and false negatives. We also handle better the variation against altitude of the cirrus cloud detection with band 10 (1.38 µm). MAJA 3.3 is the version with which we obtained the results of our recently published article. This paper shows that MAJA has slightly better performances than FMask 4.0, and much better performances than Sen2Cor.

Moreover, when we validated the results, we figured out that one of the parameters in our settings had a wrong value (10 instead of 1). It is easy to make such errors, because there are about 150 parameters in MAJA, and it's easy to make an error. We have set up a version management of MAJA settings since 2017, but the erroneous value was already there before that. And this value has a big impact ! The standard deviation of errors in AOT estimates is reduced by 30 to 40% !!

The W_dark parameter controls the weight of the dark pixel method in the AOT estimation. This method is just supposed to be used as a safeguard in case the multi-temporal or multi-spectral methods provide wrong results. It should therefore have a low weight, but with a weight of 10, it was in fact the method which had the highest weight in our estimates. As this method provides a maximum value of the AOT, it tended to reduce the provide too low AOTs. This improvement is therefore a great piece of news, but it comes with some shame not to have found this error before.

 

AOT validation against AERONET for 10 sites with the wrong W_dark value AOT validation against AERONET for 10 sites with the correct W_dark value

The blue dots correspond to validation obtained in good conditions, while red dots correspond to less reliable validation points.

These are the results of comparison of version 3.3 with CAMS activated, changing only the W_dark parameter, but if we compare with the results of version 3.1, the improvement is even more impressive :

AOT validation against AERONET for 10 sites, version 3.1/td> AOT validation against AERONET for 10 sites version 3.3, with the correct W_dark value

How to access MAJA ?

Here is how to access MAJA 3.3 :

  • MAJA 3.3 is distributed as a free software for non commercial purposes from CNES free software site (select the 3.3 TM version in the download tab). If you need it for commercial purposes, you just have to ask me for a different licence, but it still will be free of charge.
  • The best way to use MAJA is to run it with Start_MAJA, which is a simple python code that runs MAJA for a whole time series for a given Sentinel-2 tile. The Strart MAJA readme also explains how to get the good settings, with the good W_dark value, how to prepare the DEM or how to get CAMS data.
  • PEPS on-demand processing facility will be updated soon, but it is still working with MAJA 3.2 so far
  • THEIA is also running MAJA 3.2. We will update first the wdark parameter, and then MAJA 3.3 and then start production with CAMS, hopefully before summer. If everything goes well, we will then start a reprocessing of all our data set. So, stay tuned on this information channel.