THEIA/MUSCATE nears real time for Sentinel-2 L2A

THEIA MUSCATE production of Sentinel-2 L2A products nears real time, and L2A products acquired until March 2017 over France and Reunion Island have started to appear on our distribution server :

https://theia.cnes.fr

The production will progressively be extended over the whole France and over the other sites (Spain, Morocco, Belgium Tunisia, Senegal, Burkina, Mali...) and then follow the real time acquisitions with the shortest delay possible.

 

La production par MUSCATE des données de niveau 2A de Sentinel-2 s'approche du temps réel, et les données acquises jusqu'à mars 2017 sur la France et l’île de la Réunion ont commencé à apparaître sur le serveur de distribution.

https://theia.cnes.fr

 

La production va être progressivement étendue à toute la France et aux autres sites (Espagne, Maroc, Belgique Tunisie, Sénégal, Burkina Faso, Mali...), et suivra ensuite les acquisitions en temps réel avec le plus court délai possible


Quantitative comparison of cloud masks from MACCS/MAJA, Sen2Cor and GEOSYS (hand made)

=> 

As already explained in a previous post, we obtained some Sentinel-2 hand made cloud masks from GEOSYS company. We used those to validate the cloud masks from MACCS/MAJA. But we wanted to use them further to make a quantitative comparison with Sen2Cor cloud masks.
But this comparison required solving a little issue : GEOSYS cloud masks are generously dilated to avoid any risk to let a cloud pass through the operational processing. Those of MACCS:MAJA are also dilated while those of Sentinel-2 are not at all. In the following paragraphs, we'll explain how we solved that issue. Sen2cor (v2.3.0) has also three levels of cloud mask (High Medium and Low probability). We used here the Medium Probability mask. But let's start with the final result comparing the performances of Sen2Cor and MACCS:MAJA.

 

Overall accuracies for MACCS/MAJA, in red and Sen2cor, in blue for 11 images compared to GEOSYS cloud masks.

Continue reading

5000 produits L2A Sentinel-2A ajoutés sur le serveur Theia

=>

:) :) :) Après avoir répété pendant des mois, "désolé, notre production a pris du retard pour telle ou telle raison", dans des articles de blog, au téléphone, en réunion, par email, sur twitter, vous ne pouvez pas imaginer le plaisir que j'éprouve à annoncer que le CNES vient de mettre en ligne de grandes séries de données Sentinel-2A, au niveau 2A, sur la France, la Réunion, le Nord de l'Espagne, le et des sites au Burkina Faso, en Ethiopie, et  en Tunisie (partiellement encore).  La période traitée s'étend sur un an, du 1er décembre 2015 au 30 novembre 2016. Au total, plus de 5000 tuiles sont disponibles. Les traitements sur les zones du Mali, du Maroc et du Sénegal vont commencer, puis ce sera le tour de Madagascar et des différentes zones affichées ici. Les données ont été traitées avec la chaîne MACCS.

Les données sont disponibles ici :

https://theia.cnes.fr/

N'oubliez pas de de consulter la page d'aide du site, nottamment pour accéder à la documentation du format du produit et pour installer l'outil de téléchargement.

Continue reading

MACCS qualified for Atmospheric Correction Euro and World Cup

Don't worry, I have not been converted to a football fan, it is still giving me allergies ! But after a post about sex ratio, we are still trying to increase the amount of connexions to this blog.

 

However, the title of this post is partly true, since, simultaneously, we are engaged to two competitions to compare cloud screening and atmospheric correction softwares applicable to Sentinel-2 or Landsat-8. The Atmospheric Correction Euro is organised by ESA, to select the method that will be used by ESA to systematically produce Sentinel-2 images to level 2A. The Euro world cup will last one year, with successive iterations between 4 producers, and a Jury of experts. Of course MACCS is participating, and we (DLR + CNES/CESBIO) are using this opportunity to merge MACCS and ATCOR methods, in order to use the methods from each of these two codes. In this framework, the resulting software we will get was renamed MAJA (say MAYA), for MACCS-ATCOR Joint Algorithm.

 

The Atmospheric Correction World cup is organized by NASA and ESA in the framework of the committee on earth observation satellites (CEOS). Here, about 12 codes will be compared, and the comparison will be done with LANDSAT 8 and Sentinel-2 data, it will seemingly focus on atmospheric correction rather than cloud screening, although it is not always easy to separate both aspects.

 

The World Cup will start this summer, while the Euro started last month. I have already shown a few validation results from this exercise :

- cloud masks comparison with Sen2cor

- validation of the consistency of processing among various tiles

- and thanks to Camille Desjardins (CNES), we just obtained validations of AOT estimates and Water Vapour estimates. The water vapour validation results are quite accurate and largely good enough to perform the atmospheric correction. The AOT validation results are decent, with rmse a little above what we observed for LANDSAT 8 over France. But the choice of sites is more difficult, with 2 arid sites, and the repetitivity of Sentinel-2 acquisitions is still far from what it should be (because of issues on board or on the ground segment). As already said in this blog, the accuracy of MACCS software increases with repetitivity.

Water vapour content validation wrt aeronet data for 4 sites (France, Morocco, Israel, SouthAfrica)

Aerosol optical thickness validation wrt aeronet data for 4 sites (France, Morocco, Israel, SouthAfrica)

 

We only are in the early stage of the EURO, and our wise coach did not introduce the best players in the first rounds. In next round, we will introduce MACCS 5.0, with its much enhanced shadows mask, and for the final stages, we will include the selected modules from ATCOR. But here is a glimpse of the MACCS 5.0 player.
On some images, the previous version of MACCS missed quite a lot of shadows, while with 5.0, they are detected (click on the images below to see the details). And by the way, the images were created within the Sen2Agri system, which is now taking shape and is a little ahead of MUSCATE in terms of versions. Many thanks to our colleagues from the Sen2Agri consortium (UCL, C-S France, C-S Romania, CESBIO).

Image generated at MUSCATE with MACCS 4.8, clouds and shadows are circled in green

Image generated at Sen2Agri with MACCS V5.0. The detected shadows are circled in yellow

And finally, I found an analogy between Football and Atmospheric Correction (I had to search a lot). Some say that football is a game which is played by two teams of eleven players and for which, at the end, Germany wins. For the Atmospheric Correction Euro, all the four teams competing include German institutes or companies. So here also, Germany wins.

New game : spot the artefacts in S2A L2A mosaics

The validation of MUSCATE, CNES ground segment to process Sentinel-2 data,  and MACCS (our L2A software) is going on, in order to start processing Sentinel-2A data at level 2A soon (in a few weeks...). A previous post compared MACCS cloud masks with those of Sen2cor. For this one, time series have been generated for 4 sites, and on each site, 4 sentinel-2 tiles were processed.

 

As MACCS processes tiles independently, there is a risk of having discrepancies when merging the tiles in one image. However, this risk is low, as MACCS does not estimate a constant aerosol optical thickness (AOT) per tile but produces AOT images for each tile. Since it uses the same data in the overlap region of each tile, the results should be quite alike in each tile. However, because the AOT is smoothed on each tile, some differences may still exist.

 

So, we decided to check, and produced some of mosaics which are shown below. Each image of 210x210 km2 is made from 4 tiles of 110*110km2 which overlap by 10 km. They have been subsampled for a quicker display.  I let you find the artifacts, as I didn't find them... These artifacts would appear as vertical or horizontal lines in the middle of the images. Should you like to repeat the same experiment with a different atmospheric correction, my code snippet (using gdal and imagemagick) is available here.

Provence, France, 2016/03/26 (Copernicus/ESA/CNES/CESBIO)

 

Middle East, Jourdain Valley, Sinai, 2016/04/06
(Copernicus/ESA/CNES/CESBIO)

On this image, the diagonal, highly visible in the North West corner, is not an artefact of MACCS, it is the border between Egypt and Israel.

 

Pretoria, South Africa, 2016/04/17(Copernicus/ESA/CNES/CESBIO)

Atlas Mountains, Morocco, 2016/04/07(Copernicus/ESA/CNES/CESBIO)

If the Atlas mountains look strangely flat on this image, it is because MACCS corrects for the illumination differences due to terrain. But if you plan to climb the Toubkal (4167m), be sure it is not flat and that MACCS will not help you.

(Wikipedia photograph, not mine...)

 

 

And finally, maybe did you see the African "cloudless mosaic" released by ESA from Sentinel-2 images. Well indeed, it shows very well the acquisition potential of Sentinel-2 but if you dislike artefacts... don't zoom !

It is only level 1C, not corrected from atmospheric effects, without any cloud detection, remaining clouds are not flagged, it is just a NDVI maximum composite. Not very useful... except for press releases.

(Copernicus/ESA/Brockmann/UCL)

Nouveau jeu, cherchez l'artefact dans les mosaiques de Niveau 2A de Sentinel-2

=>

La validation de MUSCATE et MACCS se poursuit, dans le but de démarrer prochainement le traitement des données de niveau 2A (dans quelques semaines). Un post précédent comparait les masques de nuages issus de MACCS avec ceux de Sen2cor, la chaîne de l'ESA. Des séries temporelles ont récemment été traitées sur 4 sites, et sur chaque site, 4 tuiles ont été traitées.

 

Comme MACCS traite les tuiles indépendamment les unes des autres, il y a un risque que des différences apparaissent lorsqu'on fusionne les tuiles adjacentes d'un même produit. C'est la mauvaise connaissance des aérosols qui peut introduire des erreurs de correction atmosphérique, et des différences peuvent apparaître d'une tuile à l'autre si on utilise des propriétés optiques des aérosols constantes par tuile. Ce n'est pas le cas avec MACCS, qui estime des images d'aérosols. La région où les tuiles se superposent contient les mêmes données dans chaque tuile et doit fournir la même quantité d'aérosols. Le risque d'ecart entre tuiles est donc faible. Cependant, comme on effectue des lissages par tuiles, de petites différences peuvent exister..

 

Nous avons donc décidé de vérifier, et nous avons produit des mosaïques de 4 tuiles qui sont affichées ci-dessous. Chaque image de  210x210 km2 est composée de 4 images de 110*110km2 qui se superposent de 10 km. Je vous laisse donc chercher les artefacts, pour ma part, je n'en ai pas trouvé. Ces artefacts pourraient apparaître sous la forme de lignes verticales ou horizontales, au milieur de l'image.  Si vous souhaitez répéter cette expérience avec un autre logiciel de correction atmosphérique, je vous laisse utiliser mon petit code (basé sur gdal et imagemagick) : il est disponible ici.

Provence, France, 2016/03/26(Copernicus/ESA/CNES/CESBIO)

Moyen orient, vallée du Jourdain, Sinai, 2016/04/06
(Copernicus/ESA/CNES/CESBIO)

Sur cette image, la diagonale nettement visible dans le coin Nord Ouest n'est pas un artefact de MACCS, c'est la frontière entre Israël et Égypte.

 

Pretoria, Afrique du Sud, 2016/04/17(Copernicus/ESA/CNES/CESBIO)

Montagnes de l'Atlas, Maroc, 2016/04/07(Copernicus/ESA/CNES/CESBIO)

Si les montagnes de l'Atlas vous paraissent bizarrement aplaties sur cette image,c'est parce que MACCS corrige les  variations d'éclairement dues au relief. Mais si vous avez l'intention de gravir le Toubkal (4167m), soyez bien sûrs que ce n'est pas plat, et que MACCS ne vous aidera pas..

(La photo n'est pas de moi, elle vient de Wikipedia)

 

Enfin, peut-être avez vous vu la mosaïque Africaine "sans nuages" publiée par l'ESA à partie de Sentinel-2, et dont on fait tout un plat. Certes, elle met bien en valeur le potentiel d'acquisition de la mission Sentinel-2. Mais si vous n'aimez pas les artefacts... ne zoomez pas !

C'est du Niveau 1, sans correction atmosphérique, et sans détection de nuage, les nuages résiduels ne sont pas identifiés, la méthode utilisée cherche juste le maximum de NDVI. Bref,  pas très utile... sauf pour les communiqués de presse.

(Copernicus/ESA/Brockmann/UCL)

First test of Sen2cor 2.2.1 (and comparison with MACCS)

A couple of days before the living planet symposium, ESA issued a new version of Sen2cor (V2.2.1), the Sentinel-2 atmospheric correction toolbox. I had made a first comparison of MACCS and Sen2cor results using V2.0.6, regarding the cloud masks mainly, in which I concluded the scene classification was really bad, but probably due to bugs and lack of validation, and that we should wait for next version.

 

First of all, I found the installation of Sen2cor straightforward, at least on an Ubuntu 14.04 linux platform. You just have to read the doc and do as advised. The command line is also very simple and processing time correct, 30 minutes per tile (when processing without MNT), which is a little faster than MACCS (which does much more things). And just bravo ! to the team for being able to do that on all kinds of platforms.

 

I have made my first tests on a scene in Morocco (tile n°29RNQ), which I had already analysed with MACCS runs. If you look at the figures below, you will see that the claim of a large improvement of the scene classification is perfectly true. The obtained results are logical given the information used to detect the clouds, ie thresholds on reflectances and reflectances ratio, but no multi-temporal stuff. I have drown the contours of cloud masks in green, the contours of snow masks in pink and the contours of water in blue. For Sen2cor, I selected the clouds with medium or high probability, plus the high clouds. Here are the images, I will comment them below.

 

 

Sen2cor MACCS
2016-03-18 2016-03-18
2016-04-07 2016-04-07
2016-04-17 2016-04-17

What first strikes is the thickness of the contours in Sen2cor, which is due to the fact that scene classification is done at full resolution with Sen2cor, and at a lower resolution with MACCS and then interpolated. The edges of the clouds in sen2cor are made of a patchwork of cloud/no cloud pixels, which give a greater thickness to the contours. I know that's subjective but it seems that sen2cor finds too many clouds, for instance in the March image, at the North West of the scene.

 

On the second image, Sen2cor still detects too many clouds, and confuses snow and clouds even more than MACCS (which is far from perfect with partial snow cover).

 

Finally, on the third image, Sen2cor also misses a few clouds (small ones, but also some very bright low clouds, this is strange and I do not know how it happens. MACCS uses a 400m buffer around each cloud, in case it has fuzzy edges. This is very useful for all types of clouds except for the small cumulus clouds, which invade the north-eastern part of the image, where some clear pixels are lost. But to reduce the buffer only on cumulus clouds, it would be necessary to identify the type of cloud, which is not that easy.

 

One may also note that the tints of the images are very similar, which shows that the atmospheric corrections are equivalent. and the stability with time of these colours also show that they are good !

 

More data should be processed to confirm this first impression, but we may conclude that Sen2cor was indeed improved, and even if some classification errors can be found, the available information seems to be well used. As already said, MACCS seems more accurate in terms of cloud detection  thanks to the use of multi-temporal criteria, but because of that, it is less easy to use.

 

In a few weeks, you should have access to Sentinel-2 data processed with MACCS, either within THEIA ground segment, or through the Sen2Agri package.

 

 

 

 

 

First comparison of Sentinel-2 cloud masks delivered by MACCS and SEN2COR

 

=>

A new version o Sen2cor has been published recently, and this comparison was re-iterated with Sen2cor 2.2.1. The corresponding post is published here.

 

While CNES is getting ready to produce and distribute Sentinel-2A  products obtained with our MACCS processor, I have been asked by impatient users what I thought of SEN2COR Sentinel-2 cloud masks. In this aim, I have downloaded SEN2COR and made a few runs. SEN2COR works on all sorts of multi-platform, and is rather easy to install and to run in its nominal configuration, which is not the case of MACCS, which is intended to be implemented in ground segments, and only works on a Red Hat environment. However, I have been able to process the same date on two sites, and here are the results I obtained.

 

MACCS

SEN2COR

Comparison of MACCS and SEN2COR cloud masks on a cloud free image of Toulouse. The contours of detected clouds (green), shadows (yellow), water (blue), and snow (pink) are overlayed on the images.
Continue reading

Première comparaison des masques de nuages Sentinel-2 produits par MACCS et SEN2COR

=>

Une nouvelle version de Sen2cor a été publiée récemment, et nous avons refait la comparaison. Les résultats sont publiés ici.

 

Alors que le CNES se prépare à produire et distribuer des données de niveau 2A pour Sentinel-2 à partir de notre processeur MACCS, quelques utilisateurs impatients m'ont demandé ce que je pensais du masque de nuages fourni par l'outil SEN2COR distribué par l'ESA. Pour me faire une idée, j'ai téléchargé SEN2COR et je l'ai fait tourner sur quelques produits. SEN2COR fonctionne sur plusieurs types de plate-formes, dont Linux, et il est plutôt facile à installer et utiliser dans sa configuration nominale.

 

 

MACCS 

SEN2COR 

Comparaison des masques de nuages obtenus par MACCS et SEN2COR sur une scène complètement claire acquise sur Toulouse. Les contours des masques sont superposés aux images et tracés en vert pour les nuages, en jaune pour leurs ombres, en bleu pour l'eau, et en rose pour la neige.

Continue reading