Update January 31st, 10h00 : Distribution was just restarted, a few hundreds of images will be added today, and meanwhile, data are processed in Near Real Time.
This January was a nightmare for MUSCATE. Following database issues to send data to the distribution server, our production system was stopped during the Christmas break and unstable the weeks after.
After understanding the issues, the MUSCATE team stopped the automatic sending of products, resumed the production, and started to update the production server manually. We were nearly back on track last Thursday, when the distribution server refuse to accept any new product. The explanation was found, a directory in the High Performance Storage System (a robot that handles tapes and disks) had 65535 files and could not accept a new one. We need a little reorganisation of the folder structure to overcome that, and meanwhile, the distribution is stalled again.
Still, more than 60 000 L2A products are now available, and we have started distributing the snow cover products, in NRTWD ("Near real time with delay"). We hope to be soon really in NRT.
As you have probably noticed, our production rate has been very low these days and we are more than 10 days late in our delivery of L2A and snow cover products.
This seems to be due to an intervention on CNES cluster end of December to add new nodes and disk space. MUSCATE sometimes loses communication with the platform that handles the databases and crashes. As it also happened when CNES was closed for Christmas, we really lost a lot of time. All the teams are on the deck to try to solve this issue and catch the delay up. We are very sorry for that inconvenience.
Since spring 2017, we have made the MAJA cloud screening and atmospheric correction processor available for commercial use. A bit later, end of June, the Sen2agri software package, which includes MAJA older version (named MACCS) , was also released to the public. We did not expect a large success, as these two packages are quite heavy ones, do not work on laptops, and require a specific linux system powerful computers (Red Hat or CentOS).
Anyway, the MAJA processor has had quite a large success, even if, I guess, it is far from the success of Sen2cor, which is much easier to install and use, even if the performances are not the same. The figures below correspond nearly to one download per day.
|Number of downloads of MAJA (stand alone version)
|Number of downloads of MACCS (Sen2Agri version)i
To celebrate this fact, we just published a brand new MAJA detailed description.
I have always wanted to provide an Algorithm Theoretical Basis Document related to MAJA, but never had time, because I always had more urgent things to do. Some papers had been published, allowing MAJA users to get a good idea or the methods we use, but the published articles did not cover all the features of MAJA.
But this time, due to a contractual engagement with ESA, it was the urgent thing to do. So, at last, after a few weeks of hard work, here it is.
If you have already read the papers from our team, you will recognize some text published quite a long time ago, but we updated all the text and added some parts which had not been explained yet in journal publications, and of course the new parts recently added to MAJA. This ATBD is now in line with version 2.0 of MAJA.
The ATBD can be cited with the following reference :
Olivier Hagolle, Mireille Huc, Camille Descardins, Stefan Auer, Rudolf Richter, MAJA Algorithm Theoretical Basis Document, https://doi.org/10.5281/zenodo.1209633
The main difficulty of the atmospheric correction comes from the determination of the aerosols optical properties: one has to know the optical properties of the aerosol type present in the atmosphere and determine their optical thickness. Using Sentinel-2 data to determine the aerosol type is very complicated, and our MAJA processor, used to generate Theia L2A products, only computes the aerosol optical thickness, while assuming a specific aerosol type.
The current operational version of the MAJA processor uses a constant aerosol type during the atmospheric correction, independently from the location and from the time of the year, thus affecting the quality of the atmospheric correction if the chosen aerosol type is not appropriate.
As an alternative, we tried to use the information from CAMS (Copernicus Atmosphere Monitoring Service), whichprovides forecasts of the Aerosol Optical Thickness (AOT, see figure below) of five different aerosol types: dust, black carbon, sea salt, sulfate and organic matter.
CAMS aerosol optical thickness (AOT) forecasts at 550 nm on 14 June 2016, 03:00 UTC: (top left) Dust, (top right) Sea Salt, (bottom left) Black Carbon, and (bottom right) Sulfate.
La principale difficulté de la correction atmosphérique réside dans la détermination des propriétés optiques des aérosols : il faut connaître les propriétés optiques du type d'aérosols présent dans l'atmosphère et déterminer leur quantité, symbolisée par l'épaisseur optique. Il est très difficile, à partir des données Sentinel-2, de déterminer le type d'aérosols, et notre chaîne MAJA, utilisée pour générer les produits L2A de Theia se contente de déterminer l'épaisseur optique des aérosols en supposant le type d'aérosols connu.
La version opérationnelle actuelle de MAJA utilise, durant la correction atmosphérique, un type d'aérosol constant spatialement et temporellement, ce qui affecte la qualité de la correction atmosphérique si le type d'aérosol choisi n'est pas le bon. L'alternative proposée ici est d'utiliser l'information venant de CAMS (Copernicus Atmosphere Monitoring Service), qui fournit des prévisions d'épaisseur optique (AOT pour Aerosol Optical Thickness, voir figure ci-dessous) pour cinq types d'aérosols différents : dust, black carbon, sea salt, sulfate et organic matter.
Cartes d'épaisseur optique des aérosols issues de CAMS (AOT) à la longueur d'onde 550 nm le 14 Juin 2016, 03:00 UTC: (haut gauche) pussières, (haut droit) sel de mert, (bas gauche) carbone noir, and (bas droit) sulfate.
Since it became operationnal in December last year, MUSCATE has produced 50 000 level 2A products from Sentinel-2A. Let's recall what has been processed so far :
- For 550 tiles, we have processed all Sentinel-2A data acquired since December 2015.
- For 100 tiles, mainly in South America, and in Italy, we have processed all Sentinel-2A data from December 2016. We are currently catching up the backlog for Italy, and later on, for South American sites.
- For all these 650 tiles, we are producing all Sentinel-2 data (Sentinel-2A and Sentinel-2B) in near real time. I think THEIA is the only place where you can download Sentinel-2B L2A data so far. ESA has not started that production yet (nah, nah, nah )
- For all these 650 tiles, we have processed all Sentinel-2B data since beginning of October 2017. We will soon catch-up with the Sentinel-2B data acquired from July 2017.
See full screen
Map of the 650 tiles currently processed in near real time (in red). The blue tiles will be added beginning of next year.
All these products are available from https://theia.cnes.fr
Let's recall that MUSCATE uses the MAJA L2A processor, which uses multi-temporal criteria to perform a high quality cloud detection and atmospheric correction. Despite the recent installation of version 2.4, MUSCATE still regularly suffers from instability as soon as CNES High Performance Computer is overloaded. The problem does not lie in MAJA, but in the information exchanges between all the components of MUSCATE which need to respect an accurate timing (sorry, I am not able to explain better).
The exploitation team just installed a new version of MUSCATE (v 2.4.16.p2 (!)), which is expected to increase stability. But that's the theory, let's see if it works in the coming days and if we are able to increase our production rate.
L'ESA a déclaré Sentinel-2B opérationnel a la fin du mois d'Octobre, même si des données préliminaires étaient déjà disponibles depuis juillet. Depuis hier matin, MUSCATE distribue aussi les produits Sentinel-2B de niveau 2A en utilisant le processeur MAJA (les produits L2A sont exprimés en réflectance de surface après correction atmosphérique et sont munis d'un masque de nuage de bonne qualité). La production actuelle a démarré à partir des acquisitions du 1er octobre 2017 et se poursuivra en temps réel. De plus, nous ajouterons progressivement les produits Sentinel-2B acquis depuis juillet 2017.
Premiers produits S2B disponibles sur https://theia.cnes.fr
Comme toujours, les données peuvent être téléchargées gratuitement depuis https://theia.cnes.fr
Voici un petit exemple de série temporelle près de Baotou, en Chine, acquises en alternance par Sentinel-2A et Sentinel-2B. Les images se ressemblent beaucoup, excepté là ou la surface a clairement changé, dans la vallée Est-Ouest au centre de l'image.
Comme le traitement MAJA est multi-temporel, sa précision bénéficiera de la répétitivité doublée des acquisitions, ce qui devrait avoir un effet sur nos résultats de validation à venir. A partir des productions de qualification de la nouvelle version, nous avons déjà vérifié que les réflectances fournies par les deux capteurs sont assez proches et concordent bien avec les mesures in situ obtenues avec la station de mesure de réflectance de surface du CNES à La Crau (Provence, France). Quelques exemples de résultats sont fournis ci-dessous.
ESA declared Sentinel-2B operational at the beginning of October, although preliminary data were already available. Since this morning, MUSCATE is producing and distributing Sentinel-2B Level 2A products using the MAJA processor (the L2A products are expressed in surface reflectance after atmospheric correction and are provided with a good quality cloud mask). The current production starts from the first of October 2017 and will go on in real time, and we will progressively add the Sentinel-2B products acquired since July 2017.
First S2B products available for download from https://theia.cnes.fr
As always, the data can be freely downloaded from https://theia.cnes.fr
Here is a little example of time series of iimages acquired over Baotou, China, alternatively by Sentinel-2A and Sentinel-2B, here again, the images look the same, except where something has clearly changed on the ground, in the east-west irrigarted valley in the image center.
As MAJA processing is multi-temporal, its accuracy will benefit from the doubled repetitivity of acquisition, which should have an effect on our validation results. Using the production we did to check our parameters, we have already checked that the reflectances provided by both sensors are quite close, and agree well with in situ measurements obtained with CNES surface reflectance measuring station in La Crau. The results are provided below.
You must have seen, in the press or social networks, images like these ones, showing yellow skies ( Loup de Bretagne said "Blade Runner skies"). Such a coloured sky was seen in the west of France, then later in England, and further East in the next days.
The Copernicus Atmosphere Monitoring Service, which keeps track of aerosols has also monitored these aerosols and tracked their provenance : is due to the combined presence of dust, ash from forest fires in Spain and Portugal, and humidity brought by the Ophelia cyclone. The IASI sensor on-board METOP satellites has also monitored the gases absorption due those wildfires.
So our question is: how such an event is handled by atmospheric correction software ?