[MUSCATE news] Distribution server in maintenance

New update: Muscate distribution server is back but you should use the following address until Monday: https://theia.cnes.fr/atdistrib/rocket/#/home . New products will be uploaded starting from Monday too. And since the week-end is there, it is not compulsary to downloard our L2A images until then (even if you can...)

Update : the server will also be down of Friday . Sorry for that

In order to solve the issue described in this post, MUSCATE distribution server is in maintenance. The distribution of new products should resume  tomorrow (Wednesday) or Thursday. Meanwhile, the production is going on. Sorry for the inconvenience.

Le serveur de distribution de MUSCATE est en maintenance pour résoudre le problème décrit dans cet article. La distribution de nouveaux produits devrait reprendre demain (mercredi) ou jeudi. Pendant les travaux, la production continue. Veuillez nous excuser pour ces perturbations.

Improvements of MAJA cloud masks in production

We are continuously working on improving the Sentinel-2 L2A products delivered by Theia. Since the 2nd of February, a new parametrization was put in production in order to improve two points of the cloud masks. There is a page which summarizes the variations of Theia product versions. Because of some error within Muscate production centre, we are not able to change the product version when we change a parameter. You have to look at the production date to know which version was used. This inconvenience will be corrected soon.

Clouds above water

Although MAJA is optimized for cloud detection and atmospheric correction over land, it is important to detect clouds well over water. For instance to detect the shadows of these clouds that are cast over land. But until now, the clouds mask was missing quite a lot of clouds over water. This was not a surprise as we had passed only a few hours determining the thresholds to apply. We recently found some more time to obtain a better tuning of the detection thresholds above water.

Thresholds in the SWIR have been halved (from 0.08 to 0.04), in the absence of sunglint. When sunglint is likely, due to the geometry of acquisition, the threshold is still higher (0.016, but it was 0.25 before).This means that when the sunglint flag is raised, the accuracy of cloud detection is reduced.

Impact of new cloud detection threshold over water : left, old threshold, right, new threshold.

Cloud dilation

 

 

 

 

The plane contrail illustrates the difference in observation angles of the spectral bands of Sentinel-2. The cumulus clouds above has the same effect, although lower because the cloud is at a lower altitude.

As it may be seen on the image above, the cloud limits are fuzzy, and except for the big and nice cumulus clouds, the pixel next to the cloud mask is also affected by clouds. Even if the cloud has sharp limits, it also changes the illumination of neighbouring pixels, and, in the case of Sentinel-2, as all the bands do not observe exactly in the same direction, there is a "parallax effect" which results in different cloud positions depending on the spectral band. For all these reasons, we need to dilate the cloud mask (by the way, this is an identified drawback of Sen2cor, which does not dilates its clouds).

 

Our dilation buffer was originally 480m. Following complaints from some users working in very cloudy countries, we had reduced the buffer to 240m in may 2017. But we recently figured out that the reduced dilation was degrading our estimates of  aerosols, as undetected clouds are considered as aerosols.  Due to this, the new version which runs since the2nd of February has once again a dilation of 480m.

Left, aerosol validation results with a dilation of 240m, right with a dilation of 480m. The new result brings a significant improvement (Merci à Bastien Rouquié pour ce résultat)

 

Users of THEIA Sentinel2 products are welcome in Toulouse, 13-14th June 2018

This is Toulouse in mid-June last year. Isn't that a nice place and time to gather the users of Sentinel-2 products delivered by Theia ?

CNES and THEIA are very happy to invite you to provide your feedback on the Sentinel-2 products we have been delivering for more than one year, thanks to the MUSCATE ground segment and the MAJA processor.

 

All users of MACCS/MAJA procesor and of THEIA products are welcome to tell us their findings, suggestions, and share experiences on methods, applications, and results.  We will also provide an update  about the processing perspectives, validation results, description of new versions, and you will have the opportunity to influence us on the related choices.

 

Please register and send your abstract to the workshop site. Here are the important dates to remember :

  • abstract submission deadline : March the 8th.
  • registration deadline : June the 3rd (we only have 100 seats, so please register quickly)
  • workshop dates 13-14th June 2018

 

 

The operational production of the Theia Snow collection has started

Great news, we can announce that the operational production of the Theia snow collection has started well. It means that maps of the snow cover area are now constantly added to the Theia portal. These maps are automatically generated from Sentinel-2 observations and have a spatial resolution of 20 m. The Snow collection will progressively cover most mountain regions in west Europe, but also the Atlas in Morocco, eastern Canada... The Snow collection can be freely downloaded from http://theia.cnes.fr by any registered user.

 
Today's front page of the Theia website featured this nice example in Sierra de Ancares (western end of the Cantabrian Mountains, Spain). In the southeast, snow was also detected on the Montes Aquilanos, including the small ski resort El Morredero. The image was captured yesterday! It illustrates well the value of multispectral imagery to discriminate the snow cover from the clouds. There is a cloud which looks alike snow but it is actually a valley fog confined by local topography.
 

Theia Sentinel-2 level 2A and snow product in the region de los Ancares, Spain. Image captured by Sentinel-2A on 30 Jan 2018.

Theia Sentinel-2 level 2A and snow product in the region de los Ancares, Spain. Image captured by Sentinel-2A on 30 Jan 2018.


Continue reading

[MUSCATE News] Production still on, but distribution stalled again

Update January 31st, 10h00 : Distribution was just restarted, a few hundreds of images will be added today, and meanwhile, data are processed in Near Real Time.


This January was a nightmare for MUSCATE. Following database issues to send data to the distribution server, our production system was stopped during the Christmas break and unstable the weeks after.

 

After understanding the issues, the MUSCATE team stopped the automatic sending of products, resumed the production, and started to update the production server manually. We were nearly back on track last Thursday, when the distribution server refuse to accept any new product. The explanation was found, a directory in the High Performance Storage System (a robot that handles tapes and disks) had 65535 files and could not accept a new one.  We need a little reorganisation of the folder structure to overcome that, and meanwhile, the distribution is stalled again.

 

Still, more than 60 000 L2A products are now available, and we have started distributing the snow cover products, in NRTWD  ("Near real time with delay"). We hope to be soon really in NRT.

[MUSCATE News] A difficult start of 2018 for our production center

As you have probably noticed, our production rate has been very low these days and we are more than 10 days late in our delivery of L2A and snow cover products.

This seems to be due to an intervention on CNES cluster end of December to add new nodes and disk space. MUSCATE sometimes loses communication with the platform that handles the databases and crashes. As it also happened when CNES was closed for Christmas, we really lost a lot of time.  All the teams are on the deck to try to solve this issue and catch the delay up. We are very sorry for that inconvenience.

 

Tous nos voeux pour 2018 !

=>

Que cette nouvelle année vous apporte joie et bonheur, et pas seulement dans l'utilisation de séries temporelles !

 

Sans aucune originalité, ce début d'année est l'occasion de faire un petit bilan de l'année 2017. Voici, dans notre domaine, quelques uns des faits que je retiendrai :

  • la consécration du programme Copernicus, et des satellites Sentinel. Plus de 110 000 personnes se sont inscrites pour accéder aux données depuis 2015 !  A mon avis, ce succès est dû à la combinaison de plusieurs facteurs : les données sont gratuites et faciles d’accès, les observations sont répétitives, régulières et fréquentes sur le monde entier, et les données sont de grande qualité. Un grand bravo à l'ESA et à l'UE, sans oublier la contribution du CNES pour la qualité des images de Sentinel-2 et l'étalonnage de Sentinel-3. Continue reading

MAJA Sentinel-2 L2A processor downloaded more than 200 times

Since spring 2017, we have made the MAJA cloud screening and atmospheric correction processor available for commercial use. A bit later, end of June, the Sen2agri software package, which includes MAJA older version (named MACCS) , was also released to the public. We did not expect a large success, as these two packages are quite heavy ones, do not work on laptops, and require a specific linux system powerful computers (Red Hat or CentOS).

Anyway, the MAJA processor has had quite a large success, even if, I guess, it is far from the success of Sen2cor, which is much easier to install and use, even if the performances are not the same. The figures below correspond nearly to one download per day.

 

Number of downloads of MAJA (stand alone version) 93
Number of downloads of MACCS (Sen2Agri version)i 116

 

To celebrate this fact, we just published a brand new MAJA detailed description.

[MUSCATE] A little error on Sentinel-2B L2A products processed before December 1st.

When we started making tests of the L2A production at MUSCATE with Sentinel-2B data, we considered using a small correction of Sentinel-2B to correct for a eventual bias between the reflectances of each satellite. Such differences were observed by colleagues form CNES, and were comprised between 0 and 2.5%.  Finally, we decided not to implement this in the operational line, because the figures we had were based on a small duration of acquisitions and were possibly not accurate enough.

 

Band name Coefficient
B1 1.002
B2 0.995
B3 1.000
B4 1.008
B5 1.009
B6 1.017
B7 1.014
B8 1.024
B8A 1.012
B9 1.000
B10 1.000
B11 1.004
B12 0.986

 

Due to a human error, it turns out that the parameter file that we used for these tests made its way to the operational line. MAJA parameters are kept under a GIT version manager, which should reduce the probablility of such errors, but that parameter file is a parameter of MUSCATE, not MAJA, which was not managed yet. And as you know, according to Mr Murphy, when something can turn wrong, it turns wrong. the test calibration adjustment made its way to the production line.

 

For the Sentinel-2B L2A products produced until the 30th of November, the L1C TOA reflectance  values were divided by the coefficients provided in the table above. On the first of December, this error was corrected. As the differences are quite small, we have not removed the Sentinel-2B products produced before December 1st, but we are going to reproduce them and replace them by the correct values during the coming weeks.

 

We are sorry for hat error, which will make us improve our verification procedure.