THEIA : Une nouvelle zone de production des données Sentinel-2 L2A au Sahel

=>

La production des données Sentinel-2 au niveau 2A, avec la chaîne MAJA, par le centre MUSCATE de Theia est maintenant rapide et efficace, nous disposons donc de quelques marges pour ajouter de nouvelles zones (les zones actuelles sont visibles ici). Les organismes français intéressés par l'ajout de nouvelles zones peuvent d'ailleurs me contacter, il devrait nous rester une petite cinquantaine de tuiles à choisir, compte tenu des demandes déjà reçues.

 

Certains collègues bien informés ont devancé cet appel, et parmi eux, Santiago Pena Luque,  du CNES, qui travaille pour le projet SWOT. Dans le cadre de la préparation du projet, deux grands bassin fluviaux africains ont été choisis pour concentrer les premières expérimentations. Il s'agit des bassins du Niger et du Sénégal. Les applications concerneront le suivi des cours d'eau bien sûr, mais aussi de l'occupation des sols et de la dynamique de la végétation. Plusieurs laboratoires y seront impliqués, dont le GET, le CESBIO, le LEGOS à Toulouse, le laboratoire TETIS à Montpellier.

Voici donc la zone acceptée par Theia :

 

Nouvelle zone de traitement Sentinel-2 sur le Sahel . En vert les tuiles déjà traitées, en bleu, les tuiles que nous allons ajouter.

La zone a été établie à partir des contraintes suivantes :
- ne pas dépasser 300 tuiles
- proposer une zone contiguë
- éviter les zones les plus nuageuses, comme les côtes du golfe de Guinée
- couvrir la quasi totalité des bassins du Sénégal et du Niger (à l'exception des zones systématiquement nuageuses ou complètement désertiques)
- compléter éventuellement des zones administratives connexes: nous avons pu par exemple couvrir la totalité du Sénégal, de la Gambie, du Burkina Faso, tout l'Ouest et le Sud du Mali, le Nord des guinées, de la Côte d'Ivoire, du Bénin et du Nigeria, le Sud du Niger et l'Ouest du Tchad.

 

La production va démarrer très prochainement, avec les données acquises en décembre 2016. Il nous faudra probablement quelques mois pour rejoindre le traitement en temps réel. Celà va donner beaucoup de travail à nos équipes de production, mais c'est pour la bonne cause.

 

Sentinel-2 Level3A time series (July, August, September 2018)

If you are not afraid to spend too much time while you have urgent things to do, you may have a look to the mosaic of Sentinel-2 monthly syntheses for September over France. You may access to each monthly synthesis using the following links :

 
Or you may also use the viewer below to compare with the previous months and see how France became brown in September :
 

See it full screen

The monthly syntheses are produced using the WASP processor, which is described here.

By comparing the various syntheses, you will see the evolution of the landscape, generally much brownler in September, but this representation will also help you spot the composite artefacts. These are not very numerous, but you will see them :

  • on some web browsers (firefox V58), geometrical differences appear even at a low resolution. Other browsers and versions do not have this defect. It is really not due to Sentinel-2 or Theia products
  • above water and snow (we must work on this defect)
  • where clouds have covered a place during the whole month of July or August. These pixels are flagged as invalid in the products (but not on the mosaic).
  • where clouds or shadows were not properly detected by MAJA
  • at the edges of Sentinel-2 swath. For the first time, in october, a swath edge is clearly visible near Cambrai. The area must have been quite cloudy, and we observe here a greener part on the right, observed later in October, that the browner part on the left. The only way to correct this kind of atefact while keeping a physical meaning to the reflectances, would be to improve Sentinel-2 revisit time
  • some tile edges in July, due to the fact that Level 3A products were not all generated for the 15th of July, but for dates between the 8th and the 26th. This has been corrected for the next months

 

[Nouvelles de MUSCATE] Un été très productif

=>

Mise à jour le 4 octobre 2018

Cet été, alors que la plupart d'entre nous prenions des coups de soleil sur la plage, nous faisions dévorer par des moustiques, attrapions des courbatures en rando, ou faisions de longues siestes pour nous remettre de nuits fatigantes, les équipes de production et de distribution de MUSCATE ont eu la chance de bénéficier d'une climatisation efficace, de fauteuils confortables, d'ordinateur rapides et de l'accès à la meilleure cantine du monde au CNES. Il n'est donc pas surprenant que la production ait fait de gros progrès, mais nous pouvons néanmoins leur adresser un grand merci, car les résultats sont impressionnants.

 

Amélioration des performances de production

Theia's L2A counter reached 100 000 images on August 12th

 

Tout d'abord, l'anomalie qui avait perturbé la production à la fin du printemps a été résolue. C'était un problème de capacité de mémoire sur la machine qui héberge le catalogue interne de MUSCATE. Petit à petit, la besoin de mémoire dépassait le volume prévu, et les performances se dégradaient avant que le système ne plante, quasiment tous les jours. Il a suffi de quelques ajustement et d'un doublement de la mémoire pour que le problème disparaisse.

Ainsi, depuis début juillet, MUSCATE n'a plus connu que deux courtes interruptions dues à une phase de maintenance du centre informatique du CNES. Comme on peut le voir sur la figure qui suit,  la courbe orange, qui moyenne la production sur un mois n'a jamais été aussi haute, depuis le démarrage de la production.

 

Number of L2A produced each day by MUSCATE (after removing the products with more than 90% of clouds)

Continue reading

[MUSCATE News] A very productive summer

=>

Updated on October 4th

 

This summer, while most of us were getting sunburnt near the coast, or getting sore legs after hiking in mountains, or having long naps after hectic nights, the lucky MUSCATE teams stayed at work, with a good and efficient air conditioning, convenient armchairs, fast computers and accessed to the best canteen in the world at CNES. So it's no surprise they progressed a lot in all aspects, but even if they benefited from perfect conditions, we should thank them a lot.

 

Improved production performances

Theia's L2A counter reached 100 000 images on August 12th

 

First of all, the colleagues solved the issue that delayed a lot MUSCATE production in late spring. It was a memory issue, with a heavy swap consumption that slowed then crashed the MUSCATE ground segment. Doubling the available memory on the catalogue server solved the issue.

As a result, since beginning of July, MUSCATE had only two interruptions of production, due to maintenance interventions on the CNES platform. As it may be seen below, the monthly average plot in orange shows that the production rate is at its best in two years.

 

Number of L2A produced each day by MUSCATE (after removing the products with more than 90% of clouds)

Continue reading

On-demand SENTINEL2 L2A processing with MAJA on PEPS

Update : added a script to submit and download MAJA jobs on PEPS

 

Getting a Sentinel-2 L2A product (with atmospheric correction and cloud detection) anywhere has never been so easy.  Until now, you had two options to access to Sentinel-2 L2A products obtained from the MAJA processor :

 

 

Starting from today, there is another possibility: an on-demand MAJA processing from the French Sentinel collaborative ground segment PEPS.  The possibility is a proof of concept, and if a lot of people use it at once, we do not know how the processing time will evolve. But the only way to know it is to try it, hoping the processing and download capacity is sufficient. Anyway, it you intend to process a whole country, we hope you are from Belgium, and not from Australia.

 

However, if your site is above 60°N or under 60°S, you are still unlucky, as MAJA needs SRTM DEM to work. We will try to implement the use of Arctic DEM one day.

 

PEPS/MAJA processing page (don't worry, english language is also available)

Continue reading

A seamless and cloudless Sentinel-2 image of France in July 2018

=>
UPDATE: resolution improved to 20m !
 
I had to tinker a couple of days with Theia's Sentinel-2 monthly syntheses, in order to produce the mosaic below. For that, I used:

  • monthly syntheses of Sentinel-2 L3A products delivered by Theia in July 2018; this products are made from Level2A products, corrected from atmospheric effects, and provided with a good cloud mask, thanks to MAJA processor. Such L3A products will be delivered every month.
  • a script developed by Simon Gascoin
  • good advice from Simon and Michel Le Page
  • and Gdal, (Thanks Gdal !)

This mosaic best resolution is 20m. It is already requiring 8 GB. We could have provided it at 10m resolution, but it would have required 32 GB and several dozens of hours of computation.


See it full screen

Continue reading

[MUSCATE news] Two new sites added to Sentinel-2 L2A production : Lebanon and Telangana (India)

After a very difficult period, and thanks to the installation of new improvements MUSCATE ground segment is back in shape, and margins have been found to add new sites.

Number of L2A products produced every day.

 

We have added two new zones where Sentinel-2 data are processed to Level 2A, which provide surface reflectance after atmospheric correction and a good quality cloud mask, tanks to MAJA processor. These two sites are Lebanon and Telangana region in India. The data are processed in near real time, since May 2018, and we will later on add the acquired before that date.
As usual, the data are available for download from https://theia.cnes.fr

[Venµs news] Distribution of Level2A has started

You might have noticed the apparition of the first Venµs L2A products on Theia web site within the real time production, since last Friday.  A first global processing will start this summer, to provide you with the data acquired from November until now. There will be probably further reprocessings to benefit from the fine tuning of all the parameters and to propagate the further evolution of Level 1 improvement.

Even if it took us a few months to check the software and set the parameters up, what took us very long... was waiting for the level 1 validation and calibration phase. As you know, our colleagues from CNES did a great work to rescue the Venµs raw data which were full of surprises. They started to provide us with calibrated products in April only, and that's when we started the validation.

 

We were quite happy with the first results, as our processor MAJA did not show any bug, and the first images looked good.  But the first validation results were quite poor, with undetected thin clouds, with biases in the estimates of atmospheric properties (Aerosol, water vapour), as well as biases in reflectances (with a lot of negative values). We then started iterating tests on the parameters, and after several iterations we corrected several errors in the parameters (Venµs band numbers are different from those of Sentinel-2, and in a couple of cases, I forgot to change them:( ), and we tuned better all the thresholds. Among those, we had to change the calibration of band 910 band by 6% (this band is hard to calibrate in flight due to the presence of water vapour and is also affected by some newly discovered stray light).

 

 

The following table compares the results we had initially, on the left, and the results obtained after tuning the parameters, on the right. Of course, what we distribute is on the right ! We will of course need to increase the number of validation points, and we expect that the low level stray light in band 910 that was discovered during the commissioning phase and is not yet corrected will introduce some site related bias in the water vapour estimates. We will therefore need a reprocessing after this defect has been fixed, if the Level 1 team finds a way to fix it. And finally, we have still some issues to solve with the shadows mask which can often be quite poor.

 

 

Before tuning After Tuning

RGB Quicklook with cloud mask contour

RGB Quicklook with cloud mask contour

Water vapour in g/cm2 compared with Aeronet

Water vapour in g/cm2 compared with Aeronet

Aerosol Optical Thickness compared with Aeronet (sorry for the scale different from that on the right)

Aerosol Optical Thickness compared with Aeronet (sorry for the scale different from that on the left)