A pause in MUSCATE production, end of February

The MUSCATE production centre will be offline for a week from February 25th to March 4th (or maybe the week after, please see Theia's RSS feed to see the exact date). This pause will be necessary to upgrade the processing centre to V2.5. As a result, we will not be able, for a week, to process Sentinel-2 data in real time, and hope it will not be too inconvenient to you. In case you need some data urgently, you can ask PEPS to produce MAJA L2A data for you during that period.

 

The main change regards the internal format used by MUSCATE for Sentinel-2 Level 2A products. This internal format is different from the external format that we distribute, and this results in unnecessary product format conversions, as well as the necessity to develop drivers for the various formats (internal and external) for the processors that use Sentinel-2 L2A data within MUSCATE. To enable this modification, it will be necessary to convert all the L2A data from the internal format to the external format, which will take a whole week.

 

MUSCATE V2.5 will bring other improvements, such as the integration of MAJA 3.1, with possibility to use Copernicus Atmosphere data, or a new version of LIS, the processor that delivers the snow maps.

MUSCATE  V2.6 is also ready and queuing to be installed, with MAJA 3.2, WASP and the possibility to process Venµs L2A data within MUSCATE and not externally on Venµs ground segent.

 

 

 

S1-Tiling, on demand ortho-rectification of Sentinel-1 images on Sentinel-2 grid

=>

Sentinel-1 is currently the only system to provide SAR images regularly on all lands on the planet. Access to these time series of images opens an extraordinary range of applications.
In order to meet the needs of a large number of users, including our needs, we have created an automatic processing chain to generate "Analysis Ready" time series for a very large number of applications. Sentinel-1 data is ortho-rectified on the Sentinel-2 grid to promote joint use of both missions.

 

 

Continue reading

S1Tiling : ortho-rectification à la demande des données Sentinel-1 sur la grille Sentinel-2

=>

​Sentinel-1 est actuellement le seul système à fournir des images SAR régulièrement sur toutes les terres de la planète. L'accès à ces séries temporelles d'images ouvre un champ d'application hors du commun.

Afin de répondre aux besoins d'un grand nombre d'utilisateurs, dont les nôtres, nous avons créé une chaîne de traitement automatique permettant de générer des séries temporelles "prêtes à l'emploi" pour un très grand nombre d'applications. Les données Sentinel-1 sont ortho-rectifiées sur la grille Sentinel-2 pour favoriser l'usage conjoint des deux missions.

[Nouvelles de MUSCATE] Un été très productif

=>

Mise à jour le 4 octobre 2018

Cet été, alors que la plupart d'entre nous prenions des coups de soleil sur la plage, nous faisions dévorer par des moustiques, attrapions des courbatures en rando, ou faisions de longues siestes pour nous remettre de nuits fatigantes, les équipes de production et de distribution de MUSCATE ont eu la chance de bénéficier d'une climatisation efficace, de fauteuils confortables, d'ordinateur rapides et de l'accès à la meilleure cantine du monde au CNES. Il n'est donc pas surprenant que la production ait fait de gros progrès, mais nous pouvons néanmoins leur adresser un grand merci, car les résultats sont impressionnants.

 

Amélioration des performances de production

Theia's L2A counter reached 100 000 images on August 12th

 

Tout d'abord, l'anomalie qui avait perturbé la production à la fin du printemps a été résolue. C'était un problème de capacité de mémoire sur la machine qui héberge le catalogue interne de MUSCATE. Petit à petit, la besoin de mémoire dépassait le volume prévu, et les performances se dégradaient avant que le système ne plante, quasiment tous les jours. Il a suffi de quelques ajustement et d'un doublement de la mémoire pour que le problème disparaisse.

Ainsi, depuis début juillet, MUSCATE n'a plus connu que deux courtes interruptions dues à une phase de maintenance du centre informatique du CNES. Comme on peut le voir sur la figure qui suit,  la courbe orange, qui moyenne la production sur un mois n'a jamais été aussi haute, depuis le démarrage de la production.

 

Number of L2A produced each day by MUSCATE (after removing the products with more than 90% of clouds)

Continue reading

On-demand SENTINEL2 L2A processing with MAJA on PEPS

Update : added a script to submit and download MAJA jobs on PEPS

 

Getting a Sentinel-2 L2A product (with atmospheric correction and cloud detection) anywhere has never been so easy.  Until now, you had two options to access to Sentinel-2 L2A products obtained from the MAJA processor :

 

 

Starting from today, there is another possibility: an on-demand MAJA processing from the French Sentinel collaborative ground segment PEPS.  The possibility is a proof of concept, and if a lot of people use it at once, we do not know how the processing time will evolve. But the only way to know it is to try it, hoping the processing and download capacity is sufficient. Anyway, it you intend to process a whole country, we hope you are from Belgium, and not from Australia.

 

However, if your site is above 60°N or under 60°S, you are still unlucky, as MAJA needs SRTM DEM to work. We will try to implement the use of Arctic DEM one day.

 

PEPS/MAJA processing page (don't worry, english language is also available)

Continue reading

Speed-up downloads from PEPS S2 mirror site with peps_download.py

The French Sentinel mirror site, PEPS, has a very clever data management facility. All the products are stored on tapes, with a capacity of several PB, and there is some sort of cache made of disks. The products accessed recently are on disks, while the other products stay on tapes. The storage costs and also power consumption are therefore largely optimized.

 

The drawback is that before accessing a file on tape, some time is needed to get the tape, and read the file on tapes. This can take something like 2 to 10 minutes. My little tool, peps_download.py was designed when most of the products were on disks, and it was quite slow to download products on tapes. As I am not a patient person, I have tried to speed it up, and it works well, thanks to good advise from CNES peps  colleagues (Christophe Taillan and Erwann Poupart).

 

The previous version was working like that :

Make catalog request

For all product in the request result :

- while product is not downloaded

 - try to download the product

 - if still on tape, wait for 2 minutes

As a result, for each product on tape, it was necessary to wait for 2 to 10 minutes.

 
Now, it works like that

Make catalog request

For all products on tape in the request result

- ask to read it on disks

While (still some products to download):

- Redo catalog request

- Download products on disk

- If some products are not on disk yet

 - wait for 2 minutes

 
On my computer, it used to take more that 12 hours to download 2 years of Sentinel-2 data for a given tile. It has now been reduced to less that 3 hours (but my computer is on CNES network). I hope you will have similar results !

Yesterday's snow cover area in the Pyrenees

Olivier pointed to me that ESA's ground segment, PEPS and MUSCATE were all in really good shape today... And the sky was clear yesterday at the time of the Sentinel-2A acquisition!

So I could download the Level-2A product from theia.cnes.fr, run our let-it-snow processor, start QGIS and here it is: the map of yesterday's snow cover area at 20 m resolution. If you know the region, you might notice that there is currently a big contrast in the snow cover extent between the French and the Spanish Pyrenees. This is due to the blocking of the moist air masses coming from the north.

Snow cover area on 22 Nov 2017. blue: snow, grey: no snow, white: cloud.

Stay tuned! Theia should start to distribute these Sentinel-2 snow products in near real time very soon.

[Muscate news] temporary slowlyness

Update : MUSCATE is back on track for the real time production, thanks to PEPS return to nominal production, and a better stability of our platform.

Some of you will have probably noticed that Theia production of Sentinel-2 L2A data is quite slow these days, and is not producing all the data it should in real time. The reason is we started to implement a correction to solve the performance issues we had. This correction was tested and qualified during several days, in operational conditions,  but when put in production, turned out to be unstable, due to occasional slowness in CNES cluster which did not happen when the correction was in tests. Moreover, our source of Sentinel2 data, PEPS, was experiencing some difficulties, providing a much reduced number of images.

 

We are waiting for a correction of the correction (the cause has been found and corrected but needs to be tested), The new version should be installed in less than 2.3 days, as Dilbert (dilbert.com) says. And all the team is very sorry for the delays and the inconvenience it is causing.

New version of PEPS (French Sentinel mirror site)

As you probably know, PEPS is the French Collaborative ground segment for Copernicus Sentinel program. And, first of all, it is a mirror site that distributes all the Sentinel data in near real time. These last weeks, real time was not available for Sentinel-2, as the data format and structure of Sentinel-2 products had deeply changed, and the software needed adaptation. PEPS team created a new collection, named "Sentinel-2 Single Tiles", coded "S2ST" to separate the old format from the new one. Now that the new version has been installed and validated, the PEPS mirror site is once again up to date.

Continue reading