MUSCATE S2 product versions

V1_0 was a preliminary processing performed with early version of MACCS adapted to Sentinel-2. It was plagued with a bug in the cloud shadow mask : when more than 255 clouds were present in one image, shadow detection went completely wrong and shadows were detected anywhere. V1_0 was not produced in MUSCATE operationnal context, but in an earlier validation context. Production was available over France only.

 

V1_1 replaces V1_0 over France, Production started mid November, but was only released in February because of many difficulties encountered by MUSCATE. It corrects for the bug related to cloud shadows observed with V1_0, and  was fully processed by the operational MUSCATE center.

 

V1_2 was used in another context not related to MUSCATE, you will not find any product with this version number within MUSCATE server.

 

V1_3 was applied starting from February 2017, to data sets produced above Reunion, Burkina, Senegal, Tunisia, Morocco.The aerosol estimate is improved compared to V1_1. Together with Bastien Rouquié, from CESBIO, we worked on the tuning of the blue-red ratio which is used in the multi-spectral method to estimate aerosols (which is combined with a multi-temporal method). Initially, we used bands B2 (Blue) and B4 (Red), with a ratio of 0.45. We found out that better results were obtained with B1 (Blue) and B4 (Red), still with a ratio of 0.45. More accurate studies tend to recommend a higher value, closer to 0.5

 

As can be seen in the figure below, the estimates obtained with V1_3 are not biased anymore, and have a reduced standard deviation. As a consequence of getting a lower aerosol optical thickness, the surface reflectances of V1_3 products are 2% higher than those of the earlier versions (some user feedbacks from V1_0 said reflectances were sometimes too low).

 

Using B2/B4 ratio

Using B1/B4 ratio

 

V1_4 will be provided with a better shadows mask, as it has been found that the shadows masks are too severe and contain too many commission errors. It is in its final stages of validation.

 

 

5000+ L2A Sentinel-2A products released on Theia server

=> 

:) :) :) After having repeated month after month " sorry, our L2A product production is late again for  this ou that reason" in this blog, on the phone, during meetings, via email or on social networks, you can't imagine my pleasure to announce that CNES just released large time series of Sentinel-2A L2A data, over France, Reunion Island, North of Spain, Belgium, Luxembourg, and on some large sites in Burkina Faso, Mali, Ethiopia, and Tunisia. We process one whole year of data from December 1st 2015 to December 1st 2016. More than 5000 tiles where just released.  Processing over Tunisia goes on and Morocco and Senegal are next on the list. Then, we will process Madagascar, and the  all the zones displayed here. The data were processed using the MACCS processor.

Data are available here :

https://theia.cnes.fr/

Please have a look at the help page to access the product format documentation and install the download tool.

 


Continue reading

MUSCATE : Nouvelles, News... y Noticias

On dirait que mes annonces concernant Muscate ne portent pas chance à ce système. La semaine dernière, alors que j'annonçais une production régulière, les performances d'accès à la baie de disques du nouveau cluster se sont nettement dégradées (nous utilisons encore l'ancien cluster, mais nos données sont sur la baie du nouveau, avec un accès par NFS (Network File System) qui a tendance à ralentir les performances). Après un arrêt d'une journée, les ingénieurs du centre informatique ont pu faire des tests et quelques réglages qui ont nettement amélioré les performances, permettant à Muscate de repartir sur de meilleures bases.

 

En conséquence, la production est maintenant plus soutenue (jusqu'à 160 produits par jour), aucune nouvelle anomalie sur Muscate n'a été détectée lors des dernières semaines, et la production des données sur la France jusqu'à début décembre est terminée. Comme vous pouvez le voir sur les images jointes, la production réalisée s'étend au Nord de l'Espagne. Les nouvelles données, qui portent le numéro de version 1_1 ont commencé à être distribuées sur l'atelier de distribution https://theia.cnes.fr. Vu le volume de données, celà prendra quelques jours !

 

Enfin, le traitement des zones Burkina, Sénégal, Maroc et Tunisie commence aujourd'hui, et d'ici peu, l'adaptation au nouveau format des tuiles sera validée et nous pourrons nous approcher d'une production en temps réel. En espérant par ce nouvel article ne pas avoir jeté le mauvais oeil à Muscate...

 

It seems that my posts on this blog throw an evil eye to Muscate. Last week, while I was announcing a steady production, the performances of the new cluster considerably degraded. We do not use the new cluster yet (in a few weeks), apart from it disks racks, accessed through a quite slow Network File System. After a one day closure of the system, to allow tests and tuning by the computer center engineers, the production resumed with much improved performances.

 

As a result, Muscate production is much faster (up to 160 products per day), no new anomaly has been detected for weeks now, and the production of data above France until December 2016 just ended. It also extends to the Noth of Spain. The data have started to be pushed to the distribution server, http://theia.cnes.fr. Given the data volume, it will take a few days.
We are starting, today, the production of 4 other zones : Burkina, Senegal, Morocco and Tunisia, and as the adaptation to the new Sentinel-2 data format is nearly qualified, we will soon start real-time production. Hoping this new post is not a bad curse for Muscate...

 

 

La producción de Muscate se extiende al Norte de España, Aquí, una tela al norte de Burgos.

Parece que mis artículos sobre Muscate no traen suerte a este sistema. La semana pasada, cuando anuncié que la producción estaba regular, el acceso al nuevo cluster se degrado considerablemente. Después de una parada para un día, los ingenieros de centros de datos fueron capaces de hacer pruebas y ajustes que mejoraron significativamente el rendimiento, lo que permite Muscate trabajar en mejores condiciones.

 

 

En consecuencia, la producción es ahora mas rápida (hasta 160 productos por día), no se detectó ninguna anomalía nueva en Muscate en las últimas semanas, y se completó la producción de datos para Francia hasta principios de diciembre. Como se puede ver en las imágenes adjuntas, la producción obtenida se extiende al norte de España. Los nuevos datos, que llevan el número de versión 1_1 comenzaron a distribuirse en el taller de distribución https://theia.cnes.fr. Dado el volumen de datos, se tardará unos pocos días!

 

Por último, el tratamiento de las zonas de Burkina Faso, Senegal, Marruecos y Túnez comienza hoy, y en poco tiempo, se validará la adaptación al nuevo formato de la baldosa. Por fin podremos nos acercar a una producción en tiempo real.

Nouvelles de MUSCATE / MUSCATE news

Depuis que le nouveau cluster du CNES a été installé, le centre de production MUSCATE a fonctionné de manière régulière, avec un rythme de production de 100 tuiles Sentinel-2 traitées au Niveau 2 par jour. Nous sommes encore loin de la pleine vitesse :, il faudra d'une part encore quelques mises à jours pour que le système soit plus robuste, et d'autre part que le nouveau centre de calcul du CNES termine sa phase de rodage car il tourne tourne encore assez lentement. Cependant, la production des données sur la France, la Belgique, les Pyrénées, le bassin de l'Ebre et la Catalogne en Espagne devrait se terminer ce week-end et les données seront ensuite poussées verts le serveur de distribution dans le courant de la semaine prochaine. Les nouvelles données porteront la version V1_1.

 

Nous démarrerons par la suite la production des données de trois nouvelles zones, au Maroc, en Tunisie et au Burkina Faso. Quant au traitement en temps réel, il nous faut attendre la validation de la modification qui permet de gérer le nouveau format de données Sentinel-2, mis en place le 4 décembre. C'est pour très bientôt !

 

Since the new CNES cluster has been installed, the MUSCATE production rate has been steady, around 100 products per day. This is not full speed, as MUSCATE still needs a few updates to be more robust and can't still be used at full capacity, and as the CNES cluster is in its tuning phase and is often running quite slow. However, the production of all data above France, Belgium, and North of Spain (Pyrenees, Ebro basin, Catalonia) should be completed this week-end and pushed to the distribution server next week. The new products will have version number V1_1. This version brings mainly a few bug corrections, regarding the cloud shadows for instance.

 

Then we will proceed with 3 other zones ; Burkina, Morocco and Tunisia. To start real time production, we are waiting for the end of validation of the adaptation to the new format of Sentinel-2 data, which started to be used of the 4th of December 2017. It's nearly ready !

New version of PEPS (French Sentinel mirror site)

As you probably know, PEPS is the French Collaborative ground segment for Copernicus Sentinel program. And, first of all, it is a mirror site that distributes all the Sentinel data in near real time. These last weeks, real time was not available for Sentinel-2, as the data format and structure of Sentinel-2 products had deeply changed, and the software needed adaptation. PEPS team created a new collection, named "Sentinel-2 Single Tiles", coded "S2ST" to separate the old format from the new one. Now that the new version has been installed and validated, the PEPS mirror site is once again up to date.

Continue reading

Festival time with sentinel-2

Burning Man, black rock desert, USA.
Area : about 5 km2
Official attendance in 2016:: about 70000 people per day during 8 days.
1 people each 72 m2

Conclusion about my search of the event on Sentinel-playground: EASY! The event is iin the middle of the desert, high contrasts, long preparation time, no cloud, .

Burning Man

Festival des Vieilles Charrues, Carhaix, France..
Area: about 8 hectares for the music festival and 20 hectares for the camping area
Official attendance in 2016: about 70000 people / day during four days.
Density: 1 people each 4 m2

Conclusion about my search of the event on Sentinel-playground: LUCKY! I searched for other big european music festivals and found only this one. Most of the dates appeared to be cloudy or without image during the festival..This one was perfectly clear.

Conclusion: I still can't hear the music

Sentinel-2A unveils two new lakes in Borneo

The remote sensing company Planet has published images, which were captured by its fleet of nano-satellites, of Lake Bakun on the island of Borneo [1]. Planet tells us that this artificial lake is the largest lake in Southeast Asia since its flooding in 2011. Yet it is not visible on mapping web services like Bing Maps or Google Maps! It does show in OpenStreetMap at low resolution. This is due to the fact that the island of Borneo is one of the most cloudy places in the world so the usable optical remote sensing images are very rare.
Continue reading

Release of Theia Sentinel-2 L2A data over la Réunion

Because of the little issues we stumbled upon when producing Sentinel-2 level 2A data over France with MUSCATE, we had not yet demonstrated an end-to-end production, with a distribution on the operational server. While waiting for the end of the production of data over metropolitan France, we did this test the processing of Réunion Island.

 

September 2016 eruption witnessed by Sentinel-2A

The island is covered by only one Sentinel-2 tile, but the processing worked very smoothly, which is a very good sign after all teh difficulties we met. The data are now released on :

https://theia.cnes.fr

 

We processed all the available data, strating from January the 18th, 2016, until end of November. To produce the data acquired later on, we are waiting for the new version of PEPS, compatible with the new Sentinel-2 data format. We hope it will be possible as soon as February. .
The data set includes a nice image of the eruption of the Piton de la Fournaise volcano, in September. Regarding the data quality, you may find that the cloud shadows are a little exaggerated. We will try to fix that with a better tuning of the threshold for the next versions.