THEIA/MUSCATE nears real time for Sentinel-2 L2A

THEIA MUSCATE production of Sentinel-2 L2A products nears real time, and L2A products acquired until March 2017 over France and Reunion Island have started to appear on our distribution server :

https://theia.cnes.fr

The production will progressively be extended over the whole France and over the other sites (Spain, Morocco, Belgium Tunisia, Senegal, Burkina, Mali...) and then follow the real time acquisitions with the shortest delay possible.

 

La production par MUSCATE des données de niveau 2A de Sentinel-2 s'approche du temps réel, et les données acquises jusqu'à mars 2017 sur la France et l’île de la Réunion ont commencé à apparaître sur le serveur de distribution.

https://theia.cnes.fr

 

La production va être progressivement étendue à toute la France et aux autres sites (Espagne, Maroc, Belgique Tunisie, Sénégal, Burkina Faso, Mali...), et suivra ensuite les acquisitions en temps réel avec le plus court délai possible


Quantitative comparison of cloud masks from MACCS/MAJA, Sen2Cor and GEOSYS (hand made)

=> 

As already explained in a previous post, we obtained some Sentinel-2 hand made cloud masks from GEOSYS company. We used those to validate the cloud masks from MACCS/MAJA. But we wanted to use them further to make a quantitative comparison with Sen2Cor cloud masks.
But this comparison required solving a little issue : GEOSYS cloud masks are generously dilated to avoid any risk to let a cloud pass through the operational processing. Those of MACCS:MAJA are also dilated while those of Sentinel-2 are not at all. In the following paragraphs, we'll explain how we solved that issue. Sen2cor (v2.3.0) has also three levels of cloud mask (High Medium and Low probability). We used here the Medium Probability mask. But let's start with the final result comparing the performances of Sen2Cor and MACCS:MAJA.

 

Overall accuracies for MACCS/MAJA, in red and Sen2cor, in blue for 11 images compared to GEOSYS cloud masks.

Continue reading

Quantitative validation of MACCS cloud masks

=> 

At the annual review  of PEPS project (CNES Sentinel global mirror site), the GEOSYS company showed its operational activities centered on the use of Sentinel-2 data to provide advise on agriculture on many regions of the world. On this occasion, GEOSYS showed their cloud detection process for Sentinel-2 images. The Sen2cor solution was not considered reliable enough y GEOSYS, and the regions processed by MACCS within MUSCATE are far from covering all the regions of interest of the company. GEOSYS decided to rely on human operators to improve the cloud masking. For each processed Sentinel-2 image, a man made valid pixel mask is build ("valid" means without clouds and cloud shadows).

 

Continue reading

Theia releases Sentinel-2A L2A data over Morocco

=> 

After France, Belgium, North of Spain, Tunisia, a transect from Burkina Faso to Mali and some tiles in Ethiopia, Theia just released the production of one year of Sentinel-2A data at Level 2A above a large region of Morocco, from Ouarzazate to Casablanca. Level 2A products provide surface reflectances corrected from atmospheric effects, with a high quality cloud mask.

 

The L2A data are, as always, available from https://theia.cnes.fr. Please refer to the help page to find the data format description. To download the data without a click, you may use our python download tool (after having registered on Theia's site).

Continue reading

MUSCATE S2 product versions

V1_0 was a preliminary processing performed with early version of MACCS adapted to Sentinel-2. It was plagued with a bug in the cloud shadow mask : when more than 255 clouds were present in one image, shadow detection went completely wrong and shadows were detected anywhere. V1_0 was not produced in MUSCATE operationnal context, but in an earlier validation context. Production was available over France only.

 

V1_1 replaces V1_0 over France, Production started mid November, but was only released in February because of many difficulties encountered by MUSCATE. It corrects for the bug related to cloud shadows observed with V1_0, and  was fully processed by the operational MUSCATE center.

 

V1_2 was used in another context not related to MUSCATE, you will not find any product with this version number within MUSCATE server.

 

V1_3 was applied starting from February 2017, to data sets produced above Reunion, Burkina, Senegal, Tunisia, Morocco.The aerosol estimate is improved compared to V1_1. Together with Bastien Rouquié, from CESBIO, we worked on the tuning of the blue-red ratio which is used in the multi-spectral method to estimate aerosols (which is combined with a multi-temporal method). Initially, we used bands B2 (Blue) and B4 (Red), with a ratio of 0.45. We found out that better results were obtained with B1 (Blue) and B4 (Red), still with a ratio of 0.45. More accurate studies tend to recommend a higher value, closer to 0.5

 

As can be seen in the figure below, the estimates obtained with V1_3 are not biased anymore, and have a reduced standard deviation. As a consequence of getting a lower aerosol optical thickness, the surface reflectances of V1_3 products are 2% higher than those of the earlier versions (some user feedbacks from V1_0 said reflectances were sometimes too low).

 

Using B2/B4 ratio

Using B1/B4 ratio

 

V1_4 will be provided with a better shadows mask, as it has been found that the shadows masks are too severe and contain too many commission errors. It is in its final stages of validation.

 

 

5000+ L2A Sentinel-2A products released on Theia server

=> 

:) :) :) After having repeated month after month " sorry, our L2A product production is late again for  this ou that reason" in this blog, on the phone, during meetings, via email or on social networks, you can't imagine my pleasure to announce that CNES just released large time series of Sentinel-2A L2A data, over France, Reunion Island, North of Spain, Belgium, Luxembourg, and on some large sites in Burkina Faso, Mali, Ethiopia, and Tunisia. We process one whole year of data from December 1st 2015 to December 1st 2016. More than 5000 tiles where just released.  Processing over Tunisia goes on and Morocco and Senegal are next on the list. Then, we will process Madagascar, and the  all the zones displayed here. The data were processed using the MACCS processor.

Data are available here :

https://theia.cnes.fr/

Please have a look at the help page to access the product format documentation and install the download tool.

 


Continue reading

MUSCATE : Nouvelles, News... y Noticias

On dirait que mes annonces concernant Muscate ne portent pas chance à ce système. La semaine dernière, alors que j'annonçais une production régulière, les performances d'accès à la baie de disques du nouveau cluster se sont nettement dégradées (nous utilisons encore l'ancien cluster, mais nos données sont sur la baie du nouveau, avec un accès par NFS (Network File System) qui a tendance à ralentir les performances). Après un arrêt d'une journée, les ingénieurs du centre informatique ont pu faire des tests et quelques réglages qui ont nettement amélioré les performances, permettant à Muscate de repartir sur de meilleures bases.

 

En conséquence, la production est maintenant plus soutenue (jusqu'à 160 produits par jour), aucune nouvelle anomalie sur Muscate n'a été détectée lors des dernières semaines, et la production des données sur la France jusqu'à début décembre est terminée. Comme vous pouvez le voir sur les images jointes, la production réalisée s'étend au Nord de l'Espagne. Les nouvelles données, qui portent le numéro de version 1_1 ont commencé à être distribuées sur l'atelier de distribution https://theia.cnes.fr. Vu le volume de données, celà prendra quelques jours !

 

Enfin, le traitement des zones Burkina, Sénégal, Maroc et Tunisie commence aujourd'hui, et d'ici peu, l'adaptation au nouveau format des tuiles sera validée et nous pourrons nous approcher d'une production en temps réel. En espérant par ce nouvel article ne pas avoir jeté le mauvais oeil à Muscate...

 

It seems that my posts on this blog throw an evil eye to Muscate. Last week, while I was announcing a steady production, the performances of the new cluster considerably degraded. We do not use the new cluster yet (in a few weeks), apart from it disks racks, accessed through a quite slow Network File System. After a one day closure of the system, to allow tests and tuning by the computer center engineers, the production resumed with much improved performances.

 

As a result, Muscate production is much faster (up to 160 products per day), no new anomaly has been detected for weeks now, and the production of data above France until December 2016 just ended. It also extends to the Noth of Spain. The data have started to be pushed to the distribution server, http://theia.cnes.fr. Given the data volume, it will take a few days.
We are starting, today, the production of 4 other zones : Burkina, Senegal, Morocco and Tunisia, and as the adaptation to the new Sentinel-2 data format is nearly qualified, we will soon start real-time production. Hoping this new post is not a bad curse for Muscate...

 

 

La producción de Muscate se extiende al Norte de España, Aquí, una tela al norte de Burgos.

Parece que mis artículos sobre Muscate no traen suerte a este sistema. La semana pasada, cuando anuncié que la producción estaba regular, el acceso al nuevo cluster se degrado considerablemente. Después de una parada para un día, los ingenieros de centros de datos fueron capaces de hacer pruebas y ajustes que mejoraron significativamente el rendimiento, lo que permite Muscate trabajar en mejores condiciones.

 

 

En consecuencia, la producción es ahora mas rápida (hasta 160 productos por día), no se detectó ninguna anomalía nueva en Muscate en las últimas semanas, y se completó la producción de datos para Francia hasta principios de diciembre. Como se puede ver en las imágenes adjuntas, la producción obtenida se extiende al norte de España. Los nuevos datos, que llevan el número de versión 1_1 comenzaron a distribuirse en el taller de distribución https://theia.cnes.fr. Dado el volumen de datos, se tardará unos pocos días!

 

Por último, el tratamiento de las zonas de Burkina Faso, Senegal, Marruecos y Túnez comienza hoy, y en poco tiempo, se validará la adaptación al nuevo formato de la baldosa. Por fin podremos nos acercar a una producción en tiempo real.

Nouvelles de MUSCATE / MUSCATE news

Depuis que le nouveau cluster du CNES a été installé, le centre de production MUSCATE a fonctionné de manière régulière, avec un rythme de production de 100 tuiles Sentinel-2 traitées au Niveau 2 par jour. Nous sommes encore loin de la pleine vitesse :, il faudra d'une part encore quelques mises à jours pour que le système soit plus robuste, et d'autre part que le nouveau centre de calcul du CNES termine sa phase de rodage car il tourne tourne encore assez lentement. Cependant, la production des données sur la France, la Belgique, les Pyrénées, le bassin de l'Ebre et la Catalogne en Espagne devrait se terminer ce week-end et les données seront ensuite poussées verts le serveur de distribution dans le courant de la semaine prochaine. Les nouvelles données porteront la version V1_1.

 

Nous démarrerons par la suite la production des données de trois nouvelles zones, au Maroc, en Tunisie et au Burkina Faso. Quant au traitement en temps réel, il nous faut attendre la validation de la modification qui permet de gérer le nouveau format de données Sentinel-2, mis en place le 4 décembre. C'est pour très bientôt !

 

Since the new CNES cluster has been installed, the MUSCATE production rate has been steady, around 100 products per day. This is not full speed, as MUSCATE still needs a few updates to be more robust and can't still be used at full capacity, and as the CNES cluster is in its tuning phase and is often running quite slow. However, the production of all data above France, Belgium, and North of Spain (Pyrenees, Ebro basin, Catalonia) should be completed this week-end and pushed to the distribution server next week. The new products will have version number V1_1. This version brings mainly a few bug corrections, regarding the cloud shadows for instance.

 

Then we will proceed with 3 other zones ; Burkina, Morocco and Tunisia. To start real time production, we are waiting for the end of validation of the adaptation to the new format of Sentinel-2 data, which started to be used of the 4th of December 2017. It's nearly ready !