Did you know that the St. Moritz Casino is the highest in Switzerland? If you like gambling, I have a little game for you: what are the odds to find snow near St. Moritz?
Fortunately, I just finished the processing of 218 Sentinel-2 dates from 2015-Dec-04 to 2018-Apr-10 of tile 32TNS with our let-it-snow processor. I did this off-line production for a colleague because, as of today, Theia only distributes the snow products after July 2017 in this region of Switzerland (see the available products here).
A quick way to check the output is to compute a snow cover probability map: that is, for each pixel, the number of times that snow was observed divided by the number of times that the snow could be observed.
To compute this map we just need to know that the Theia snow products (LIS_SEB.TIF raster files) are coded as follows:
205: Cloud including cloud shadow
254: No data
Here is a piece of script to do this:
# initialize snow.tif with zeros
# store in Byte because we have less than 255 dates
f0=$(find . -name LIS_SEB.TIF | head -1)
gdal_calc.py --overwrite -A $f0 --type=Byte --calc=A*0 --outfile=snow.tif
# accumulate snow pixels in snow.tif
for f in $(find . -name LIS_SEB.TIF)
# snow is coded with 100
gdal_calc.py --overwrite -A $f -B snow.tif --type=Byte --calc="B+(A==100)" --outfile=snow.tif
# now do the same for clear.tif
gdal_calc.py --overwrite -A $f0 --type=Byte --calc=A*0 --outfile=clear.tif
# accumulate clear pixels in clear.tif
for f in $(find . -name LIS_SEB.TIF)
# only snow and no snow are coded with values lower than 101
gdal_calc.py --overwrite -A $f -B clear.tif --type=Byte --calc="B+(A<101)" --outfile=clear.tif
# Finally compute the snow probability in % (100.0* makes the calculation in float)
gdal_calc.py -A snow.tif -B clear.tif --type=Byte --calc="(100.0*A)/B" --outfile=snowProba.tif
This is the output:
The images are scaled from 0 (black) to 100 (white). The units are number of days for snow and clear, percentage for snowProba.
From which you can map the odds to find snow near St. Moritz (click on the image to animate)!
MUSCATE sentinel-2 L2A counter just reached 80000 products, but a lot more products were processed these past weeks. Indeed, the MUSCATE team started processing the early Sentinel-2B images acquired from July to October 2017, which were missing on our catalog. Indeed. We only started Level 2A treatments in October 2017, in real time (a bit late, but before the ESA;)). We take advantage of this processing to reprocess Sentinel-2A data, since the quality of the products from MAJA improves with the repetitiveness of the observations. And since we replace old versions with new ones (v 1.7), the product counter is not affected by the reprocessing of S2A data.
The reprocessing of data on France and its overseas territories is finished, and the treatment of the other European zones is in progress. We will continue soon with the African zones then those of the rest of the world. If you are using the Sentinel-2 data acquired in the second half of 2017, we encourage you to download them again. We took advantage of this reprocessing to also produce snow products in the France area.
Le compteur de produits de Niveau 2A de Sentinel-2 de MUSCATE vient de franchir les 80000 produits, mais en fait, MUSCATE en a produit beaucoup plus. En effet, l'équipe d'exploitation (Joel, Raquel, Sylvie) a commencé à traiter les données de Sentinel-2B acquises de juillet à octobre 2017, qui manquaient dans notre catalogue. Nous n'avions démarré les traitements de niveau 2A qu'en octobre 2017, en temps réel (avec un peu de retard, mais avant l'ESA ). Nous profitons de ce traitement pour retraiter les données Sentinel-2A, puisque la qualité des produits issus de MAJA s'améliore avec la répétitivité des observations. Et comme nous remplaçons les anciennes versions par les nouvelles (v 1.7), le compteur de produits n'est pas affecté par le retraitement des données S2A.
Le retraitement des données sur la France et ses territoires d'outre-mer est terminé, et le traitement des autres zones Européennes est en cours. Nous poursuivrons prochainement avec les zones Africaines puis celles du reste du monde. Si vous utilisez les données Sentinel-2 acquises au deuxième semestre 2017, nous vous encourageons donc a les télécharger à nouveau. Nous avons profité de ce retraitement pour produire également les produits neige sur la zone France.
Great news for our little Venµs satellite project: Theia has started distributing Venµs L1C data in near real time. The data acquired yesterday are already available on https://theia.cnes.fr
These L1C products are tagged with 0.9 version, which means that their quality is not as good as what we think we will be able to deliver in a couple of months. The multi-spectral and multi-temporal registration can still be improved, even if it is not too far from our expectations yet in most cases. The image quality teams are finishing an error budget of the current situation, and also preparing the improvement with our Israeli partner who manage the satellite. We will tell you about that.
The L2A delivery will also start in a few weeks, but the L1C teams, who had much more work than expected with this satellite, handled us correct products very late, and we still need to tune a few parameters to provide good quality products.
On the Theia website, you will find several ways to download the tiles. My little download tool has also been updated. Once you have registered, and updated the config.cfg file, you will be able to download at once all the products in Australia with the following command line :
python theia_download.py -l 'Australia' -c VENUS -a config.cfg -d 2018-01-01
The data format is explained here. Its packaging with useless zips and tars is still provisional and will be simplified very soon.
A first set of 35 images (L1C) is available for download on the following CNES web site:
Venµs is now systematically acquiring data on all selected sites since January 2018, although a few acquisitions were missed from time to time for various technical reasons, which should become progressively less frequent.
The commissioning phase of the Venµs mission is not yet completed regarding image quality (radiometry and geometry). This delay is due to several issues, the most important of which are detailed hereafter. However, since these issues do not prevent the use of the data, CNES decided to process and distribute a first set of 35 Level-1 (L1) images for which the radiometric and geometric qualities are acceptable. Level 1 corresponds to orthorectified Top of the Atmosphere reflectances. This preliminary data set is intended to allow the users to familiarize with the data and their format.
These 35 images are only a subset of the data that are systematically acquired over the Venµs sites since January 2018. We plan to start the distribution of these L1 Venµs time series by April 2018. We expect to start the distribution of the Level-2 products (surface reflectances) by late May or beginning of June, 2018.
- The absolute calibrations of the spectral bands B1 (415 nm) and B2 (440 nm), both located in the blue, are less accurate than the calibration of the other spectral bands. This issue prevents the use of B1 and B2 for retrieving the aerosol optical depth (AOD) or for water color applications.
- The accuracy of the satellite attitude restitution is for now lower than expected. Due to the fact that the different bands are not acquired simultaneously this has mainly impacts on the inter-band registration. For some sites with heavy cloud cover or uniform landscape, this issue also impacts the multi-temporal registration.
The characterization of radiometric and geometric performances is still ongoing. Efforts are also devoted to the improvement of the preprocessing algorithms.
As soon as significant progresses are made, the whole data set acquired since January 2018 will be reprocessed with the new parameters and algorithms.
The product format document is available here. A more complete version will soon be made available there.
The Venµs page on the Theia web site will keep you informed on the progress made.
Time series of Venµs images acquired over Oklahoma (USA) and processed at level 2A (with a first try of atmospheric effects correction applied)
Very recently, Camille Desjardins (from CNES), who is handling the validation of the L2A products generated by THEIA, has set up a systematic validation of the products delivered by MAJA, with the help of an operational service from CNES (OT/PE) (Bruno Besson, and Nicolas Guilleminot from Thales Services, using tools developed by Aurélie Courtois, also from Thales)
Systematically, a comparison of AOT and water vapour is made for every Sentinel-2 L2A product from THEIA which observes one of the sites of the Aeronet network.
Both plots below show the results obtained during the month of February, for the Aerosol Optical Thickness (left), and for the water vapour content (right). Blue dots correspond to validations in ideal conditions (low cloud amount, no gap filling, and quality assured Aeronet data (Level 2.0). The red dots allow degraded conditions, and most of them correspond to the unavailability, yet, of version 2.0 Aeronet data. As data are processed in near real time, and level 2.0 data are made available a few months later, these plots rely mainly on Level 1.5 data, which are more prone to errors (such as a calibration drift... or the presence of a spider in the instrument tubes).
|Aerosol optical thickness validation of Sentinel-2 L2A for all Aeronet match-ups gathered in February 2018
||Water vapour validation of Sentinel-2 L2A for all Aeronet match-ups gathered in February 2018 (in g/cm2)
MUSCATE is in a good shape these days thznkd to the continuous efforts of the development team (CNES and CAP GEMINI) who solved several issues. The counter of Sentinel-2 Level 2A products reached 70 000 products this night, just one month after reaching 60000. If we sum all the products delivered by MUSCATE, we are reaching 99 000 images. MUSCATE also distributes Sentinel-2 Snow masks over mountains, and Spot World Heritage data (old SPOT data reprocessed after ortho-rectification.and made available for free).
This good shape allows us to increase our production rhythm. We have started processing the Sentinel-2B data acquired between July and October 2017, as we had started processing S2B in November 2017 only. But since MAJA is a multi-temporal processor, we are in fact starting a complete reprocessing of the data, including S2A and S2B. The quality of S2A products should therefore also benefit from the improved repetitivity of observations.
This reprocessing will last several weeks. We are starting with data from France and will go on with our neighbouring European countries, then sites in Maghreb, the remaining sites of Africa, and finally, the rest of the sites in the world.
In case you have an urgent need for some tiles, please ask ! (of course, it is only applicable to the tiles already in our list)
In this post, a validation of the land-cover map of France produced by CESBIO for the 2016 period was presented. This validation used independent data (that is data collected by different teams and using different procedures than the data used for the classifier training), but the validation procedure consisted in applying classical machine learning metrics which, as described in this other post, have some limitations.
A fully independent validation following a sound protocol is costly and needs skills and expertise that are very specific. SIRS is a company which is specialised in the production of geographic data from satellite or aerial images. Among other things, they are the producers of Corine Land Cover for France and they are also responsible for quality control and validation of other Copernicus Land products.
SIRS has recently performed a validation of the 2016 France land-cover map. The executive summary of the report reads as follows:
This report provides the evaluation results of the CESBIO OSO 2016 10m layer and the CESBIO OSO 2016 20m layer.
The thematic accuracy assessment was conducted in a two-stage process:
- An initial blind interpretation in which the validation team did not have knowledge of the product’s thematic classes.
- A plausibility analysis was performed on all sample units in disagreement with the production data to consider the following cases:
- Uncertain code, both producer and operator codes are plausible. Final validation code used is producer code.
- Error from first validation interpretation. Final validation used is producer code
- Error from producer. Final validation code used is from first validation interpretation
- Producer and operator are both wrong. Final Validation code used is a new code from this second interpretation.
Resulting to this two-stage approach, it should be noticed that the plausibility analysis exhibit better results than the blind analysis.
The thematic accuracy assessment was carried out over 1,428 sample units covering France and Corsica.
The final results show that the CESBIO OSO product meet the usually accepted thematic validation requirement, i.e. 85 % in both blind interpretation and plausibility analysis. Indeed, the overall accuracies obtained are 81.4 +/- 3.68% for the blind analysis and 91.7 +/- 1.25% for the plausibility analysis on the CESBIO OSO 10m layer. The analysis on the 20m layer shows us that the overall accuracy for the blind approach is 81.1 +/-3.65% and 88.2 +/-3.15% for the plausibility approach.
Quality checks of the validation points have been made by French experts. It should be noticed that for the blind analysis, the methodology of control was based mostly on Google Earth imagery, no additional thematic source of information that could provide further context was used such as forest stand maps, peatland maps, etc.
These results are very good news for us and for our users. The report also contains interesting recommendations that will help us to improve our algorithms. The full report is available for download.
Les tuiles en bleu viennent d'être ajoutées aux zones où Theia fournit des produits de niveau 2A.
Depuis la découverte des récents bugs et la mise en place de moyens de les contourner, le rythme de production de MUSCATE s'est accru, et nous avons donc pu étendre un peu les zones où nous fournissons des données Sentinel-2 (A&B) corrigées des effets atmosphériques, avec un bon masque de nuages: il s'agit de produits de niveau 2A fournis par la chaîne MAJA.
Nous venons de mettre en ligne toutes les données acquises sur la zone côtière du Maghreb, du Maroc à la frontière entre Algérie et Tunisie. Quelques tuiles manquantes ont également été rajoutées au sud du Maroc, et sur le Cap Bon, en Tunisie. Avec cet ajout de 50 tuiles, toutes les terres qui bordent la méditerranée occidentale sont donc suivies pat les produits de Theia (enfin, il manque les Baléares).
The blue tiles were just added to the zones where Theia provides level 2A products with MAJA.
Since we understood the bugs which were slowing MUSCATE, and we found ways to mitigate them, the production rhythm of MUSCATE improved and we were able to extend the zones where we provide Sentinel-2 (A&B) L2A products. L2A products provide surface reflectances after correction of atmospheric effects and with a high quality cloud mask. The products we deliver are provided by MAJA processor.
We just released all the data acquired on the Maghreb coastal zone, from Morocco to Tunisia. A few missing tiles have also been added in South Morocco, and on Cap Bon in Tunisia. With these new tiles, we now monitor all the lands that surround the occidental part of Mediterranean sea, adding 50 tiles to those already processed.