Speed-up downloads from PEPS S2 mirror site with peps_download.py

The French Sentinel mirror site, PEPS, has a very clever data management facility. All the products are stored on tapes, with a capacity of several PB, and there is some sort of cache made of disks. The products accessed recently are on disks, while the other products stay on tapes. The storage costs and also power consumption are therefore largely optimized.

 

The drawback is that before accessing a file on tape, some time is needed to get the tape, and read the file on tapes. This can take something like 2 to 10 minutes. My little tool, peps_download.py was designed when most of the products were on disks, and it was quite slow to download products on tapes. As I am not a patient person, I have tried to speed it up, and it works well, thanks to good advise from CNES peps  colleagues (Christophe Taillan and Erwann Poupart).

 

The previous version was working like that :

Make catalog request

For all product in the request result :

- while product is not downloaded

 - try to download the product

 - if still on tape, wait for 2 minutes

As a result, for each product on tape, it was necessary to wait for 2 to 10 minutes.

 
Now, it works like that

Make catalog request

For all products on tape in the request result

- ask to read it on disks

While (still some products to download):

- Redo catalog request

- Download products on disk

- If some products are not on disk yet

 - wait for 2 minutes

 
On my computer, it used to take more that 12 hours to download 2 years of Sentinel-2 data for a given tile. It has now been reduced to less that 3 hours (but my computer is on CNES network). I hope you will have similar results !

Using NDVI with atmospherically corrected data

NDVI

 

NDVI is by far the most commonly used vegetation index. NDVI was developed in the early seventies (Rouse 1973, Tucker 1979), and widely used with remote sensing in the nineties until now. It is computed from the surface reflectance in the red and near infra-red channels on each side of the red-edge.

 

 NDVI=\frac{\rho(NIR)-\rho(RED)}{\rho(NIR)+\rho(RED)}

 

where \rho(NIR) and  \rho(RED) are reflectances in the NIR and RED. Although several users still use top-of atmosphere reflectances (TOA), surface reflectances should be used to reduce sensitivity to variations of aerosol atmospheric content.

 

A time profile of surface reflectance from Sentinel2 satellite for the blue, green, red and NIR spectral bands for a summer crop in South East France. The observation under constant viewing angles minimizes directional effects.One can also notice that reflectance variations related to vegetation status are greater in the near infra-red, while the noise is usually lower. As a result, a vegetation index should rely more on the NIR than on the red.

I think NDVI is mainly used for the following reasons (but feel free to comment and add your reasons) :

  • it has the large advantage of qualifying the vegetation status with only one dimension, instead of N dimensions if we consider the reflectances of each channel. Of course, by replacing N dimensions by only one, a lot of information is lost.
  • it  enables to reduce the temporal noise due to directional effects. But with the Landsat, Sentinel-2 or Venµs satellites, which observe under constant viewing angles, the directional effects have been considerably reduced.

 

I therefore tend to tell students that if NDVI is convenient, it is not the only way to monitor vegetation.

Continue reading

Canigou 3D

Lo Canigó és una magnòlia immensa
que en un rebrot del Pirineu se bada
- Jacint Verdaguer i Santaló

 

The Canigó is an immense magnolia
that blooms in an offshoot of the Pyrenees

 

3D view of the Canigou on 19-Dec-2017 (with a fancy tiltshift effect)

Continue reading

New version of PEPS (French Sentinel mirror site)

As you probably know, PEPS is the French Collaborative ground segment for Copernicus Sentinel program. And, first of all, it is a mirror site that distributes all the Sentinel data in near real time. These last weeks, real time was not available for Sentinel-2, as the data format and structure of Sentinel-2 products had deeply changed, and the software needed adaptation. PEPS team created a new collection, named "Sentinel-2 Single Tiles", coded "S2ST" to separate the old format from the new one. Now that the new version has been installed and validated, the PEPS mirror site is once again up to date.

Continue reading

(Enfin !) Téléchargement par script des produits de niveau 2A de Sentinel-2 de Theia

=>

La production des données de niveau 2A de Sentinel-2 se poursuit au CNES, mais un peu moins vite que prévu pour le moment.  Nous avons connu un jour faste pendant lequel 600 tuiles ont été produites, mais le rythme de production a souvent été plus lent : nous avons  résolu progressivement de petites anomalies et en même temps, le centre informatique du CNES sur lequel nous nous appuyons a connu de petits soucis.

 

Pendant ce temps, mes collègues de l'équipe MUSCATE, et notamment Dominique Clesse et Remi Mourembles de CAP GEMINI, on ajouté au site de distribution la possibilité de télécharger les données par script, sans un clic.

 

Le script est très facile à utiliser, par exemple, la ligne ci-dessous télécharge les données Sentinel-2 de la tuile 31TCJ (Toulouse), acquises en Septembre 2016 :

python ./theia_download.py -t 'T31TCJ' -c SENTINEL2 -a config_theia.cfg -d 2016-09-01 -f 2016-10-01

Continue reading

(At last !) Automated download of Sentinel-2A Level 2A products from Theia

=>

The production of Sentinel-2 L2A data is on-going at CNES THEIA, but it is still a little slower than expected. We had one fast day on which the exploitation team managed to process 600 tiles, but the production has often been slower as we needed to solve a few glitches, and as the whole CNES processing center had also its own issues.

 

Meanwhile, my colleagues at CNES MUSCATE Center, with the precious help of Dominique Clesse (CAP GEMINI) and Remi Mourembles (CAP GEMINI), have implemented the possibility to download the images via a script and no click. By the way, the shop cart, which did not work when we ordered more than 10 products has also been repaired.

 

The script is very easy to use, for instance, the following line downloads the SENTINEL-2 products above tile T31TCJ (Toulouse), acquired in September 2016 :

python ./theia_download.py -t 'T31TCJ' -c SENTINEL2 -a config_theia.cfg -d 2016-09-01 -f 2016-10-01

Continue reading

The iota2 Land cover processor has processed some Sentinel-2 data

=> 

You already heard about iota2 processor, and you must know that it can process LANDSAT 8 time series et deliver land cover maps for whole countries. These las days, Arthur Vincent completed the code that allows processing Sentinel-2 time series. Even if atmospherically corrected Sentinel-2 data are not yet available above the whole France, we used  the demonstration products delivered by Theia to test our processor.

 

Everything seems to work fine, and the 10 m resolution of Sentinel-2 seems to allow seeing much more details. The joined images show two extracts near Avignon, in Provence, which show the differences between Landsat 8 and Sentinel-2. Please just look only at the detail level, and not at the differences in terms of classes. Both maps were produces using different time periods, and a period limited to winter and beginning of spring for Sentinel-2, and the learning database is also different. Please don,'t draw conclusions too fast about the thematic quality of the maps.

 

First extract shows a natural vegetation zone, with some farmland (top LANDSAT8, bottom Sentinel-2)

coudoux.png

Continue reading

On Google Earth Engine, beware of the Mrs-Armitage-on-Wheels Syndrom

=> 

A few colleagues replied to our campaign to explain some of the dangers of Google Earth Engine. They said :

"well, after all you are probably right, but don't worry, we only use it to do quick and dirty stuff, not real scientific work"

 

As most (...) of these colleagues are quite sensible, I am not worrying too much. But as far as I am concerned, I would have some chances to be a victim of Mrs-Armitage-on-wheels Syndrom (AWS). I guess I do not need to explain it to our british colleagues who consult this blog, this syndrom originates form the great children book from Quentin Blake, that I used to read to my children, some time ago (every night for the two first weeks, then once in a while...) : Mrs Armitage on wheels. Another daddy reads it for you here.

Continue reading

Sur Google Earth Engine, attention au syndrome de Mrs-Armitage-on-wheels

=>

Quelques collègues ont répondu à nos avertissements sur Google Earth Engine, en nous disant :

Oui, vous avez probablement raison, mais ne vous inquiétez pas, nous n'y faisons que des petits tests vite faits, mal faits, pas du vrai travail scientifique

Ces collègues étant pour la plupart (...) des scientifiques raisonnables, je ne m'inquiète pas trop.  Mais pour ma part, me connaissant, j'aurais un peu peur d'être victime du syndrome de Mrs-Armitage-on-wheels. Le nom de ce syndrome vient d'un livre génial de Quentin Blake que je lisais à mes enfants, en VO (tous les soirs pendant 15 jours...): Mrs Armitage on wheels.  Un autre papa vous le lit ici toujours en VO.

(en Français, le titre a été joliment traduit par "Ameline Fourchedrue")

 

Continue reading

Mapping flooded areas using Sentinel-1 in Google Earth Engine

Exceptional rainfall in May caused heavy flooding in the Paris region. Newspapers and TVs reported that the Seine flood forced the Louvre staff to move away from rising waters the art pieces that were stored in their cellar. But they did not tell you that about 50 km east of the Louvre museum, the flood of the Grand Morin river in Coulommiers also inundated the cellar of my parents-in-law. I'm really concerned about this cellar because I care about my parents-in-law of course, and also because I have let some of my bottles of wine in their cellar. Continue reading