Speed-up downloads from PEPS S2 mirror site with peps_download.py

The French Sentinel mirror site, PEPS, has a very clever data management facility. All the products are stored on tapes, with a capacity of several PB, and there is some sort of cache made of disks. The products accessed recently are on disks, while the other products stay on tapes. The storage costs and also power consumption are therefore largely optimized.

 

The drawback is that before accessing a file on tape, some time is needed to get the tape, and read the file on tapes. This can take something like 2 to 10 minutes. My little tool, peps_download.py was designed when most of the products were on disks, and it was quite slow to download products on tapes. As I am not a patient person, I have tried to speed it up, and it works well, thanks to good advise from CNES peps  colleagues (Christophe Taillan and Erwann Poupart).

 

The previous version was working like that :

Make catalog request

For all product in the request result :

- while product is not downloaded

 - try to download the product

 - if still on tape, wait for 2 minutes

As a result, for each product on tape, it was necessary to wait for 2 to 10 minutes.

 
Now, it works like that

Make catalog request

For all products on tape in the request result

- ask to read it on disks

While (still some products to download):

- Redo catalog request

- Download products on disk

- If some products are not on disk yet

 - wait for 2 minutes

 
On my computer, it used to take more that 12 hours to download 2 years of Sentinel-2 data for a given tile. It has now been reduced to less that 3 hours (but my computer is on CNES network). I hope you will have similar results !

Yesterday's snow cover area in the Pyrenees

Olivier pointed to me that ESA's ground segment, PEPS and MUSCATE were all in really good shape today... And the sky was clear yesterday at the time of the Sentinel-2A acquisition!

So I could download the Level-2A product from theia.cnes.fr, run our let-it-snow processor, start QGIS and here it is: the map of yesterday's snow cover area at 20 m resolution. If you know the region, you might notice that there is currently a big contrast in the snow cover extent between the French and the Spanish Pyrenees. This is due to the blocking of the moist air masses coming from the north.

Snow cover area on 22 Nov 2017. blue: snow, grey: no snow, white: cloud.

Stay tuned! Theia should start to distribute these Sentinel-2 snow products in near real time very soon.

[Muscate news] temporary slowlyness

Update : MUSCATE is back on track for the real time production, thanks to PEPS return to nominal production, and a better stability of our platform.

Some of you will have probably noticed that Theia production of Sentinel-2 L2A data is quite slow these days, and is not producing all the data it should in real time. The reason is we started to implement a correction to solve the performance issues we had. This correction was tested and qualified during several days, in operational conditions,  but when put in production, turned out to be unstable, due to occasional slowness in CNES cluster which did not happen when the correction was in tests. Moreover, our source of Sentinel2 data, PEPS, was experiencing some difficulties, providing a much reduced number of images.

 

We are waiting for a correction of the correction (the cause has been found and corrected but needs to be tested), The new version should be installed in less than 2.3 days, as Dilbert (dilbert.com) says. And all the team is very sorry for the delays and the inconvenience it is causing.

New version of PEPS (French Sentinel mirror site)

As you probably know, PEPS is the French Collaborative ground segment for Copernicus Sentinel program. And, first of all, it is a mirror site that distributes all the Sentinel data in near real time. These last weeks, real time was not available for Sentinel-2, as the data format and structure of Sentinel-2 products had deeply changed, and the software needed adaptation. PEPS team created a new collection, named "Sentinel-2 Single Tiles", coded "S2ST" to separate the old format from the new one. Now that the new version has been installed and validated, the PEPS mirror site is once again up to date.

Continue reading