How to automatically download Sentinel data from PEPS collaborative ground segment

=>

peps_download

This is a simple piece of code to automatically download the products provided by the French Sentinel collaborative ground segment named PEPS : https://peps.cnes.fr. PEPS is mirroring all the Sentinel data provided by ESA, and is providing a simplified access.

This code was written thanks to the precious help of one my colleagues at CNES Jérôme Gasperi who developed the "rocket" interface which is used by Peps.

This code relies on python 2.7 and on the curl utility. Because of that, I guess it only works with linux.

The tool is available on my github repository : https://github.com/olivierhagolle/peps_download

Examples

This software is still quite basic, but if you have an account at PEPS, you may download products using command lines like

  • python ./peps_download.py -c S2 -l 'Toulouse' -a peps.txt -d 2015-11-01 -f 2015-12-01 which downloads the Sentinel-2 products above Toulouse, acquired in November 2015.
  • python ./peps_download.py -c S2 --lon 1 --lat 43.5 -a peps.txt -d 2015-11-01 -f 2015-12-01 which downloads the Sentinel-2 products above --lon 1 --lat 43.5 (~Toulouse), acquired in November 2015.
  • python ./peps_download.py -c S1 --lonmin 1 --lonmax 2 --latmin 43 --latmax 44 -a peps.txt -d 2015-11-01 -f 2015-12-01 which downloads the Sentinel-1 products in latitude, longitude box around Toulouse, acquired in November 2015.

Authentification

The file peps.txt must contain your email address and your password on the same line, such as follows :your.email@address.fr top_secret

To get an account : https://peps.cnes.fr/rocket/#/register

 

Posted under: Code, In English, Sentinel-2

Tagged as: ,

33 comments

  • Christina on 10/02/2016 at 20:16 said:

    I am fairly unfamiliar with Python and am having issues getting this code and the Sentinel_download.py code to work. I keep getting syntax errors for the input line (i.e. the decimal degrees for the longitude and latitude and the file path). I have added back slashes and apostrophes in the file path to get it to recognize it as a string, but there continue to be errors. Is there any other formatting that needs to be done to the line? I guess I assumed I could strictly copy and paste your input line and change to suit my parameters. Merci!

    • Olivier Hagolle on 11/02/2016 at 08:26 said:

      Dear Christina,
      I lack information to be sure, but it seems you are first launching python, then pasting the command lines, Which is wrong.
      You do not need to start python, these command lines are shell command lines.
      Best regards,
      Olivier

  • Denis on 10/03/2016 at 12:15 said:

    Dear Olivier,

    First, I would like to thank you so much for your work and all the information (and even codes !) you share.
    I had two questions about downloading and using Sentinel-2 images :
    1. Is there a way to download specific tiles without having to download the whole scene ?
    2. Sentinel-1 products are available on the Google Earth Engine API, Sentinel-2 are not (maybe not yet). Do you have an idea of the reason for it ?

    Many thanks again,

    Denis

    • Olivier Hagolle on 10/03/2016 at 17:39 said:

      Thanks for your kind words, Denis,

      Yes it is possible to access directly to the tiles, I have tried to add it in my sentinel-2 download module (with the -t option), but it is nor working yet.
      The issue is that you have to download all the files one by one, so it is a little complicated.

      For that, you need to use Url like this : (provided in an other comment by vince)
      https://scihub.copernicus.eu/dhus/odata/v1/Products('141ea4dd-2c99-4ffb-ac07-f5775e08b58f')/Nodes('S2A_OPER_PRD_MSIL1C_PDMC_20160209T011325_R008_V20160208T104841_20160208T104841.SAFE')/Nodes('GRANULE')/Nodes('S2A_OPER_MSI_L1C_TL_SGS__20160208T194222_A003294_T31TCJ_N02.01')/Nodes('IMG_DATA')/Nodes('S2A_OPER_MSI_L1C_TL_SGS__20160208T194222_A003294_T31TCJ_B02.jp2')/$value

      Sol, if I find some time I will do it.

      On your second point, of course Sentinel-2 data will be made available on GEE, and it should start quickly. But you have to be aware that GEE can use your code and applications and does not guarantee the platform will still be available next year.

      And finally, starting this summer, Theia will distribute atmospherically corrected data, by tiles, on selected regions such as Tunisia ;-)

      Best regards,
      Olivier

      • Olivier Hagolle on 13/03/2016 at 20:15 said:

        Bonjour Denis,
        My little download tool : Sentinel-download, now allows to download specific tiles instead of whole S2 products thanks to the -t option. I have finally found some time to finish it. I have not tested it much, though, please tell me if you find some bugs.
        Olivier

        • Denis on 18/03/2016 at 14:32 said:

          Bonjour Olivier,

          I've just tried... and didn't manage to make it. I can't determine whether the bug is in your script or between my keyboard and my screen, so I post the results here ; I might have misunderstood or been too fast on something.

          My command is :
          python Sentinel_download.py --lat 34.75 --lon 10.8 -a apihub.txt -t 32SPF -d 20151225 -f 20151229

          I tried it both with the guest account and with my personal account (created october, 1st 2015). I also tried with a bigger time period.

          I browsed the archive on the scihub interface. The download link given for the whole 20151228 S2 scene on my site is
          https://scihub.copernicus.eu/dhus/odata/v1/Products('83297ffc-7c04-49f4-b80f-4f88ee77e1c3')/$value

          The http requests, give first error codes 401 and then correct code 200. I get the query_results.xml but I don't find file_dir.xml neither do I find granule_dir.xml...
          At the end I'm said "Tile 32SPF is not available within product"..

          Any idea ?

          Many thanks in advance,

          Denis

          • Denis on 18/03/2016 at 14:43 said:

            By the way, with the command :
            python Sentinel_download.py --lat 43.6 --lon 1.44 -a apihub.txt -o 51 -t 31TCJ -d 20160101 -f 20160228

            I manage to get image files (I stopped the download after the first successful download of a .jp2 file).

            I tried to add the row number for my image of interest :
            python Sentinel_download.py --lat 34.75 --lon 10.8 -a apihub\ copy .txt -o 122 -t 32SPF -d 20151225 -f 20151229

            but I get the same result as in the comment above...

          • Olivier Hagolle on 18/03/2016 at 19:31 said:

            Dear Denis,
            I tried the command :

            python Sentinel_download.py --lat 34.75 --lon 10.8 -a apihub.txt -t 32SPF -d 20151225 -f 20151229

            And what I get is this message :
            ========================================
            Tile 32SPF is not available within product
            ========================================
            And in fact, the tile number and the coordinates do not belong to the same place. For the Cap Bon, you are two degrees below. The following command line works for me.

            python Sentinel_download.py --lat 36.75 --lon 10.8 -a apihub.txt -t 32SPF -d 20151225 -f 20151229

            It was easy for me to find, I did the same error in one of my first tests...
            A bientôt,
            Olivier

          • Denis on 22/03/2016 at 18:04 said:

            Perfect !

            How could I have set up this 34.75 ? Now it works, the answer has no interest ! Many thanks again Olivier.

            Denis

  • Denis on 18/03/2016 at 13:43 said:

    Waw !

    Thank you so much Olivier. One comment and you develop a tool.
    I'll test it as soon as possible. I'm very interested in downloading a specific tile... over Tunisia ! ;-)

    Denis

  • Frank on 27/07/2016 at 12:13 said:

    Hallo Olivier,
    is it possible to run that script under Windows? I have installed python 2.7- it came with ArcGIS -, but no skills in using python.
    Is it possible that you could give me a short hint how to start the script? Would be a great help!

    Thank you, greetings
    Frank

    • Olivier Hagolle on 01/08/2016 at 08:24 said:

      Dear Frank, you have several examples above. It is supposed to work on windows, it needs installing aria2 software.
      As I am not a windows user, I am not sure it works as well as on linux. Some users reported issues.
      Best regards,
      Olivier

  • Denis on 20/09/2016 at 08:53 said:

    Dear Olivier,

    I'm back on trying to download tiles on our site, and I was facing a strange behaviour with the same working command as those days :
    python Sentinel_download.py --lat 36.75 --lon 10.8 -a apihub.txt -t 32SPF -d 2015-12-25 -f 2015-12-29
    It seems that the date format has change somwhere because with the following command, it works :
    python Sentinel_download.py --lat 36.75 --lon 10.8 -a apihub.txt -t 32SPF -d 20160316 -f 20160318

    Hope this helps,

    Denis

  • Olivier Hagolle on 20/09/2016 at 09:13 said:

    Thanks Denis, you are right, I figured it out some time ago. I had changed the examples in many places blog, github) and in the description with -h option. But not in the comments. I need to correct it too.
    Olivier

  • Mario on 23/11/2016 at 12:42 said:

    I am using peps but I am getting:

    python peps_download.py -c S2 -a /home/atos/cron/peps.txt -21 --latmax=41.9744 --lonmax=-8.7556
    curl -k -o search.json https://peps.cnes.fr/resto/api/collections/S2/search.json?axRecords=500
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
    Traceback (most recent call last):
    File "peps_download.py", line 130, in
    with open('search.json') as data_file:
    IOError: [Errno 2] No such file or directory: 'search.json'

    What is the problem?
    This used to work...

    • Olivier Hagolle on 25/11/2016 at 14:00 said:

      It seems to work on my side. Maybe Peps was in maintenance ?
      Please try again.
      Olivier

  • Denis on 07/12/2016 at 15:25 said:

    Dear Olivier,

    We have a trouble with the sentinel download script. In our lab, we have installed a script on a server which automatically downloads latest image of our 3 tiles of interest.
    At first the script was launched once a day (now once a week).
    We had a correct behaviour since the end of october.
    For the 32SPF tile, the trouble appeared somewhere after the 24th of october image.
    The behaviour now is that the script creates an empty directory tree (the GRANULE directory contains an empty directory), and runs endlessly.

    Any idea ?

    Many thanks in advance,

    Denis

    • Denis on 13/12/2016 at 16:04 said:

      Thanks to a private communication with Olivier, we went to the following solution : using the peps_download tool to download from the PEPS repository. It's ok for dates after the 24th of october since PEPS has tiles from then.
      Caution : before the 24th of octobre, the peps_download tool will download >5Go files (full images).

      Denis

  • Bogoljub on 27/03/2017 at 15:52 said:

    >python download_SentFromPEPS.py -c S2ST --lon -114 --lat 55 -a peps.txt -d 2017-03-14 -f 2017-14-03
    I am trying to download data for an area in Canada. Following are errors i receive.
    'lon' is not recognized as an internal or external command,operable program or batch file.
    'startDate' is not recognized as an internal or external
    'completionDate' is not recognized as an internal or
    'maxRecords' is not recognized as an internal or external command
    Value for "lat" must be numeric

    what is it that i am missing in my command??

    • Olivier Hagolle on 28/03/2017 at 13:09 said:

      Hi the date provided in the -f format has a wrong format, which should be YYYY-MM-DD. It is the cause of the problem. (I recognise the error message is wrong...)

  • Max on 04/04/2017 at 12:37 said:

    Hi, thanks for the software, it might help me a lot for my research. But for now, I keep getting errors about date/location. Have I made something wrong ? Here is what I type : python peps_download.py -c S1 -p GRD -lonmin 6 --latmin 44.918 --lonmax 7.173 --lonmax 7.173 --latmax 45.559 -a peps.txt -d 2017-02-15 -f 2017-03-15

    And I get :

    'startDate' n'est pas reconnu en tant que commande interne
    ou externe, un programme exécutable ou un fichier de commandes.
    'completionDate' n'est pas reconnu en tant que commande interne
    ou externe, un programme exécutable ou un fichier de commandes.
    'maxRecords' n'est pas reconnu en tant que commande interne
    ou externe, un programme exécutable ou un fichier de commandes.
    'productType' n'est pas reconnu en tant que commande interne
    ou externe, un programme exécutable ou un fichier de commandes.
    'sensorMode' n'est pas reconnu en tant que commande interne
    ou externe, un programme exécutable ou un fichier de commandes.
    No product matches the criteria

    The ESA copernicus data hub seems to be under maintenance, which might explain for the last message, but not the first.

    Thanks a lot,
    Max

    • Olivier Hagolle on 04/04/2017 at 15:48 said:

      Hi,

      this type of error corresponds to a URL non understood
      I found it : you used -lonmin instead of --lonmin ;)

      Best regards,
      Olivier

      • Max on 05/04/2017 at 07:33 said:

        Hi,
        You are right, I also wrote --lonmax twice. But those mistakes only apperead when I copied by hand my command line in this writing box ;) I get a new one when correcting everything (and even copy-pasting example files) : Value for "box" must follow the pattern ^[0-9\.\,\-]*$

        If I only specify a value for --lat, I get the previous error messages for " 'xxx' is not recognised "and : "Value for "lat" must be numeric"

        Thanks!
        Max

  • Inti on 16/08/2017 at 03:37 said:

    Hello Olivier,

    I just wanted to thank you for the great tool. Also wanted to inform you that it works on windows. I installed aria2 and wget to try out sentinel_download.py but it did not work for me, in which I got a server error.

    Anyway, with this Peps tool it works. i used:
    python ./peps_download.py -c S2 -l 'Toulouse' -a peps.txt -d 2015-11-01 -f 2015-12-01 and is downloading the image from toulouse.

    I just have a couple of questions:
    1.Is is possible to download only tiles as in Sentinel_download.py?
    2. how can set max cloud cover?
    3. how can a list all files fitting my request for an area?

    Many thanks,
    Inti

    • Olivier Hagolle on 16/08/2017 at 10:13 said:

      Hi Inti,
      I am not maintaining much the Sentinel-download, it is difficult to keep up with the changes on ESA side.
      Regarding peps_download, :
      - it is not possible to download single tiles acquired before December 2016 (I asked that to PEPS, but when ESA decided to switch to single tiles, they didn't go on with the changes)
      - using -n option, you get only the list of tiles without downloading
      - I have not implemented the filtering on cloud percentage (I usually download alll the tiles), but that would be possible of course.

      Best regards
      Olivier

  • Inti on 16/08/2017 at 04:06 said:

    Hello Olivier,

    Something weird is when I try another of the examples:

    python ./peps_download.py -c S2 --lon 1 --lat 43.5 -a peps.txt -d 2015-11-01 -f 2015-12-01 it gives the following error message:

    curl -k -o search.json https://peps.cnes.fr/resto/api/collections/S2/search.json?lat=43.500000\&lon=1.000000\&startDate=2015-11-01\&completionDate=2015-12-01\&maxRecords=500
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    100 68 100 68 0 0 43 0 0:00:01 0:00:01 --:--:-- 43
    'lon' is not recognized as an internal or external command,
    operable program or batch file.
    'startDate' is not recognized as an internal or external command,
    operable program or batch file.
    'completionDate' is not recognized as an internal or external command,
    operable program or batch file.
    'maxRecords' is not recognized as an internal or external command,
    operable program or batch file.
    Value for "lat" must be numeric

    What could be the problem? I tried all your examples in the README file and they don't work for me except for: python ./peps_download.py -c S2 -l 'Toulouse' -a peps.txt -d 2015-11-01 -f 2015-12-01

    Many thanks for your help

  • Yinat on 17/10/2017 at 07:13 said:

    Hi Olivier,

    I tried to downloas Sentinel-1 data from peps with the following command :
    python ./peps_download.py -c S1 --lonmin 1 --lonmax 2 --latmin 43 --latmax 44 -a peps.txt -d 2015-11-01 -f 2015-12-01

    but i got this error :
    curl -k -o search.json https://peps.cnes.fr/resto/api/collections/S1/search.json?box=1.0,43.0,2.0,44.0\&startDate=2015-11-01\&completionDate=2015-12-01\&maxRecords=500
    sh: 1: curl: not found
    Traceback (most recent call last):
    File "./peps_download.py", line 172, in
    with open(options.search_json_file) as data_file:
    IOError: [Errno 2] No such file or directory: 'search.json'

    I tried other examples with Sentinel-2 but always the same error.

    I verified that the library json exists in python but it seems not to be the problem.
    Did you know where is the problem ?

    Best regards

  • Yinat on 23/10/2017 at 08:23 said:

    Hi Olivier,

    I come back with a problem when specifying the orbit number with the -o option.
    I tried the command : python peps_download.py --latmin 43.7 --latmax 44.5 --lonmin 0.7 --lonmax 2 -a peps.txt -c S1 -p GRD -d 2017-01-01 -f 2017-01-30 -o 132
    I got :

    curl -k -o search.json https://peps.cnes.fr/resto/api/collections/S1/search.json?box=0.7,43.7,2.0,44.5\&startDate=2017-01-01\&completionDate=2017-01-30\&maxRecords=500\&productType=GRD\&sensorMode=
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    100 375k 0 375k 0 0 148k 0 --:--:-- 0:00:02 --:--:-- 148k
    S1B_IW_GRDH_1SDV_20170129T060743_20170129T060808_004059_007044_05DE 12ae7aa2-f2cd-5910-b8dd-ff8ca60e7454 2017-01-29T06:07:43.127Z disk
    S1A_IW_GRDH_1SDV_20170126T173918_20170126T173943_015006_018823_5477 05ddd06c-8a27-5ba1-8258-62599226f0ae 2017-01-26T17:39:18.059Z disk
    S1A_IW_GRDH_1SDV_20170126T173853_20170126T173918_015006_018823_7F07 8aabcf23-4938-5b8d-993d-72411fcd4051 2017-01-26T17:38:53.591Z tape
    S1B_IW_GRDH_1SDV_20170125T174644_20170125T174709_004008_006EB7_4145 d8ff30b8-8fc4-5efe-bf71-2278b1631dfe 2017-01-25T17:46:44.986Z disk
    Traceback (most recent call last):
    File "peps_download.py", line 205, in
    download_list[prod]=feature_id
    NameError: name 'download_list' is not defined

    But it downloads data without the -o option. Is there a problem with specifying the orbit number?

    Thank you in advance

    • Olivier Hagolle on 23/10/2017 at 09:06 said:

      Thanks for the feedback.
      I am not using S1 a lot... it was a wrong variable name. It has been corrected and pushed to github.
      For next time, please use the "issues" of github, rather than commenting here, it is much easier to handle for me.
      Olivier

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>