=>

peps_download

This is a simple piece of code to automatically download the products provided by the French Sentinel collaborative ground segment named PEPS : https://peps.cnes.fr. PEPS is mirroring all the Sentinel data provided by ESA, and is providing a simplified access.This code was written thanks to the precious help of one my colleagues at CNES Jérôme Gasperi who developed the « rocket » interface which is used by Peps.This code relies on python 2.7 and on the curl utility. Because of that, I guess it only works with linux.The tool is available on my github repository : https://github.com/olivierhagolle/peps_download

Examples

This software is still quite basic, but if you have an account at PEPS, you may download products using command lines like

  • python ./peps_download.py -c S2 -l 'Toulouse' -a peps.txt -d 2015-11-01 -f 2015-12-01 which downloads the Sentinel-2 products above Toulouse, acquired in November 2015.
  • python ./peps_download.py -c S2 --lon 1 --lat 43.5 -a peps.txt -d 2015-11-01 -f 2015-12-01 which downloads the Sentinel-2 products above –lon 1 –lat 43.5 (~Toulouse), acquired in November 2015.
  • python ./peps_download.py -c S1 --lonmin 1 --lonmax 2 --latmin 43 --latmax 44 -a peps.txt -d 2015-11-01 -f 2015-12-01 which downloads the Sentinel-1 products in latitude, longitude box around Toulouse, acquired in November 2015.

Authentification

The file peps.txt must contain your email address and your password on the same line, such as follows :your.email@address.fr top_secretTo get an account : https://peps.cnes.fr/rocket/#/register 

41 thoughts on “How to automatically download Sentinel data from PEPS collaborative ground segment

  1. I am fairly unfamiliar with Python and am having issues getting this code and the Sentinel_download.py code to work. I keep getting syntax errors for the input line (i.e. the decimal degrees for the longitude and latitude and the file path). I have added back slashes and apostrophes in the file path to get it to recognize it as a string, but there continue to be errors. Is there any other formatting that needs to be done to the line? I guess I assumed I could strictly copy and paste your input line and change to suit my parameters. Merci!

    1. Dear Christina,\nI lack information to be sure, but it seems you are first launching python, then pasting the command lines, Which is wrong.\nYou do not need to start python, these command lines are shell command lines.\nBest regards,\nOlivier

  2. Dear Olivier,First, I would like to thank you so much for your work and all the information (and even codes !) you share.I had two questions about downloading and using Sentinel-2 images :1. Is there a way to download specific tiles without having to download the whole scene ?2. Sentinel-1 products are available on the Google Earth Engine API, Sentinel-2 are not (maybe not yet). Do you have an idea of the reason for it ?Many thanks again,Denis

    1. Thanks for your kind words, Denis,\n\nYes it is possible to access directly to the tiles, I have tried to add it in my sentinel-2 download module (with the -t option), but it is nor working yet.\nThe issue is that you have to download all the files one by one, so it is a little complicated.\n\nFor that, you need to use Url like this : (provided in an other comment by vince) \nhttps://scihub.copernicus.eu/dhus/odata/v1/Products(‘141ea4dd-2c99-4ffb-ac07-f5775e08b58f’)/Nodes(‘S2A_OPER_PRD_MSIL1C_PDMC_20160209T011325_R008_V20160208T104841_20160208T104841.SAFE’)/Nodes(‘GRANULE’)/Nodes(‘S2A_OPER_MSI_L1C_TL_SGS__20160208T194222_A003294_T31TCJ_N02.01’)/Nodes(‘IMG_DATA’)/Nodes(‘S2A_OPER_MSI_L1C_TL_SGS__20160208T194222_A003294_T31TCJ_B02.jp2’)/$value\n\nSol, if I find some time I will do it.\n\n\nOn your second point, of course Sentinel-2 data will be made available on GEE, and it should start quickly. But you have to be aware that GEE can use your code and applications and does not guarantee the platform will still be available next year. \n\nAnd finally, starting this summer, Theia will distribute atmospherically corrected data, by tiles, on selected regions such as Tunisia ;-)\n\nBest regards,\nOlivier

      1. Bonjour Denis,\nMy little download tool : Sentinel-download, now allows to download specific tiles instead of whole S2 products thanks to the -t option. I have finally found some time to finish it. I have not tested it much, though, please tell me if you find some bugs.\nOlivier

        1. Bonjour Olivier,I’ve just tried… and didn’t manage to make it. I can’t determine whether the bug is in your script or between my keyboard and my screen, so I post the results here ; I might have misunderstood or been too fast on something.My command is :python Sentinel_download.py –lat 34.75 –lon 10.8 -a apihub.txt -t 32SPF -d 20151225 -f 20151229I tried it both with the guest account and with my personal account (created october, 1st 2015). I also tried with a bigger time period.I browsed the archive on the scihub interface. The download link given for the whole 20151228 S2 scene on my site ishttps://scihub.copernicus.eu/dhus/odata/v1/Products(‘83297ffc-7c04-49f4-b80f-4f88ee77e1c3’)/$valueThe http requests, give first error codes 401 and then correct code 200. I get the query_results.xml but I don’t find file_dir.xml neither do I find granule_dir.xml…At the end I’m said « Tile 32SPF is not available within product »..Any idea ?Many thanks in advance,Denis

          1. By the way, with the command :python Sentinel_download.py –lat 43.6 –lon 1.44 -a apihub.txt -o 51 -t 31TCJ -d 20160101 -f 20160228I manage to get image files (I stopped the download after the first successful download of a .jp2 file).I tried to add the row number for my image of interest :python Sentinel_download.py –lat 34.75 –lon 10.8 -a apihub\ \(copy\).txt -o 122 -t 32SPF -d 20151225 -f 20151229but I get the same result as in the comment above…

          2. Dear Denis, I tried the command :

            python Sentinel_download.py –lat 34.75 –lon 10.8 -a apihub.txt -t 32SPF -d 20151225 -f 20151229

            And what I get is this message :========================================Tile 32SPF is not available within product========================================And in fact, the tile number and the coordinates do not belong to the same place. For the Cap Bon, you are two degrees below. The following command line works for me.

            python Sentinel_download.py –lat 36.75 –lon 10.8 -a apihub.txt -t 32SPF -d 20151225 -f 20151229

            It was easy for me to find, I did the same error in one of my first tests…A bientôt,Olivier

          3. Perfect !How could I have set up this 34.75 ? Now it works, the answer has no interest ! Many thanks again Olivier.Denis

  3. Waw !Thank you so much Olivier. One comment and you develop a tool.I’ll test it as soon as possible. I’m very interested in downloading a specific tile… over Tunisia ! ;-)Denis

  4. Hallo Olivier,is it possible to run that script under Windows? I have installed python 2.7- it came with ArcGIS -, but no skills in using python. Is it possible that you could give me a short hint how to start the script? Would be a great help!Thank you, greetingsFrank

    1. Dear Frank, you have several examples above. It is supposed to work on windows, it needs installing aria2 software.\nAs I am not a windows user, I am not sure it works as well as on linux. Some users reported issues.\nBest regards,\nOlivier

  5. Dear Olivier,I’m back on trying to download tiles on our site, and I was facing a strange behaviour with the same working command as those days :python Sentinel_download.py –lat 36.75 –lon 10.8 -a apihub.txt -t 32SPF -d 2015-12-25 -f 2015-12-29It seems that the date format has change somwhere because with the following command, it works :python Sentinel_download.py –lat 36.75 –lon 10.8 -a apihub.txt -t 32SPF -d 20160316 -f 20160318Hope this helps,Denis

  6. Thanks Denis, you are right, I figured it out some time ago. I had changed the examples in many places blog, github) and in the description with -h option. But not in the comments. I need to correct it too.Olivier

  7. I am using peps but I am getting: python peps_download.py -c S2 -a /home/atos/cron/peps.txt -21 –latmax=41.9744 –lonmax=-8.7556curl -k -o search.json https://peps.cnes.fr/resto/api/collections/S2/search.json?axRecords=500 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 –:–:– –:–:– –:–:– 0Traceback (most recent call last): File « peps_download.py », line 130, in with open(‘search.json’) as data_file:IOError: [Errno 2] No such file or directory: ‘search.json’What is the problem?This used to work…

    1. It seems to work on my side. Maybe Peps was in maintenance ?\nPlease try again.\nOlivier

  8. Dear Olivier,We have a trouble with the sentinel download script. In our lab, we have installed a script on a server which automatically downloads latest image of our 3 tiles of interest.At first the script was launched once a day (now once a week).We had a correct behaviour since the end of october.For the 32SPF tile, the trouble appeared somewhere after the 24th of october image.The behaviour now is that the script creates an empty directory tree (the GRANULE directory contains an empty directory), and runs endlessly.Any idea ?Many thanks in advance,Denis

    1. Thanks to a private communication with Olivier, we went to the following solution : using the peps_download tool to download from the PEPS repository. It’s ok for dates after the 24th of october since PEPS has tiles from then.Caution : before the 24th of octobre, the peps_download tool will download >5Go files (full images).Denis

  9. >python download_SentFromPEPS.py -c S2ST –lon -114 –lat 55 -a peps.txt -d 2017-03-14 -f 2017-14-03I am trying to download data for an area in Canada. Following are errors i receive.’lon’ is not recognized as an internal or external command,operable program or batch file.’startDate’ is not recognized as an internal or external ‘completionDate’ is not recognized as an internal or ‘maxRecords’ is not recognized as an internal or external commandValue for « lat » must be numericwhat is it that i am missing in my command??

    1. Hi the date provided in the -f format has a wrong format, which should be YYYY-MM-DD. It is the cause of the problem. (I recognise the error message is wrong…)

  10. Hi, thanks for the software, it might help me a lot for my research. But for now, I keep getting errors about date/location. Have I made something wrong ? Here is what I type : python peps_download.py -c S1 -p GRD -lonmin 6 –latmin 44.918 –lonmax 7.173 –lonmax 7.173 –latmax 45.559 -a peps.txt -d 2017-02-15 -f 2017-03-15And I get : ‘startDate’ n’est pas reconnu en tant que commande interneou externe, un programme exécutable ou un fichier de commandes.’completionDate’ n’est pas reconnu en tant que commande interneou externe, un programme exécutable ou un fichier de commandes.’maxRecords’ n’est pas reconnu en tant que commande interneou externe, un programme exécutable ou un fichier de commandes.’productType’ n’est pas reconnu en tant que commande interneou externe, un programme exécutable ou un fichier de commandes.’sensorMode’ n’est pas reconnu en tant que commande interneou externe, un programme exécutable ou un fichier de commandes.No product matches the criteriaThe ESA copernicus data hub seems to be under maintenance, which might explain for the last message, but not the first. Thanks a lot,Max

    1. Hi, \n\nthis type of error corresponds to a URL non understood\nI found it : you used -lonmin instead of –lonmin ;)\n\nBest regards,\nOlivier

      1. Hi, You are right, I also wrote –lonmax twice. But those mistakes only apperead when I copied by hand my command line in this writing box 😉 I get a new one when correcting everything (and even copy-pasting example files) : Value for « box » must follow the pattern ^[0-9\.\,\-]*$If I only specify a value for –lat, I get the previous error messages for  » ‘xxx’ is not recognised « and : « Value for « lat » must be numeric »Thanks!Max

  11. Hello Olivier,I just wanted to thank you for the great tool. Also wanted to inform you that it works on windows. I installed aria2 and wget to try out sentinel_download.py but it did not work for me, in which I got a server error.Anyway, with this Peps tool it works. i used: python ./peps_download.py -c S2 -l ‘Toulouse’ -a peps.txt -d 2015-11-01 -f 2015-12-01 and is downloading the image from toulouse.I just have a couple of questions:1.Is is possible to download only tiles as in Sentinel_download.py?2. how can set max cloud cover?3. how can a list all files fitting my request for an area?Many thanks,Inti

    1. Hi Inti,I am not maintaining much the Sentinel-download, it is difficult to keep up with the changes on ESA side.Regarding peps_download, :- it is not possible to download single tiles acquired before December 2016 (I asked that to PEPS, but when ESA decided to switch to single tiles, they didn’t go on with the changes)- using -n option, you get only the list of tiles without downloading – I have not implemented the filtering on cloud percentage (I usually download alll the tiles), but that would be possible of course.Best regardsOlivier

  12. Hello Olivier,Something weird is when I try another of the examples:python ./peps_download.py -c S2 –lon 1 –lat 43.5 -a peps.txt -d 2015-11-01 -f 2015-12-01 it gives the following error message:curl -k -o search.json https://peps.cnes.fr/resto/api/collections/S2/search.json?lat=43.500000\&lon=1.000000\&startDate=2015-11-01\&completionDate=2015-12-01\&maxRecords=500 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed100 68 100 68 0 0 43 0 0:00:01 0:00:01 –:–:– 43’lon’ is not recognized as an internal or external command,operable program or batch file.’startDate’ is not recognized as an internal or external command,operable program or batch file.’completionDate’ is not recognized as an internal or external command,operable program or batch file.’maxRecords’ is not recognized as an internal or external command,operable program or batch file.Value for « lat » must be numericWhat could be the problem? I tried all your examples in the README file and they don’t work for me except for: python ./peps_download.py -c S2 -l ‘Toulouse’ -a peps.txt -d 2015-11-01 -f 2015-12-01Many thanks for your help

  13. Hi Olivier,I tried to downloas Sentinel-1 data from peps with the following command : python ./peps_download.py -c S1 –lonmin 1 –lonmax 2 –latmin 43 –latmax 44 -a peps.txt -d 2015-11-01 -f 2015-12-01but i got this error : curl -k -o search.json https://peps.cnes.fr/resto/api/collections/S1/search.json?box=1.0,43.0,2.0,44.0\&startDate=2015-11-01\&completionDate=2015-12-01\&maxRecords=500sh: 1: curl: not foundTraceback (most recent call last): File « ./peps_download.py », line 172, in with open(options.search_json_file) as data_file: IOError: [Errno 2] No such file or directory: ‘search.json’I tried other examples with Sentinel-2 but always the same error.I verified that the library json exists in python but it seems not to be the problem.Did you know where is the problem ?Best regards

  14. Hi Olivier, I come back with a problem when specifying the orbit number with the -o option.I tried the command : python peps_download.py –latmin 43.7 –latmax 44.5 –lonmin 0.7 –lonmax 2 -a peps.txt -c S1 -p GRD -d 2017-01-01 -f 2017-01-30 -o 132I got : curl -k -o search.json https://peps.cnes.fr/resto/api/collections/S1/search.json?box=0.7,43.7,2.0,44.5\&startDate=2017-01-01\&completionDate=2017-01-30\&maxRecords=500\&productType=GRD\&sensorMode= % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed100 375k 0 375k 0 0 148k 0 –:–:– 0:00:02 –:–:– 148kS1B_IW_GRDH_1SDV_20170129T060743_20170129T060808_004059_007044_05DE 12ae7aa2-f2cd-5910-b8dd-ff8ca60e7454 2017-01-29T06:07:43.127Z diskS1A_IW_GRDH_1SDV_20170126T173918_20170126T173943_015006_018823_5477 05ddd06c-8a27-5ba1-8258-62599226f0ae 2017-01-26T17:39:18.059Z diskS1A_IW_GRDH_1SDV_20170126T173853_20170126T173918_015006_018823_7F07 8aabcf23-4938-5b8d-993d-72411fcd4051 2017-01-26T17:38:53.591Z tapeS1B_IW_GRDH_1SDV_20170125T174644_20170125T174709_004008_006EB7_4145 d8ff30b8-8fc4-5efe-bf71-2278b1631dfe 2017-01-25T17:46:44.986Z diskTraceback (most recent call last): File « peps_download.py », line 205, in download_list[prod]=feature_idNameError: name ‘download_list’ is not definedBut it downloads data without the -o option. Is there a problem with specifying the orbit number?Thank you in advance

    1. Thanks for the feedback.\nI am not using S1 a lot… it was a wrong variable name. It has been corrected and pushed to github.\nFor next time, please use the « issues » of github, rather than commenting here, it is much easier to handle for me.\nOlivier

  15. I input the command ‘python Sentinel_download.py –lat 44.5 –lon 80 -a apihub.txt -d 20170101 -f 20171231 -o 158 -n -s S1A*SLC’. Then the errors ‘ File « Sentinel_download.py », line 7 ^SyntaxError: invalid syntax’ are shown. How to solve the problem.Thanks

  16. Hi Olivier,how it is possible to download data using our own geometry vector as follow:WKT,Name,description,timestamp,begin,end,altitudeMode,tessellate,extrude,visibility,drawOrder,icon »POLYGON Z ((91.0 23.5165536384341 0,90.5 24.0033838608002 0,90.2 24.5100469313974 0,90.15 25.0147145030644 0,90.5 25.5970699400967 0,91.0 25.5548536418031 0,91.5 25.5558223234974 0,92.0 25.7244080499758 0,92.5 25.9714642899991 0,93.0 26.0605811505251 0,93.5 26.1825057872809 0,94.0 26.1738683362909 0,94.5 26.3692228599101 0,95.0 26.9677550408619 0,95.5 27.0837961513984 0,96.0 27.2495743623361 0,96.5 27.2404969432799 0,96.65 27.4978440327206 0,96.5 28.4877353473867 0,96.0 28.8041171174123 0,95.5 28.8852730958532 0,95.0 28.4993551762164 0,94.5 27.7318637043648 0,94.0 27.4446587122329 0,93.5 27.3449597847392 0,93.0 27.1893206203557 0,92.5 27.1597488266727 0,92.0 26.9202623137292 0,91.5 26.7568665914695 0,91.0 26.7602952361474 0,90.5 26.7203546118003 0,90.0 26.5817349371244 0,89.5 26.4711751788167 0,89.23 25.506668929819 0,89.23 24.4934552653587 0,89.25 23.405933984027 0,89.5 22.9903128006586 0,90.0 21.0925157070459 0,90.5 21.0928410280981 0,91.0 21.2477977548221 0,91.25 21.4894497201735 0,91.5 22.0117069833903 0,91.5 22.5036642933011 0,91.0 23.5165536384341 0)) »,Region polygon – Brahmaputra river,,,,,,1,0,-1,,thanks in advance for your help.

    1. Hi, \npeps_download does not allow that (sorry).\nI would advise to find the list of tiles you need, and launch peps_download several times in a loop for each tile, with -t option.\nOlivier

  17. if I want to download special tile like S1A_EW_GRDM_1SDH_20190318T120356_20190318T120456_026392_02F3FD_AF64how can i do this?i must use this as a command line?python Sentinel_download.py -a apihub.txt -s S1A_EW_GRDM_1SDH_20190318T120356_20190318T120456_026392_02F3FD_AF64

  18. Hi Olivier,
    I have recently installed this script but every time I run it, it gives the error ‘error with password file’, but in the password file I have filled in my email and my PEPS password on the same line, so I do not understand where this error comes from.

  19. The problem I just mentioned has been solved. Now, however, using the examples you mentioned, mainly python ./peps_download.py -c S2 -l ‘Toulouse’ -a peps.txt -d 2015-11-01 -f 2015-12-01, it gives the error ‘no product corresponds to selection data’. How is this solved?

  20. Thank you Olivier for this great tool. Is there any limit, on PEPS side, in the maximum number of products you can download (maximum per day for instance) ?

    1. Hi Julien,
      I think there is a maximum per command line, so you have to split your list of products in various command lines, but there is no other explicit limit. However, if my colleagues see you are downloading the entire archive, they might reduce your band pass (Google tried that some time ago…)

Répondre à Olivier Hagolle Annuler la réponse

Votre adresse e-mail ne sera pas publiée. Les champs obligatoires sont indiqués avec *

Ce site utilise Akismet pour réduire les indésirables. En savoir plus sur comment les données de vos commentaires sont utilisées.