Download_landsat_scene.py

(updated on April the 24th, 2014, and on July the 24th, and on August the 19th, with a good simplification of interface)

Le bout de code Python (2.7) dont le lien est fourni ci-dessous, permet de télécharger automatiquement des données LANDSAT, sur la version actuelle du site EarthExplorer.

 

Il fonctionne pour LANDSAT 8 et pour LANDSAT5 et 7, et nécessite que les données soient en ligne. C'est le cas semble -t'il pour toutes les données LANDSAT 8, mais pour les données LANDSAT5 ou LANDSAT7, il peut être au préalable nécessaire de demander la production sur le site http://earthexplorer.usgs.gov. Pour que le programme fonctionne, il faut bien sûr un compte sur Earth explorer, et il vous faudra configurer vos comptes, mots de passe et répertoires dans le corps du programme.

 

Ce programme peut être lancé de deux manières différentes :

  • En choisissant une scène LANDSAT définie par ses coordonnées WRS-2, par exemple, (198,030) pour Toulouse (option -s). Il faut aussi fournir une date de début de période (option -d), et eventuellement une date de fin (option -f), sinon, c'est la date du jour d'utilisation qui est prise par défaut comme date de fin.
    • exemple :
    • download_landsat_scene.py -o scene -d 20130128 -s 181025 -u usgs.txt -p proxy.txt --output /mnt/data/LANDSAT8/N0/
  • En fournissant en entrée un fichier avec la liste des produits à télécharger (un produit par ligne)
    • exemple :
      python download_landsat_scene.py -o liste -l list2_landsat8.txt -u usgs.txt --output /mnt/data/LANDSAT8/N0/
    • Avec un fichier list2_landsat8.txt comme ci-dessous, il faut aller chercher les références des produits sur earthExplorer.
    • Tunisie LC81910352013160LGN00
      Tunisie LC81910362013160LGN00

Le fichier usgs.txt doit contenir votre compte et mot de passe sur la même ligne séparés par un espace.

Si vous n'utilisez pas l'option --output, les fichiers seront téléchargés sur le répertoire /tmp/LANDSAT, à condition qu'il existe.

Pour voir toutes les options :

download_landsat_scene.py -h

Code

J'ai déplacé le code sur GitHub, qui est beaucoup plus pratique et fait pour ça.

https://github.com/olivierhagolle/LANDSAT-Download

 

 

The PYTHON 2.7  routine provided below enables to automatically download LANDSAT data, using the actual (April 2014) version of EartExplorer.

 

It works for LANDSAT 8 and LANDSAT 5&7, but needs that the data be already online. It seems to be systematically the case for LANDSAT 8, but for the older LANDSAT, it may be necessary to first order for the production of L1T products., on the earthexplorer site. And of course, you will need to have an account and password on the Earthexplorer website, and change the username and password in the connect earthexplorer routines.

 

This routine may be used in two ways :

  • by providing the WRS-2 coordinates of the LANDSAT scene, for instance, (198,030) for Toulouse (-s option). You will have to provide also a start date (-d option) and a final date (-f option). If the final date is not provided, the default final date is the current date. Example:
download_landsat_scene.py -o scene -d 20130128 -s 181025 -u usgs.txt -p proxy.txt --output /mnt/data/LANDSAT8/N0/
  • by providing a list of products to download, as in the example below:
python download_landsat_scene.py -o liste -l list2_landsat8.txt -u usgs.txt --output /mnt/data/LANDSAT8/N0/

with a file list2_landsat8.txt as provide below (the landsat references must exist in the earthexplorer catalog) :

Tunisie LC81910352013160LGN00
Tunisie LC81910362013160LGN00

The usgs.txt must contain your username and password on the same line separated by a blank.

If you do not use the --output option, the files will be downloaded to /tmp/Landsat (provided it exists)

To see all the options :

download_landsat_scene.py -h

Code

I have moved the code to GitHub, which is much more convenient :

https://github.com/olivierhagolle/LANDSAT-Download

28 comments

  • Olivier Hagolle on 10/02/2014 at 14:41 said:

    I just corrected the value of date_asc, which was incorrect when doy was below 100. Landsat 8 first image was after doy 100 in 2013.

    date_asc="%04d%03d"%(options.annee,doy)

  • Guido Lemoine on 17/04/2014 at 13:01 said:

    Olivier,

    Thanks for the script. It's a little buggy (there is no -t option, for instance), but after some logical repairs, it seems to try out the downloads. However, I get HTTP Error: 500, for instance, when using:

    -o scene -a 2014 -p LC8 -d 60 -f 100 -s 198024

    LC81980242014060LGN01
    http://earthexplorer.usgs.gov/download/4923/LC81980242014060LGN01/STANDARD/EE
    HTTP Error: 500 http://earthexplorer.usgs.gov/download/4923/LC81980242014060LGN01/STANDARD/EE

    To understand this correctly, though, am I expected to first make this selection in EarthExplorer so that the selected imagery are queued as an order? In other words, is your script a replacement for the "Bulk Download App" provided by USGS?

    If your script can skip the pre-selection stage, it would be very useful indeed.

    With greetings,

    Guido Lemoine

  • Guido Lemoine on 17/04/2014 at 13:08 said:

    Sorry, I resolved the problem. -d 68 -f 100 works, and I had to change the version to 00.

    Wonderful. This is very useful!!

    • Olivier Hagolle on 18/04/2014 at 07:44 said:

      Thanks for the comments Guido, you are right, I need to update the post, my own version has also evolved.
      The idea for the code is to avoid the selection of data on Earth Explorer, but it only downloads images which have been processed to level 1C. On some LANDSAT tiles, you will need to first order the tiles which have not yet been processed.

  • roberto on 18/06/2014 at 10:00 said:

    Hi to all,
    this algorithm could be very usefull but in my PC it doesn't work.
    The error is in line 71 in g;10
    i'm not expert of python can you send me a reference where i can study it.
    ty
    Roberto

    • Olivier Hagolle on 18/06/2014 at 12:32 said:

      Thanks Roberto,
      there is indeed a problem with the tool that provides the syntax higlighting.
      It seems to be replacing the "" char by ">".
      So you could try to change these character sequences back to their true values in the code you copied and pasted.

      Meanwhile, I will try to find another syntax highlighting tool.
      Hope the tool will work for you now.
      Olivier

      • roberto on 13/08/2014 at 11:59 said:

        thank you oliver, I'm reading your reply only now, but i have studied python and i costumized the program for my python version and it work.
        Thank you, how can i cite you when i use this sript?

  • florian helen on 19/06/2014 at 06:54 said:

    hello,
    I'm trying to use landsat python code in order to download several images on new caledonia with a text file.

    ./landsat_download.py -o liste -l lst_landsat.txt

    Obviously i have changed login and pasword by i've a problem with connecting to the earthexplorer website.

    f = opener.open('https://earthexplorer.usgs.gov/login', params)
    doesn't work and last message error is:
    urllib2.URLError: .

    Could you please tell me if i have to replace user by Username on the website earthexplorer and passwd by password on earthexplorer

    user='CNESNET\login'
    passwd='secret'
    proxy_info = {
    'user' : 'Myself',
    'pass' : 'secret',
    'host' : 'proxy.esa.int',
    'port' : 8050

    Moreover what is 'host', do i have to change it.

    Thank you for your help.

    • Olivier Hagolle on 20/06/2014 at 20:12 said:

      Bonjour Florian,
      If your internet connection does not go through a proxy, you should uncomment
      #connect_earthexplorer_no_proxy()
      and comment the line :
      connect_earthexplorer_proxy()

      A true programmer would have added an option in the command line, but I am not a true one... sorry.
      Olivier

      • Phill on 21/07/2014 at 18:53 said:

        Hey,

        First of all, thank you for laying the groundwork for this. I have not been able to find a similar script so far. I haven't been able to get it to work, but my USGS account seems to be on the fritz (like this script the official bulk download app, return authentication failed, even though I am 100% there is no typo in the login info).

        Anyway, you might want to fix the function def of the no proxy statement, it is missing an underscore.

        right now: def connect_earthexplorer_noproxy():
        should be: def connect_earthexplorer_no_proxy():

        Also, in the if loop, where you define what happens if option == 'liste', I believe one of the def connect_earthexplorer_proxy should be a ...no_proxy.

        Again, thank you getting this started. Have you considered sharing this code on GitHub? People could more easily add to the progress of the script and add some functionality/fix bugs.

        • Olivier Hagolle on 24/07/2014 at 20:42 said:

          Thanks for the suggestions, i'll correct the typos. The wordpress plugin I use to provide the code is not easy to use and I have to make modifications in the code which do not allow then to test it.

          And I have moved the code to GitHub, which is really more convenient. Thanks for the suggestion.
          Olivier

  • Hoa Le on 15/07/2014 at 08:47 said:

    Hi,

    I'm trying to used landsat python. But I met errors.
    Could you tell me how can do it?

    Thanks

    C:\Users\Hoa\Desktop\Download_landsat_scence.py -o
    scene -a 2014 -d 228 -f 365 -s 123052
    /to/be/changed//SCENES/123_052/GZ
    http://earthexplorer.usgs.gov/download/4923/LC81230522014228LGN00/STANDARD/EE
    Traceback (most recent call last):
    File "C:\Users\Hoa\Desktop\Download_landsat_scence.py", line 168, in
    connect_earthexplorer_proxy()
    File "C:\Users\Hoa\Desktop\Download_landsat_scence.py", line 38, in connect_ea
    rthexplorer_proxy
    f = opener.open('https://earthexplorer.usgs.gov/login', params)
    File "C:\Python27\ArcGIS10.1\lib\urllib2.py", line 394, in open
    response = self._open(req, data)
    File "C:\Python27\ArcGIS10.1\lib\urllib2.py", line 412, in _open
    '_open', req)
    File "C:\Python27\ArcGIS10.1\lib\urllib2.py", line 372, in _call_chain
    result = func(*args)
    File "C:\Python27\ArcGIS10.1\lib\urllib2.py", line 1207, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
    File "C:\Python27\ArcGIS10.1\lib\urllib2.py", line 1174, in do_open
    raise URLError(err)
    urllib2.URLError:

    • Olivier Hagolle on 26/07/2014 at 14:20 said:

      Hi, the tool is not a professional one (you noticed). There are some lines that you must change within the code. I would not advise someone who does not know python.. For instance, the Landsat files are written to a directory /to/be/changed. Of course, this directory must be changed to be adapted to your platform.

      However I am trying to provide a more "profesional" code here : https://github.com/olivierhagolle/LANDSAT-Download
      Olivier

  • Dan Turner on 11/08/2014 at 20:43 said:

    Thank you for distributing the code. It is a great help! Note to potential users: python experience is necessary for debugging. This tool is appears to be very useful but will take some work to implement. I am running from a .bat file on windows. As indicated above the '-t' in the example does not work. The example line (line 134) in the code itself worked after I included the full paths for python.exe, the program and usgs.txt.

  • Mauro on 25/03/2015 at 23:00 said:

    Hi Olivier, thanks for the download_landsat_scene.py script.
    I've added the capability to resume partially downloaded files, and made some other small improvements to the script.

    I would be happy to contribute all that. Just send me an email so I can send you the file.

    Regards,
    Mauro

  • Mauro on 26/03/2015 at 02:28 said:

    Olivier, I just forked your project on github. I'll send you a pull request soon.

    • Olivier Hagolle on 26/03/2015 at 10:10 said:

      Great !
      Thanks a lot for this useful improvement. I need to find some time to test it, but it is quite hard these days.
      Olivier

      • Mauro on 27/03/2015 at 18:10 said:

        Hi, I did some manual testing, just comparing (using diff) a file downloaded at once, with the same file downloaded with multiple interruptions and resumes.
        Also tested for some basic things like resuming a download vs. removing the file and downloading it from the beginning, resuming downloading a zero sized file, etc.

      • Mauro on 27/04/2015 at 01:02 said:

        Hi Olivier,

        I noticed that download resume only works with recent(Landsat 8) images.
        Added a range header check to the download resuming code. Now it falls back to re-downloading the whole file in case the server does not support download resuming(that is, for LT5/LE7 images).

        Regards.

  • Stan on 19/04/2016 at 08:23 said:

    Hi,
    Thank you very much for automatic download algorithms, however I had a small question about the txt file for the user name. He did not recognize my login, it gives me the following error message: Authentification is probably wrong, I put the form in the txt file nom.adresse@mail.com mot de passe.
    Can you help me please.
    Thank

  • Jens Hiestermann on 31/10/2016 at 13:34 said:

    Hi Olivier,

    Great work! It was working well yesterday and then I tried the script on my other desktop but run into issues and not quite sure how to resolve it.

    The query I ran was:
    C:\Python27_64bit\Lib\site-packages\LANDSAT-Download-master>download_landsat_scene.py -o scene -d 20161001 -s 170075 -u usgs.txt --output E:/

    Then the error I got was:
    None None
    Searching for images on (julian date): 2016278...
    http://earthexplorer.usgs.gov/download/4923/LC81700752016278LGN00/STANDARD/EE
    error : file is in html and not an expected binary file
    result saved in ./error_output.html

    However when I paste "http://earthexplorer.usgs.gov/download/4923/LC81700752016278LGN00/STANDARD/EE" into my browser as a quick check, it finds the file and downloads.

    Please assist if you can, we are really keen to use this awesome script.

    Thanks

    Jens

    • Olivier Hagolle on 02/11/2016 at 14:35 said:

      I have updated the script a few days ago, having a similar problem as your's.
      Do you use the very last version ?

      Earthexplorer server is down today. I'll try that tomorrow. Please remind me if I forget.
      Olivier

  • Gui Vill on 15/02/2017 at 18:30 said:

    En que parte del código ingreso este comando

    python LANDSAT8.py -o liste -l liste_landsat8_site.txt -u usgs.txt --output /mnt/data/LANDSAT8/N0/

    Necesito programar para bajar estas 2 imágenes
    LC80110622013147LGN00
    LC80020732014135LGN00

    Por favor alguien que me pueda apoyar..

    Gracias

  • Gui Vill on 15/02/2017 at 23:26 said:

    Buenas tardes

    y para bajar el FR_BUND/EE
    como cambiaría el código Python

    Saludos

Leave a Reply to Guido Lemoine Cancel reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>