Machine learning for land cover map production - Follow-up on the TiSeLaC challenge

I discussed some important aspects to take into account when validating land cover maps in a previous post. In that same post I insisted on the fact that machine learning pipeline building using a blind optimisation of accuracy metrics can lead to unrealistic expectations about land cover maps produced using these approaches.

 

I cited as an example the TiSeLaC challenge, where 2 of the participating teams achieved FScores above 99%, which is an accuracy higher than the one we can expect from the reference data used for the validation.

 

I assumed that this unrealistic performances where due to over-fitting and the use of a validation set too similar to the training set. I have recently asked the challenge organisers about the procedure for splitting the reference data into train and test sets and they confirmed that the split was done at the pixel level and not at the polygon level. Therefore, nearly identical pixels coming from the same polygon could be used for training and validation.

 

Therefore, looking at the challenge results, one could expect that all the teams would have got similar high performances. Since this was not the case, I asked for references to the methods used. Two of the methods are published. I am assuming that these are the 2 winning methods.

 

One of the methods uses spatial nearest neighbour classification to decide the labels, that is, the class for a pixel is decided using the labels of the nearest pixels of the training set. Here, "nearest" means the closest in the image using an Euclidean distance on the spatial coordinates of the pixel. Indeed, the pixel coordinates where provided as a separate record, but I don't think they were intended to be used as features. And, yes, the best results are obtained if only pixel coordinates are used (no reflectances, no NDVI, nothing!). And 1 single neighbour works best than 2-NN or 10-NN.

 

This shows that indeed, neighbouring pixels were present in the training and test sets, and the fewer the information used (just the closest pixel) the better the result obtained.

 

To quickly check this, I ran a simple, out-of-the-box, Random Forest classifier using the coordinates as features and got 97.90% accuracy on the test set, while using the image features gives about 90%.

 

The second of the 2 winning methods (which is actually the first with an FScore of 99.29 while the method above obtains 99.03), uses 3 deep neural networks, 2 of which use temporal convolutions for each pixel. The third network is a multi-layer perceptron were the input features are statistics computed on all the pixels found in a spatial neighbourhood of the pixel to be classified. Different sizes of neighbourhoods between 1 and 17 are used. This is much more complex than using only the label of the closest pixel, but actually, contains the same information. Adding the information of the 2 first networks may allow to correctly classify the few pixels that the previous method got wrong. The performance difference between the 2 methods is less than 0.3%, which may probably fall within typical confidence intervals.

 

What can we learn from these results?

 

First of all, blind metric optimisation without domain knowledge can produce misleading results. Any remote sensing scientist knows that pixel coordinates only are not good predictors for producing a map. Otherwise, one could just spatially interpolate the reference data. Even when applying krigging, other variables are usually used!

 

Second, when organising this kind of contest, realistic data sets have to be used. The split between training and validation has to follow strict rules in order to avoid neighbouring pixels appearing in both data sets.

 

Third, map validation has to have a spatial component: are the shapes of the objects preserved, is there image noise in the generated map, etc. This is a tricky question which needs either to have dense reference data in some places or having specific metrics which are able to measure distortions without reference data. Obtaining dense reference data is very costly to and can even be impossible if some of the classes can't be identified by image interpretation (we are not tagging images of cats or road signs!). Developing specific metrics for spatial quality which don't need reference data is an open problem. Some solutions have been developed for the assessment of pan-sharpening algorithms, but the problem is rather different.

 

Finally, I hope that this critical analysis of the TiSeLaC contest will be useful for future contests, because I think that they may be very useful to get together the remote sensing and the machine learning communities.

MAJA V3.1 will be distributed this May

Example of cirrus cloud correction


We will start distributing MAJA V3.1 this May to replace MAJA V1 on CNES free software platform.

 

It is also in the pipeline of enhancements of Theia processing platform (MUSCATE), but this pipeline is quite full, so we will need to be patient (which requires a big effort for me, patience not being my best quality...)

 

MAJA V3 comes with a lot of enhancements compared to V1 :

Continue reading

ESA is making plans for a global Sentinel2 reprocessing in 2019 to enhance multi-temporal registration

A good piece of news, directly from ESA's S2 project manager: ESA is now making plans for a global reprocessing of Sentinel-2 archive in 2019. As explained here, Sentinel-2 data multi-temporal registration isn't perfect yet. It should be improved in the first quarter of 2019, thanks to the use of ground control points obtained via automatic matching, but the issue with the reprocessing of Sentinel-2 archive had not been addressed. It is now, but of course not before 2019.

 

I let you imagine the amount of processing required to do so for the complete 3.5 years archive of Sentinel-2 at that time, so it will be costly and require hard work, but yet it is indispensable. Let's thank ESA and Copernicus for considering this and letting us know !

 

[MUSCATE news] Early Sentinel-2B L2A images over France are available

=>

MUSCATE sentinel-2 L2A counter just reached 80000 products, but a lot more products were processed these past weeks. Indeed, the MUSCATE team started processing the early Sentinel-2B images acquired from July to October 2017, which were missing on our catalog. Indeed. We only started Level 2A treatments in October 2017, in real time (a bit late, but before the ESA;)). We take advantage of this processing to reprocess Sentinel-2A data, since the quality of the products from MAJA improves with the repetitiveness of the observations. And since we replace old versions with new ones (v 1.7), the product counter is not affected by the reprocessing of S2A data.

 

The reprocessing of data on France and its overseas territories is finished, and the treatment of the other European zones is in progress. We will continue soon with the African zones then those of the rest of the world. If you are using the Sentinel-2 data acquired in the second half of 2017, we encourage you to download them again. We took advantage of this reprocessing to also produce snow products in the France area.

 

 

 

 

 

Theia started the distribution of Venµs products in Near Real Time


Great news for our little Venµs satellite project: Theia has started distributing Venµs L1C data in near real time. The data acquired yesterday are already available on https://theia.cnes.fr

 

 

These L1C products are tagged with 0.9 version, which means that their quality is not as good as what we think we will be able to deliver in a couple of months. The multi-spectral and multi-temporal registration can still be improved, even if it is not too far from our expectations yet in most cases. The image quality teams are finishing an error budget of the current situation, and also preparing the improvement with our Israeli partner who manage the satellite.  We will tell you about that.

 

The L2A delivery will also start in a few weeks, but the L1C teams, who had much more work than expected with this satellite, handled us correct products very late, and we still need to tune a few parameters to provide good quality products.

 

On the Theia website, you will find several ways to download the tiles. My little download tool has also been updated. Once you have registered, and updated the config.cfg file, you will be able to download at once all the products in Australia with the following command line :

python theia_download.py -l 'Australia' -c VENUS -a config.cfg -d 2018-01-01

The data format is explained here. Its packaging with useless zips and tars is still provisional and will be simplified very soon.

 

 

Let's ask ESA to improve Sentinel-2 multi-temporal registration

Update from May 4th 2018 : ESA has started planning for a global reprocessing in 2019

 

The information gathered by Sentinel-2 system on Sentinel-2 orbit, attitude, date accuracy, and viewing directions of all detectors allows an excellent accuracy for the geolocation of all Sentinel-2 pixels. The overall geo-location accuracy is better than 11 or 12 meter, for about 97 % of the cases, which is about the size of one Sentinel-2 pixel. Such a performance is more or less equivalent to that obtained for Pleiades, but Pleiades has a resolution of 0.7m. It is therefore really an achievement, which is  to be credited to ESA, to the satellite and instrument manufacturer, and to the image quality teams (including my CNES colleagues).

 

But even if it is excellent, it is not enough. The standard need for multi-temporal registration errors is 0.3 pixels, and the current performances show that for more than 50% of the cases, the performance does not meet that requirement.

From Sentinel-2 data quality report

 

 

For many  users, I guess, these figures do not mean much, and it is not easy to figure out their impact on real life applications. The animation below (made by the twitter star Simon Gascoin) makes that much more concrete :

Time series of Sentinel-2 images of the construction of Nour solar power station near Ouarzazate in Morocco. (made by Simon Gascoin using Sentinelhub)

 

Continue reading

MAJA now corrects thin cirrus clouds on Landsat-8 and Sentinel-2 images

=>
Thanks to our collaboration with  DLR,, who developed the method [1], and with an ESA funding, we added within MAJA a correction for thin cirrus clouds that works for LANDSAT 8 and Sentinel-2. This correction uses the cirrus band  at 1.38 µm, which allows to estimate cirrus reflectance, which is then subtracted from the other bands with an empiric factor derived from the images.

 

Continue reading

Sentinel-2 goes global

Great news ! As announced in Sentinel-2 Mission status, laser links to geostationary relay satellites are now working, both for S2B (since last October) and S2B (since a few days ago). Sentinel-2 5 days repetitivity is now nominal above all lands, and that's cool ! A big thank you to ESA, Copernicus and all the engineers who strived to achieve that !

 

Map of S2A and S2B acquisition segments on the 28th of February. Almost all segments over continents were acquires, and are available on https://peps.cnes.fr

Une grand nouvelle ! Comme annoncé dans le "Sentinel-2 Mission status", les liaisons par laser vers un satellite relais géo-stationnaire fonctionnent, à la fois pour S2B (depuis Octobre 2017) et S2A (depuis quelques jours). Les deux Sentinel-2 observent les 5 continents avec la répétitivité nominale de 5 jours, et c'est chouette !  Un grand merci à l'ESA, Copernicus et tous les ingénieurs qui ont permis cette réussite !

[MUSCATE News] Reprocessing of Sentinel-2B data aquired from July to October 2017

=>

MUSCATE is in a good shape these days thznkd to the continuous efforts of the development team (CNES and CAP GEMINI) who solved several issues. The counter of Sentinel-2 Level 2A products reached 70 000 products this night, just one month after reaching 60000. If we sum all the products delivered by MUSCATE, we are reaching 99 000 images. MUSCATE also distributes Sentinel-2 Snow masks over mountains, and Spot World Heritage data (old SPOT data reprocessed after ortho-rectification.and made available for free).

 

This good shape allows us to increase our production rhythm. We have started processing the Sentinel-2B data acquired between July and October 2017, as we had started processing S2B in November 2017 only. But since MAJA is a multi-temporal processor, we are in fact starting a complete reprocessing of the data, including S2A and S2B. The quality of S2A products should therefore also benefit from the improved repetitivity of observations.

 

This reprocessing will last several weeks. We are starting with data from France and will go on with our neighbouring European countries, then sites in Maghreb, the remaining sites of Africa, and finally, the rest of the sites in the world.

 

In case you have an urgent need for some tiles, please ask ! (of course, it is only applicable to the tiles already in our list)