Comparison of Level 3A compositing methods

=>

As said in a previous post, we are testing various methods of level 3A production, using SPOT4 (Take 5). The Theia Land Data Center will the use these methods to process Sentinel 2 data. In case you did not click on the link above, let's recall that the level 3A products are monthly composite products of cloud free reflectances. For each pixel, our method computes the weighted average of the reflectances of the dates when the pixel is cloud free. For more details, you will need to follow this link.

 

The work of Mohamed Kadiri at CESBIO, which is funded by the CNES budget for Theia, adressed first the definition of quality indexes for composite products (for more details, may I suggest that you follow this link ?). This work showed that our product has nice performance, but we knew some one would ask us to compare them to the classical methods for level 3A products.

 

Therefore, we compared our product with the famous NDVI Maximum Value Composite (NDVI MVC), developped by our remote sensing ancestors, and used since the most remote antiquity to process AVHRR time series. This method consists in using for each pixel of the level 3A, the reflectances of the date which has the greatest NDVI.  Why ? Mostly because the NDVI of a cloud is very low, often negative, and therefore this method will rather select cloud free pixels. The NDVI MVC comes from a time when the cloud masks were not very accurate.

 

Example of a monthly synthesis obtained with the NDVI MVC methods Example of a monthly synthesis obtained with the weighted average method

This post uses the SPOT4-Take5 data to show a comparison of the performances obtained on the Versailles site, with the NDVI MVC method on the left, and the weighted average on the right. One can clearly see, on the left, the presence artefacts made of whiter and darker dots which are not seen on the image on the right. These artefacts appear when the selected date changes from one pixel to the other. These artefacts are much less visible on the vegetation covered plots, as, for this composite obtained in spring, the vegetation increases quickly, and all the pixels come from the last cloud free date of the synthesis.

 

If we have a look at our quality indicators, which were described in our previous post about composite products , it is obvious that the performances obtained by the weighted average method are much better than those of the NDVI MVC method, either as regards the similarity to the central date image of the Level 3A (in yellow, for the 70 % best pixels and in green for the 95% best pixels), and moreover as regards as the amplitude of artefacts (in blue). The abscissa of the plot is the half of the number of days used in the synthesis, and our recommended value is 21.

 

 

NDVI Maximum Value Composite Weighted Average Composite

Production of LANDSAT L2A data at THEIA to begin shortly.

At CNES, the prototype MUSCATE production facility of THEIA land data centre will soon start the production and distribution of Level 2A Landsat 5 and 7 data, and shortly after of LANDSAT 8 data covering the entire surface of France.

 

Mosaic of LANDSAT 5 & 7 data produced at CESBIO, from both ESA and USGS data. These data are cut in 110 x 110 km² tiles. For each tile, each LANDSAT acquisition with at least a little clear sky corner is provided.

 

For Landsat 5 and 7, we use data from both USGS and ESA : indeed, up to now only ESA has the LANDSAT 5 data that were acquired over the receiving stations of Mas Palomas (Canary Islands), Matera (Italy), and Svalbard (Norway). A transfer to USGS of ESA's data is expected, it may have started in Svalbard, but it has not yet begun for the Matera station, which covers France.

Level 1C

The USGS data are orthorectified, but those from ESA are not, so, as for SPOT4 (Take5), we set up an ortho-rectification processing using the SIGMA tool provided by CNES.  The ESA's products we received 2 years ago also have some flaws (which may have been corrected by now, but given it took months to obtain the data we did not ask for a reprocessing): the thermal band is unusable and you will find here and there colourful bright spots, such as those produced by your neighbour moped on your TV when you were a child. Nevertheless, we can produce correct Level 1 products, although we look forward to the reprocessing of these products by USGS. ESA has now it own processing of LANDSAT data, but it stops at level 1C.

For Landsat 7, this processing is not necessary because the data are already ortho-rectified. We interpolate only a small portion of the missing data due to LANDSAT 7 SLC breakdown, and then we discard the parts of the image where the gaps are too large. For LANDSAT 8, none of these processings are needed.

 

Level 2A

The Level 2A products (Cloud Mask, Atmospheric correction) will be produced by the prototype of MACCS software developed and maintained by Mireille Huc (CESBIO, CNRS). Two years ago, I had already produced such a data set on the most Southern part of France, at CESBIO. These products are already distributed on THEIA web site and are also used to illustrate this post.

 

LANDSAT 5 and 7 :

Starting in April, we will process the LANDSAT 5 & 7 data acquired above France from 2009 to 2011.

 

LANDSAT 8 :

From April or May, we will process the LANDSAT 8 data acquired since april 2013, and we will try to keep the pace so as to produce the incoming new LANDSAT 8 acquisitions with a short delay.

 

Data Format :

We will reuse the data format of SPOT4 (Take5). France will be split into 110*110 km² tiles with a 10 km overlap with their neighbours. (See the image mosaic obtained for the South of France).

Depending on the success of the distribution of these data, we will decide if it is worth producing older time periods or other regions. Please tell us if you need such data.

Exemple : available LANDSAT images from July to October 2009 for the tile centered on Toulouse. For each date, we provide the level 1C image tTOA reflectance), on the left, and the level 2A image on the right (surface reflectance). The detected clouds are circled in red.

SPOT4(Take5) data downloads statistics

Three and a half months after the first release of SPOT4 (Take5) products, we can draw a first analysis of the SPOT4(Take5) data downloads (from THEIA distribution website : http://spirit.cnes.fr/take5/)

 

  • 160 different users have downloaded data, from all over the world.
  • 75% of downloads are Level 2A products, and most of the users who downloaded Level 1C products also downloaded Level 2A.
  • 40% of downloads concern whole time series, 60% concern mono-date images, which means that, with an average number of 14 dates per site, about 90% of product downloads are downloaded as whole time series.
  • Each time series has been downloaded 12 times on average.
  • Two users (I would bet they are French) are named Titi and Toto

Number of downloads per site (only whole site archives)

We have also obtained a ranking of the 45 sites versus the number of downloads, and notice that all 45 sites where at least downloaded once. Well, although CESBIO sites come first (!), this is not a competition, and it is probably biased, as some well organised user groups centralise their downloads, while other labs (among which CESBIO...) have downloaded their sites several times. The least demanded site (Rennes), was added very recently to the list.

 

The number of communications or papers per site will be much more interesting, but as far as I know, the counter is still set to 3 (3 communications at the Living Planet Symposium). Please remember that users are requested to send us all their communications based on SPOT4(Take5) data.

 

PS : this information is gathered thanks to the information provided by users at each download, which is collected on the download site data base (designed by J.Gasperi (CNES)), and then provided to me each month by B.Specht (CNES).

SPOT4(Take5) aerosol optical thickness validation results

We are currently preparing a data reprocessing of all SPOT4 (Take5) data, to be released before the end of 2013. For this, I tested several aerosol models and compiled all the validation results for our multi-temporal Aerosol Optical Thickness (AOT) estimation method named MACCS. Our estimates are compared to AERONET in-situ AOT measurements.

The MACCS method applied to SPOT4(Take5) data, which lacks a blue band, uses two procedures to estimate AOT :

  • either the AOT is estimated by a multi-temporal method
  • or it is gap-filled. The presence of gaps may be due to clouds, water or snow, or because the pixel reflectance is too-high for an accurate estmate, or because of a too large variation of reflectance with time is detected.

 

Comparison of MACCS AOT estimates with the in-situ measurements from AERONET. The blue dots correspond to cases for which the atmosphere is stable and for which there are no clouds in the neighborhood of the AERONET site. The red dots correspond to situations when the AERONET optical thickness varies around the satellite overpass time, or when clouds are detected in the image neighbourhood (20*20 km).
On the left plot, only the dates and sites for which less than 60% of the pixels were gap-filled; wheras the right plot only tolerates 20% of gap-filled pixels. The gap-filling method does not seem to introduce large amount of errors in wases when the atmosphere is stable, but it is less accurate in unstable cases..

 

The aerosol estimates have been obtained with MACCS prototype which is developed and maintained by Mireille Huc at CESBIO. The aerosol model is not the same as the one used for SPOT4 (Take5) first processing. This model is based on greater particles (with a modal radius of 0.2µm, compared to 0.1µm in the initial processing), as it provides a better overall agreement with AERONET measurements. We will use this model for most sites for SPOT4(Take5) reprocessing.

 

The RMS error of AOT estimates is 0.06, which is a state of the art performance, obtained in a very difficult condition with no blue band available. Moreover, in order to show more validation points, a few validation sites (Bruxelles, Gwangju, Ouarzazate, Wallops, NASA_LaRC) are in fact distant by more that 60 kilometers from the image footprint, which tends to degrade the performances.

 

The AERONET sites used in this study are :

 

SPOT4 Take5 Site
Aeronet Site
Belgium Brussels
South Great Plains Cart_Site
Korea Gwangju_GIST
Chesapeake NASA_LaRC
Chesapeake Wallops
Versailles Paris
Versailles Palaiseau
Tunisia Ben Salem
Maroc Saada
Maroc Ouarzazate
Sudmipy-Est Seysses + Le Fauga
Sudmipy-Ouest Seysses
Provence Carpentras
Provence Frioul

 

The worst results are obtained for the following sites :
  • Gwangju (Korea): The SPOT footprint in on the coast, while the AERONET site is 70 km inland, near a large town.
  • Ben Salem (Tunisia): this site was very cloudy in Spring, and large reflectance variations are observed between the remaining clear dates.
  • Palaiseau and Paris : In that case, the aerosol model seems to be inappropriate, and absorbing pollution aerosol should be introduced.

On the contrary, several sites provide very accurate results, for instance in Morocco (even the desertic Ouarzazate), Provence (including the Frioul Island where the AOT is extrapolated from the coast), and also Sudmipy, Wallops et Cart_site. Some SPOT4 (Take5) users reported inaccuracies on some tropical sites but we do not have an AERONET validation site near these SPOT4(Take5) sites.

 

A new version of the SPOT4(Take5) products is available.

Here are the thumbnails from the China(2) site, for which several dates were missing on the version 1.0. Please note that on the server, you may download all the dates at once by clicking on the 1C or 2A buttons.

=>

The CNES teams of the THEIA Land Data Center have reprocessed the SPOT4 (Take5) data, in order to take into account a large number of images that were not processed in the first place, because some data had not been yet received or because their processing had failed due to a few little bugs.

 

The same processors and parameters were used and the only difference is the increased number of available dates, but as the L2A methods are multi-temporal and recurrent, when we add an image, the results on the subsequent images are also changed. It is thus advisable that you download again all the products of the sites you are interested in, from the following address : http://spirit.cnes.fr/take5

 

On this prototype ground segment, our management of product versions is basic, and only takes the processors into account. As the processors are unchanged, the new version 1.1 products are still identified as level 1.0 products in the Metadata. We are sorry for this inconvenience, you will need to pay attention not to mix them with the older version.

 

SPOT4(Take5) : Cloud statistics after one month

=>

We have now received all the L1A images of the SPOT4(Take5) experiment taken between January the 31st and March the 10th, for which at least some part of the surface is visible. We ortho-rectify these images to obtain level 1C products, but sometimes, the cloud cover is still too high to process the image. We can use all these productions to derive some statistics about cloud cover.

 

Proportion of images processed at Level 1A and Level 1C for the sites selected by each agency.
Institution Images acquired L1A processed L1C processed % L1A % L1C
CNES 324 184 157 56 % 49 %
JRC 54 29 27 53 % 50 %
ESA 84 41 34 49 % 40 %
NASA 48 26 26 54% 54%
CCRS 6 1 1 17 % 17 %

 

Between 40% and 50% of the images taken are sufficiently clear so that the ortho-rectification is feasible. When the production of all cloud masks (level2A) is finished, we will be able to compute the number of cloud free observations for each pixel.

After having looked at all the images in Europe or North Africa, we can confirm that all the pixels of these sites have been observed at least once without clouds, except for 3 sites : CAlsace, EBelgium and CTunisia (!). For the site in Alsace, we had to wait until the 4th of March, and until the 10th of March for the site in Tunisia. And up to now, only a little part of the site in Belgium has been observed, on the 8th of March.

 

Number of images acquired in February,
as a function of their cloud cover
Site Clouds < 10% 10% < Clouds < 50% 50% < Clouds < 80% 80% < Clouds
Alpes 2 0 2 2
Alsace 0 0 0 6
Ardèche 1 1 0 4
Loire 1 0 3 2
Bretagne 1 0 1 4
Languedoc 0 2 2 2
Provence 2 3 1 0
SudmipyO 1 1 1 3
SudmipyE 1 1 1 3
VersaillesE 2 0 1 3

In France, despite a very cloudy month of February, the 5 days repetitivity enabled to observe nearly each site at least once. But if SPOT4 had only imaged one out of two overpasses, only the sites in Versailles, Provence and the Alps would have been observed in any case.

 

This result confirms that it is absolutely necessary to launch both Sentinel-2 satellites with a short time interval, so enable the numerous operational applications that need to rely on a monthly clear observation. And it would be a pity if the recent GMES/Copernicus budget cuts resulted in delaying the Sentinel-2B satellite, reducing the repetitivity to only 10 days for several long years.

SPOT4(Take5) first cloud masks

=>

Now that you know almost everything on our cloud detection method and on our shadow detection method, we can show you the first results obtained by Mireille Huc (CESBIO) with SPOT4(Take5) time series. As the method is multi-temporal, it needs an initialisation phase, and we had to wait until we had a sufficient number of images to produce the masks. These first results are not (yet) perfect, but are already quite presentable.

 

The images shown below are a series of 6 Level 1C images, expressed in Top of Atmosphere reflectance, with the contours of several masks orverlayed : the clouds are circled in green, their shadows in black, the water and snow mask are respectively circled in blue and pink. You may click twice on the images to see the details of the masks. These images were acquired in Provence (France), each of them is made from 4 (60x60 km²) SPOT Images obtained on the same day, ortho-rectified, then merged.

 

Most clouds are detected, including very thin clouds, while the number of false cloud detections is very low. Most large cloud shadow are also detected, even if a few of them were missed. The water mask is also quite accurate with nearly no false detections, taking into account it is produced at 200m resolution. The snow is well classified when the snow cover is high, but often, pixels with a moderate snow cover are classified as clouds. This is a classical difficulty with snow masks.

 

However, we know that your sharp eyes will have noticed some very thin clouds partly missed by our classification in the North East of the first image, a few false cloud detections on the 3rd and the 5th images (the ground dries and becomes brighter and whiter), some missed cloud shadows for some small clouds once in a while (we know why, it is an initialisation problem, but quite long to explain...). The cloud detection threshold for water pixels (the method is different from the cloud detection above land), is maybe a little to low, as some bright Camargue Lakes are wrongly classified as cloudy. But after all, for a first run, the result is not bad, and we will refine all the parameters when we have a sufficient number of images.

On the Fourth Image, only two of the 4 (60*60 km²) images are available, because the two others are too cloudy to be ortho-rectified, as we need to see the surface to take ground control points. In fact, the ortho-rectification step is the first of our cloud masking steps.

 

The clouds are circled in green, their shadows in black, the water and snow mask are respectively circled in blue and pink. You may click twice on the images to see the details of the masks.

First Level 2A time series of SPOT4 (Take5) images

(aerosol images have been added at the end of the post)
=>

The verification of the various steps of our SPOT4(take5) processing scheme is going on. On Thursday, we received our first time series, I orthorectified them on Friday, and we were then able to start testing our level 2A processor with the first time series. The one displayed below was obtained on the CESBIO site in Tensift valley : Marrakech is near the center of the image, while the Atlas mountains are in the South East part of the image.

The images on the left column are ortho-rectified, and expressed in Top of Atmosphere reflectance (Level 1C product), while the right column displays the same images after atmospheric correction and cloud detection (Level 2A products), produced by Mireille Huc (CESBIO).

We quickly figured out that the cloud detection would be easy on these very clear images, even if on the February 10th, several diffuse plane contrails can be hardly seen but are partially detected, and some of their shadows as well (clouds are circled by red lines, while shadows are circled by a black line). No false cloud detection is visible. Water bodies and snow are also correctly detected for this first try (circled in blue and purple respectively)

The atmospheric correction, based on a multi-temporal method that detects the aerosols, enabled to detect that the image of February the 5th was hazier than the images of January 31st and February 10th.The February 5th image (left column) has a subtle blueish haze compared to the other dates. On the right column, the tint is roughly constant from one image to the other, which means that the aerosol detection and the atmospheric correction are working well. The aerosol images provided below are also very consistent, with the Atlas mountains playing their role of physical barrier blocking the aerosols on either side of the images. There is an aerosol measurement station on this site but it broke down at the end of January, just for the start of the experiment : Murphy's law...

So, we have reviewed and tested all the steps of the processing, but we still have to check that our methods are sufficiently robust to handle correctly the very diverse situations offered by the 42 sites. How do you say, in English "ce n'est pas une mince affaire" ?

Level 1C products expressed in reflectance at the top of atmosphere.
(c) CNES, processing : CESBIO
Level 2A products expressed in surface reflectance after atmospheric correction
(c) CNES, processing : CESBIO

Aerosol optical thickness images are displayed below. One can note that the image of the February 5th is consitent with a lot of aerosols in the North of the Atlas, and nearly no aerosols in the South. The mountains often act as barriers for the aerosols witch usually stay at a low altitude. The orange dots correspond to the snow mask whereas the red ones correspond to the cloud mask. The brighter spots on the last image may be artifacts.

THEIA : A new French Data Centre dedicated to Land Surfaces

(French Version)

The THEIA Land Data Centre is a French inter-agency initiative designed to promote the use of satellite data, primarily for environmental research on land surfaces but also for public policy monitoring and for management of environmental resources. Its objective is to foster the use of remote sensing data to measure the impact of human pressure and climate on ecosystems and local areas, to observe, quantify and model water and carbon cycles, to follow the evolution of societies and of their activities, including agricultural practices, and to understand the dynamics of biodiversity.

 

Within the Land Data Centre, CNES set up a production centre named MUSCATE. This centre aims are providing users with ready-to-use products derived from time series of images acquired over large areas. Sentinel-2 will of course be the spearhead of the production centre, but before the launch of the Sentinel-2, MUSCATE will already begin to produce data from the SPOT4 (Take 5) experiment. At the same time, the processing centre also prepares the production of all Landsat data acquired over mainland France from 2009 to 2011.

 

MUSCATE production centre already exists in the form of a prototype developed by CNES with strong support from Cap Gemini. This prototype is already able to handle LANDSAT, SPOT, FORMOSAT-2, Sentinel-2 and Venμs data, using processors developed by CNES for geometric processing [1], and developed by CESBIO for cloud detection [2] and for atmospheric correction [3]. Simultaneously, the development of an operational production facility is being specified.

Products provided by the MUSCATE Centre are:

Simulations of SPOT4(Take5) products from Formosat-2 data
  • Level 1C (orthorectified reflectance at the top of the atmosphere)
  • Level 2A (ortho-rectified surface reflectance after atmospheric correction, along with a mask of clouds and their shadows, as well as a mask of water and snow).
  • Level 3A (bi-monthly or monthly composite products of surface reflectances, obtained as the weighted average surface reflectance of non-cloudy pixels obtained during the period). Up to now, Level 3A chain is only available for Venμs satellite.

The data produced by MUSCATE will be freely distributed to research laboratories on the one hand, and to the French public institutions on the other, they will be if possible distributed freely to a wider community. The Land Data Center is also building a distribution server to make all these data available.

 

Further reading about these products :

[1]: Baillarin, S., P. Gigord, et O. Hagolle. 2008. « Automatic Registration of Optical Images, a Stake for Future Missions: Application to Ortho-Rectification, Time Series and Mosaic Products ». In Geoscience and Remote Sensing Symposium, 2008, 2:II‑1112‑II‑1115. doi:10.1109/IGARSS.2008.4779194.

[2]: Hagolle, Olivier, Mireille Huc, David Villa Pascual, et Gérard Dedieu. 2010. « A multi-temporal method for cloud detection, applied to FORMOSAT-2, VENµS, LANDSAT and SENTINEL-2 images ». Remote Sensing of Environment 114 (8) (août 16): 1747‑1755. doi:10.1016/j.rse.2010.03.002.

[3]: Hagolle, O, G Dedieu, B Mougenot, V Debaecker, B Duchemin, et A Meygret. 2008. « Correction of aerosol effects on multi-temporal images acquired with constant viewing angles: Application to Formosat-2 images ». REMOTE SENSING OF ENVIRONMENT 112 (4) (avril 15): 1689‑1701. doi:10.1016/j.rse.2007.08.016.