Quantitative validation of MACCS cloud masks

=> 

At the annual review  of PEPS project (CNES Sentinel global mirror site), the GEOSYS company showed its operational activities centered on the use of Sentinel-2 data to provide advise on agriculture on many regions of the world. On this occasion, GEOSYS showed their cloud detection process for Sentinel-2 images. The Sen2cor solution was not considered reliable enough y GEOSYS, and the regions processed by MACCS within MUSCATE are far from covering all the regions of interest of the company. GEOSYS decided to rely on human operators to improve the cloud masking. For each processed Sentinel-2 image, a man made valid pixel mask is build ("valid" means without clouds and cloud shadows).

 

I immediately jumped at the bait and suggested a collaboration to GEOSYS, to compare MACCS valid pixel masks to the manual classification of GEOSYS. In this framework, GEOSYS kindly gave me access to several of its cloud masks that I just compared to those from MACCS. Data were processed over 4 tiles scattered over France, during a 3 months period from December 2016 to February 2017 :

Site Tile
Toulouse 31TCJ
Arles 31TFJ
Orléans 31UDP
Rennes 30UWU

 

The tile from Rennes in Britanny did not provide resuts, all the images are fully cloudy and MACCS does not issue a product when the cloud cover is above 90%. But we obtained very good results over the three other sites which are displayed just below :

Toulouse 2017-02-15 confusion % Cloud_maccs Valid_maccs OA
Cloud_geosys 54.18 11.62
Valid_geosys 1.65 32.55 86.73
20170116 confusion % Cloud_maccs Valid_maccs OA
Cloud_geosys 49.36 7.3
Valid_geosys 2.08 41.26 90.62
Orleans 20161130 confusion % Cloud_maccs Valid_maccs OA
Cloud_geosys 0 0.55
Valid_geosys 0 99.45 99.45
20161227 confusion % Cloud_maccs Valid_maccs OA
Cloud_geosys 1.22 4.11
Valid_geosys 1.76 92.91 94.13
20170126 confusion % Cloud_maccs Valid_maccs OA
Cloud_geosys 0.81 2.66
Valid_geosys 0.13 96.4 97.21
20170215 confusion % Cloud_maccs Valid_maccs OA
Cloud_geosys 0.24 1.27
Valid_geosys 0.15 98.35 98.59
20161231 confusion % Cloud_maccs Valid_maccs OA
Cloud_geosys 21.41 2.67
Valid_geosys 2.18 73.73 95.14
Arles 20170103 confusion % Cloud_maccs Valid_maccs OA
Cloud_geosys 2.42 2.77
Valid_geosys 0.41 94.4 96.82
20170113 confusion % Cloud_maccs Valid_maccs OA
Cloud_geosys 29.98 5.11
Valid_geosys 3.89 61.03 91.01
20170202 confusion % Cloud_maccs Valid_maccs OA
Cloud_geosys 82.65 2.87
Valid_geosys 6.7 7.77 90.42
20160209 confusion % Cloud_maccs Valid_maccs OA
Cloud_geosys 87.93 4.43
Valid_geosys 0.23 7.41 95.34

 

This table displays all the confusion matrices obtained for each processed image, and the OA (Overall Accuracy) column provides the percentages of well-classified pixels. This rate goes from 86 to 99.5%. The agreement is excellent, but of course, the worse results bring us more information on how to enhance our method. You will find below two examples of disagreements, please click on the images to see them at full resolution, and see the legends for interpretation.

Cloud masks, left, from GEOSYS, and right, from MACCS, and in the center, Sentinel-2 image.. On both masks, valid pixels appear in black. Here GEOSYS classification is quite acccurate, and even the faintest clouds in the south west corner are detected. MACCS can't detect clouds that are that thin (and besides, congrats to the operator who made this classification for GEOSYS) .

 

Cloud masks, left, from GEOSYS, and right, from MACCS, and in the center, Sentinel-2 image. In that case, the observed zone is fully classified as invalid by GEOSYS, while MACCS finds a significant percentage of valide pixels. . GEOSYS takes large security margins and optimises the production time by simplifying the complex polygons. MACCS looks more accurate even if very small clouds or shadows can stay undetected because the detection is performed at 240 m resolution.

 

As a conclusion of this first independent quantitative validation of our Sentinel2 products, GEOSYS and MACCS agree on 94.5% on average, on 11 different images. The GEOSYS operators do a great work and can detect even the faintest clouds. However, they also tend to dilate the broken clouds generously. Indeed, GEOSYS needs to absolutely avoid cloud omissions, and, besides, the productivity is increased by simplifying the polygons instead of drawing a polygon around each single cloud.  MACCS clouds are dilated too, but not as much as those of GEOSYS. In some cases, MACCS may also miss the faintest clouds, and also thicker ones with a very small surface. We would like to increase the resolution of our cloud mask but it would be at the expense of computation time.

 

These results will be completed by comparison to other processors. The results will be shown on this blog of course, as well as the RAQRS V symposium in Valencia in September.

 

Many thanks to Arnaud Quesney (from GEOSYS) and GEOSYS for providing the data and helping me writing this post.

Posted under: Corrections (geometry, atmosphere, clouds), In English, Sentinel-2, THEIA

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>