Classification Accuracy Assessment

Accuracy assessment was performed using the leave-one-out cross-validation method available when using the kNN classifier.  The set of reference observations collected for kNN classification can also be used simultaneously for accuracy assessment via leave-one-out cross-validation which reduced the total number of reference sites necessary for both training and testing. (Gong 1986)  This method utilizes all available reference sites to perform accuracy assessment without introducing bias.  Accuracy assessment is stratified by SCCU and calculated using error matrices four measures of accuracy: users accuracy, producers accuracy, overall accuracy and the kappa statistic.  Results of the error matrices for each SCCU are listed below.


Minnesota 2000 Landsat Cover Type Map										
Accuracy Assessment										
Last updated: 3/10/05										
Associated File: mn_2000_k7_level1.img										
										
										
Statistics For Categorical Variables 		

Overall Accuracy:		
Categorical variable:		level1
# of Observations:		6325
Trace sum:		5345
Overall accuracy:		0.845
Kappa:		0.808


Class	P. Acc.	U. Acc.
Urban	91.67	95.4
Agriculture	87.34	80.27
Grassland	68.82	72
Forest	82.64	92.52
Water	96.61	99.23
Wetland	72.67	66.93
Shrubland	13.73	3.55
	0	0

---------------------		
SCCU:		1
Categorical variable:		level1
# of Observations:		473
Trace sum:		384
Overall accuracy:		0.812
Kappa:		0.762


Class	P. Acc.	U. Acc.
Urban	84.71	93.51
Agriculture	81.58	78.98
Grassland	75	82.5
Forest	83.49	89.22
Water	93.88	100
Wetland	61.54	39.02
Shrubland	25	20
	0	0

---------------------		
SCCU:		2
Categorical variable:		level1
# of Observations:		435
Trace sum:		368
Overall accuracy:		0.846
Kappa:		0.809


Class	P. Acc.	U. Acc.
Urban	90.8	96.34
Agriculture	90.74	63.64
Grassland	76	95
Forest	79.43	93.33
Water	98.31	100
Wetland	73.13	73.13
Shrubland	100	18.18
	0	0

---------------------		
SCCU:		3
Categorical variable:		level1
# of Observations:		339
Trace sum:		269
Overall accuracy:		0.794
Kappa:		0.725


Class	P. Acc.	U. Acc.
Urban	77.32	92.59
Agriculture	87.5	75.9
Grassland	0	0
Forest	81.3	91.74
Water	100	90.91
Wetland	65.62	50
Shrubland	0	0
	0	0

---------------------		
SCCU:		4
Categorical variable:		level1
# of Observations:		311
Trace sum:		252
Overall accuracy:		0.81
Kappa:		0.737


Class	P. Acc.	U. Acc.
Urban	89.47	70.83
Agriculture	90.62	78.38
Grassland	75	50
Forest	78.81	91.54
Water	100	97.44
Wetland	69.7	70.77
Shrubland	0	0
	0	0

---------------------		
SCCU:		5
Categorical variable:		level1
# of Observations:		294
Trace sum:		246
Overall accuracy:		0.837
Kappa:		0.783


Class	P. Acc.	U. Acc.
Urban	96.49	98.21
Agriculture	82.5	89.19
Grassland	0	0
Forest	77.69	94.39
Water	97.83	100
Wetland	66.67	34.29
Shrubland	0	0
	0	0

---------------------		
SCCU:		6
Categorical variable:		level1
# of Observations:		346
Trace sum:		311
Overall accuracy:		0.899
Kappa:		0.869


Class	P. Acc.	U. Acc.
Urban	96.67	96.67
Agriculture	88.1	92.5
Grassland	71.43	55.56
Forest	87.7	95.54
Water	100	100
Wetland	83.33	75
Shrubland	0	0
	0	0

---------------------		
SCCU:		7
Categorical variable:		level1
# of Observations:		327
Trace sum:		262
Overall accuracy:		0.801
Kappa:		0.739


Class	P. Acc.	U. Acc.
Urban	97.01	100
Agriculture	68.42	72.22
Grassland	57.14	57.14
Forest	74.62	83.62
Water	98.08	100
Wetland	62.75	53.33
Shrubland	0	0
	0	0

---------------------		
SCCU:		8
Categorical variable:		level1
# of Observations:		260
Trace sum:		197
Overall accuracy:		0.758
Kappa:		0.652


Class	P. Acc.	U. Acc.
Urban	94.44	97.14
Agriculture	0	0
Grassland	0	0
Forest	71.32	81.42
Water	100	100
Wetland	45.24	39.58
Shrubland	0	0
	0	0

---------------------		
SCCU:		9
Categorical variable:		level1
# of Observations:		414
Trace sum:		345
Overall accuracy:		0.833
Kappa:		0.787


Class	P. Acc.	U. Acc.
Urban	92.96	91.67
Agriculture	84.72	87.14
Grassland	83.33	66.67
Forest	82.05	92.09
Water	100	100
Wetland	64.91	56.92
Shrubland	0	0
	0	0

---------------------		
SCCU:		10
Categorical variable:		level1
# of Observations:		368
Trace sum:		313
Overall accuracy:		0.851
Kappa:		0.809


Class	P. Acc.	U. Acc.
Urban	96	94.74
Agriculture	75.76	62.5
Grassland	69.23	69.23
Forest	84.96	93.39
Water	100	100
Wetland	71.43	76.27
Shrubland	0	0
	0	0

---------------------		
SCCU:		11
Categorical variable:		level1
# of Observations:		371
Trace sum:		326
Overall accuracy:		0.879
Kappa:		0.848


Class	P. Acc.	U. Acc.
Urban	98.63	97.3
Agriculture	87.1	67.5
Grassland	76.47	86.67
Forest	90.83	95.61
Water	98.28	100
Wetland	69.57	78.69
Shrubland	0	0
	0	0

---------------------		
SCCU:		12
Categorical variable:		level1
# of Observations:		299
Trace sum:		256
Overall accuracy:		0.856
Kappa:		0.824


Class	P. Acc.	U. Acc.
Urban	93.55	97.75
Agriculture	90.32	66.67
Grassland	72.97	93.1
Forest	82	97.62
Water	86.79	97.87
Wetland	87.1	67.5
Shrubland	0	0
	0	0

---------------------		
SCCU:		13
Categorical variable:		level1
# of Observations:		317
Trace sum:		282
Overall accuracy:		0.89
Kappa:		0.865


Class	P. Acc.	U. Acc.
Urban	92.13	97.62
Agriculture	91.67	93.22
Grassland	71.43	86.96
Forest	81.25	86.67
Water	95.45	97.67
Wetland	91.67	83.02
Shrubland	0	0


---------------------		
SCCU:		14
Categorical variable:		level1
# of Observations:		406
Trace sum:		351
Overall accuracy:		0.865
Kappa:		0.833


Class	P. Acc.	U. Acc.
Urban	89.58	94.51
Agriculture	93.94	74.7
Grassland	79.17	95
Forest	92.31	96.97
Water	95.74	100
Wetland	68.25	74.14
Shrubland	0	0
	0	0

---------------------		
SCCU:		15
Categorical variable:		level1
# of Observations:		434
Trace sum:		374
Overall accuracy:		0.862
Kappa:		0.828


Class	P. Acc.	U. Acc.
Urban	93.6	95.9
Agriculture	77.42	72.73
Grassland	42.31	35.48
Forest	94.23	98.99
Water	98.04	100
Wetland	76.92	89.29
Shrubland	0	0
	0	0

---------------------		
SCCU:		16
Categorical variable:		level1
# of Observations:		302
Trace sum:		270
Overall accuracy:		0.894
Kappa:		0.86


Class	P. Acc.	U. Acc.
Urban	95.95	98.61
Agriculture	100	84.62
Grassland	77.78	84
Forest	85.71	95.58
Water	95.24	90.91
Wetland	89.47	80.95
Shrubland	0	0
	0	0

---------------------		
SCCU:		17
Categorical variable:		level1
# of Observations:		138
Trace sum:		114
Overall accuracy:		0.826
Kappa:		0.78


Class	P. Acc.	U. Acc.
Urban	89.74	100
Agriculture	86.67	90.7
Grassland	73.33	68.75
Forest	68.75	84.62
Water	86.67	100
Wetland	75	37.5
Shrubland	50	20
	0	0

---------------------		
SCCU:		18
Categorical variable:		level1
# of Observations:		260
Trace sum:		237
Overall accuracy:		0.912
Kappa:		0.892


Class	P. Acc.	U. Acc.
Urban	91.43	100
Agriculture	100	100
Grassland	82.35	70
Forest	90.7	100
Water	91.11	100
Wetland	87.76	86
Shrubland	0	0
	0	0


---------------------		
SCCU:		19
Categorical variable:		level1
# of Observations:		134
Trace sum:		93
Overall accuracy:		0.694
Kappa:		0.598


Class	P. Acc.	U. Acc.
Urban	62.5	58.82
Agriculture	77.59	88.24
Grassland	27.78	33.33
Forest	82.76	96
Water	87.5	100
Wetland	0	0
Shrubland	66.67	20
	0	0

---------------------		
SCCU:		20
Categorical variable:		level1
# of Observations:		97
Trace sum:		77
Overall accuracy:		0.794
Kappa:		0.747


Class	P. Acc.	U. Acc.
Urban	82.61	90.48
Agriculture	93.1	93.1
Grassland	50	42.86
Forest	75	85.71
Water	100	100
Wetland	100	75
Shrubland	14.29	16.67
	0	0

---------------------
-----

< 7.5 meters

< 7.5 meters

Image Selection

To cover the entire state of Minnesota in a single season with Landsat imagery a minimum of 19 images were needed.   Previous studies suggest that using a multi-temporal approach can provide better discrimination among land-use classes of interest (Yuan et al., 1998).  Therefore, nineteen images for each of three dates, spring, summer, and fall were collected.   Fifty seven images total, acquired by both Landsat-7 ETM+ and Landsat-5 TM between 1999 and 2001, were selected.  These landsat imagery is listed in the file named "2000_Minnesota_Classification_Scene_Summary_Map.ppt".  Further, to reduce complicating factors, such as atmospheric scattering, only clear, cloud-free images were chosen.  


Rectification and Resampling

Images were rectified to UTM zone 15, GRS1980, NAD83 projection.  A second order rectification model was calculated for each image using approximately 20 well distributed ground control points.  The nearest neighbor algorithm was used for image resampling to the new coordinate system and to a common image resolution of 30-meter pixels.  For each image, the root mean square error of the rectification was less than 7.5 meters (1/4-pixel).  


Radiometric Normalization and Tasseled Cap Transformation 

After rectification each image was transformed to at-satellite reflectance to remove or normalize variation arising from changing view and illumination geometry (Markham and Barker, 1986).  Additionally, tasseled cap transformation algorithms for ETM+ imagery require that the imagery be converted to at-satellite reflectance prior to the transformation.  Each image was transformed to tasseled cap features to remove inter-correlation of spectral bands and reduce the dimensionality of image features  Coefficients used for computing tasseled cap values for the at-satellite corrected ETM+ and TM imagery were those reported by Huang et al., (2002). 


Image Mosaicing  and Stacking

Mosaicing of imagery was performed per season to produce three statewide images.  Images were first mosaicked along north-south along paths, then the resulting five images were mosaicked east-west to create a single statewide image.  Image overlap was utilized to reduce and remove any cloud or haze covered areas.  Following the mosaic, each date was clipped with a polygon vector file of the Minnesota state boundaries.  The three statewide images were layer stacked to create a single image with 12 features.  These included, tasseled cap brightness, greenness, and wetness features from spring, summer, and fall imagery along with the thermal band from each date.  This image named "statewide_mosaic_mn_2000.img"  was used to perform all classifications.


Classification Scheme

The classification scheme was modeled after the Upper Midwest Gap Analysis Program Image Processing Protocol (Lillesand et al. 1998).  This classification scheme identifies the land cover types of the Upper Midwest, is compatible with existing national systems, and provides a realistic classification hierarchy for the Landsat TM and ETM+ sensors.  Further, these classes are consistent with earlier classifications allowing for year to year classification comparison.  The levels and classes considered in this classification are listed in the entity and attribute section of this metadata document.


Reference Data

One of the challenges to large area image classification is the collection of reference data that is complete in both spatial and spectral extent.  Various methods have been established for creating these reference data sets ranging from statistical sampling techniques coupled with exhaustive manual land cover classification from high resolution color photography to random selection with in situ checks (Lillesand et al. 1998).  Needless to say, collection of reference data over such a large region is a time consuming and costly part of good classification schemes.   

Creating reference data from scratch using intensive data collection processes, however, may not be necessary when other available alternatives exist.  The most important factor when collecting reference data over a large region is that a statistically significant and up-to-date reference data set is available for each land cover class.   Fortunately, federal, state, and county agencies who are charged with management of natural resources often acquire land cover data sets.  These data alone do not often comprise all the data necessary to create a complete classification data set, however, combining two or more can provide the data needed by remote sensors to produce a sufficient and effective classification reference set. 

Several agencies and programs collect land cover data appropriate for use as classification reference data for a Minnesota land cover classification.  These data sets come from the following agencies:  (1) Minnesota Department of Natural Resources - Forest Inventory and Management Program (FIM); (2) US Census Bureau - 2000 Blocks Population; (3) DLG Hydrography lake and wetland polygons; and (4) USDA Farm Services Agency - National Agricultural Imagery Program (NAIP)photography.  With a bit of sorting and manipulation the reference data required for classification can be developed from these datasets.

The FIM inventory data is another source of forest reference data.  This data is obtained by the Minnesota Department of Natural Resources (MNDNR) Division of Forestry to produce a set of digital forest stand data.  The data was originally photo-interpreted, however, since then many stands have been field measured.  The data contains the following attributes for each stand: cover type, cover size (DBH), stocking (BA), volume, age, measure year, site index, and how the stand data was collected (ground or photo).  For 2000 Minnesota classification, stands with the following attributes were selected for reference sites: (1) cover type forested (values 1-22), (2) measured year = 1998 and = 2002, and (3) data was ground collected (value of 1).

The DLG Hydrography lake and wetland polygons provide lake, stream, and wetland reference data.  This data is produced by the MNDNR and consists of 1:100,000 scale hydrography derived from USGS DLG's of the same scale.  DLG data are automated from the most recent USGS sources available.  The purpose of this data set is regional hydrographic analysis, medium scale base mapping, and limnological studies. This data identifies and attributes hydrological features such as lakes, wetlands, inundated areas, tailings ponds, sewage ponds, fish hatcheries, and other minor water body types. Attributes include: hydrologic feature type, field validation, lake name, and lake class.  This data set was utilized to identify lake and wetland reference sites for the 2000 Minnesota classification.  Lake reference sites were selected with the following criteria: (1) hydrologic feature type a lake or pond (value of 421) and (2) field validated (value of 1).  Wetland reference sites were selected were those identified with a hydrologic feature type of a marsh, wetland, swamp, or bog (value of 111).

The 2000 Block Population data set maps areas across the state of varying levels of urban intensity.  This data set consists of areas were population data was collected and tabulated for the 2000 census.  The block boundaries are physical features, such as streets, highways, rivers, lakes, pipelines, and power lines; and political boundaries, such as counties, cities, and towns.  These data have a multitude of attributes including block identifier, tract identifier, population for 1990, 1999, and 2000, and area.  For the purposes of the 2000 Minnesota classification two additional fields were added: acres and population density.  The acres field is the area of the block calculated in acres.  Population density field is the number of people per acre and was calculated for each block by dividing the 2000 population by the block size in acres.  Urbanized areas were selected at two different intensity levels, high intensity and low intensity.  Low intensity urban areas were census blocks with = 8 and = 16 people per acre.  High intensity urban areas were census blocks with > 16 people per acre.

Reference data for agricultural cover types came from photo interpretation of NAIP Digital Ortho Photos (DOQ) of Minnesota taken during the summer of 2002 and 2003.  Selection of agricultural areas was done with the following process.  A previous statewide land cover map completed for 1991 by the Minnesota DNR was used to identify agricultural areas statewide.  These areas were randomly sampled to produce 800 potential agricultural reference sites.  Each randomly selected site was overlaid on the NAIP photo's, verified as agriculture, and then its boundaries manually digitized.  The digitized polygon was then overlaid on the three dates of Landsat imagery for further identification of the agricultural crop existing at the time of image acquisition.  This was done to ensure that all agricultural types were present in the reference data.

Shrubland cover types data was available from the MN DNR FIM data set.  Stands labeled in the FIM dataset with a covertype code of Upland Brushland or Lowland Brushland were utilized as shrubland reference sites.

Once the reference data set was collected, the polygon data was adjusted to make it suitable for use with Landsat imagery and for use in the kNN classifier.  First, to remove the effects of mixed pixels on the signatures, each of the selected polygons was buffered internally by 30-meters.  Second, to ensure that the area of the polygon consisted of > 4 pixels, all potential sites < 3-acres (1.215-hectares) were removed.  Third, areas where an overabundance of reference sites existed, such as urban reference sites in the Minneapolis/St. Paul metropolitan area, an appropriate number of sites were random selected.  Lastly, since the kNN classifier requires point data, centroid points were generated within each of the selected polygons.  These points were then visually referrenced against the 2003 NAIP photograph to check for discrepencies.  This produced the final classification reference data set named "mn2000_reference_sites_randomly_selected_points_final.shp".


Stratified Classification Units

The area of Minnesota was stratified in sub-regions or spectrally consistent classification units (SCCU) of similar biophysical and spectral characteristics.   In earlier regional classification of Minnesota, Bauer et al. (1994) found a significant increase in classification accuracy by stratifying images by physiographic regions.  This procedure was also used by Lillesand et al. (1998) for GAP classifications in the Upper Midwest Region.  The SCCU's for the 2000 Minnesota classification were based on two factors: (1) ecoregion section boundaries, similar to the physiographic regions used by Bauer et al., from the Ecoregion Subsections of Minnesota data provided by the MNDNR, and (2) image boundaries.  These two data sets were combined and modified to delineate areas of uniform appearance, particularly with respect to phenology and atmospheric effects.  20 SCCU's were used for the 2000 Minnesota classification.


k-Nearest Neighbor_Classification

The k-Nearest Neighbor classifier was used to perform the statewide classification.  Although, the ML classifier is a more common approach, several attributes of the kNN classifier provide advantages over the ML approach that are beneficial for use with large areas image classifications such as the 2000 Minnesota classification.  Among these attributes is that the kNN algorithm is non-parametric.  Beyond the basic assumption that training data is representative of the image being classified, kNN does not rely on underlying statistical assumptions.  Further, because of this non-parametric quality, kNN can perform classification using a relatively small set of reference observations.  Lastly, the set of reference observations collected for kNN classification can also be used simultaneously for accuracy assessment via leave-one-out cross-validation reducing the total number of reference sites necessary for both training and testing. (Gong 1986)

In brief, the kNN classifier assigns each unknown (target) pixel the field attributes of the most similar training record(s) (Franco-Lopez et al., 2001).  Several different distance metrics can be utilized with kNN for assessing this similarity between target pixels and training records.  For the 2000 Minnesota classification, the Euclidean distance metric was chosen to measure the degree of similarity in feature space (Franco-Lopez et al., 2001 and Sohn, et al., 1999).  Additionally the kNN classifier has the option of using different values of k.  Previous studies show that varying k will affect classification accuracy (Haapanen et al., In press; McRoberts et al., 2002), however, the there is no universally correct k.  The optimal k-value is specific to each classification and as a result, it is necessary to explore a variety of values.  Trials indicated that a k =7 was most suitable for the 2000 Minnesota classification. 

The image classification process was stratified by SCCU.  During classification both the pixels available for classification and reference signatures used for classification were limited by the boundary of the SCCU's.  Another benefit of the kNN classifier was realized here, since the program can be adjusted to separate the areas and reference points by SCCU with an automated process.  This saved us from having to manually separate the reference sites and imagery into 20 separate classification data sets, one for each SCCU.


Post Classification Processing

Cloud and cloud shadow removal was performed in the following manner.   First, clouds and cloud shadows were identified on all imagery by visual inspection of the landsat imagery and then  polygons were digitized around the clouds and shadows and the date of the cloud noted.  Second, three separate statewide level one landcover classifications were performed for each two date combination of landsat imagery (i.e summer /fall, spring/fall and spring/summer).  Third, the area identified as a cloud was removed from the three date land covertype classification and replaced with the land covertype classification for the appropriate cloud free two date classification.  Example: cloud in spring is replaced with summer/fall two date classification.


Majority filtering was performed following the cloud and cloud shadow replacement proceedure.  This was done using a 3 pixel by 3 pixel majority filter.  The purpose was to reduce the "salt and pepper" effect and create more contiguous covertypes.

Road overlay was the final post processing proceedure performed.  Federal, state, county, and township roads layers (in vector line format) were aquired from the Minnesota Department of Transprotation and overlayed on the landcover classification.  Pixels that intersected with  any of the roads were reclassified as the Urban covertype class.

Manual adjustment of missclassified areas was performed using high resolution color photography (NAIP03).  The covertype map was overlayed on the high resolution photography and missclassified areas identified.  Missclassified areas were digitized with polygons representing the true covertype.  This process was performed in patches throughout the state where missclassification was identified for specific covertype classes.  A spatial model was used to reclass the pixels in the covertype map that fell within the areas identified by the polygons as missclassified.

Seven level one land cover classes including:

1. Urban/Developed

2. Agriculture

3. Grassland

4. Forest

5. Water

6. Wetland

7. Shrubland

Seven level one land cover classes are listed along with a detailed description below:

1. Urban/Developed -  

An area containing any amount of impervious cover of man-made solid materials or compacted soils including areas with interspersed vegetation.  Examples: parking lots, shopping malls, warehouses, industrail parks, highways, sparse development, single family residential developments, single lane roads, and mines.


2. Agriculture -

An area where the primary cover type during the growing season is an agricultural covertype including row crops, forage crops and small grains.  Examples: corn, soybeans, alfalfa, oats, wheat and barley.

3. Grassland - 

An upland area covered by cultivated or non-cultivated herbaceous vegetation predominated by grasses, grass-like plants and forbs.  Includes non-agricultural upland vegetation dominated by short manicured grasses and forbs as well as non-cultivated herbaceous upland vegetation dominated by native grasses and forbs.  Examples: golf courses, lawns, athletic fields, dry priaries and pastures.

4. Forest -

An upland area of land covered with woody perennial plants, the tree reaching a mature height of at least 6 feet tall with a definite crown.  To be considered a forested cover type the stand must have a combined species minimum of 3 cords/acre or 1,251 bdft/acre or 251 stems per acre depending on size class (MNCSA Standards) Note: all forest  training sites were obtained from the MNDNR Forest Inventory and Management (FIM) dataset and thus an effort was made to match the cover type descriptions between the two data sets within the limitations of remote sensing capabilities.  Examples: white pine, red pine, oak, mixed conifer and mixed deciduous.

5. Water -

An area of open water with none or very little above surface vegetaton.  Example:lakes, streams, rivers and open wetlands.

6. Wetland -

A lowland area with a cover of persistent and non-persistent herbaceous plants standing above the surface of wet soil or water.  Examples: cattails, march grass, sedges and peat.

7. Shrubland -

An upland or lowland area with vegetation that has a persistent woody stem, generally with several basal shoots, low growth of less than 20 feet in height. Area has less than 251 stems per acre of commercial tree species, the shrub species are fairly uniformly distributed throughout and the density of the coverage is moderate to high. (Examples: alder, willow, buckthorn, hazel, sumac, and scrub oak)  Note: all shrubland  training sites were obtained from the MNDNR Forest Inventory (CSA) and thus an effort was made to match the cover type descriptions between the two data sets within the limitations of remote sensing capabilities.

This data may be used for educational and non-commercial purposes, provided proper attribution is given. Secondary distribution of the data is permitted, but not supported by the University of Minnesota. By accepting the data, the user agrees not to transmit this data or provide access to it or any part of it to another party unless the user includes with the data a copy of this disclaimer.

see website or contact info