HAS-Motion Software Documentation

User Tools

Site Tools

You are here: Main Page » sift » Principal Component Analysis » Outlier Detection for PCA
sift:principal_component_analysis:outlier_detection_for_pca

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
sift:principal_component_analysis:outlier_detection_for_pca [2024/08/28 18:46] – [Squared Prediction Error (SPE)] wikisysopsift:principal_component_analysis:outlier_detection_for_pca [2024/08/28 19:04] (current) – [Reference] wikisysop
Line 7: Line 7:
 ==== Local Outlier Factor ==== ==== Local Outlier Factor ====
  
-Local Outlier Factor (LOF) is an unsupervised method of finding outliers through a data points local density, introduced in 2000 by Breunig et al. [[[https://www.dbs.ifi.lmu.de/Publikationen/Papers/LOF.pdf|[1]]]]. It compares the local density of each point to its local neighbours, and calculates a Local Outlier Factor as the average ratio between its own density and the neighbours. By doing so, it can find local outliers that a global method might not find.+Local Outlier Factor (LOF) is an unsupervised method of finding outliers through a data points local density, introduced in 2000 by Breunig et al. [[https://www.dbs.ifi.lmu.de/Publikationen/Papers/LOF.pdf|[1]]]. It compares the local density of each point to its local neighbours, and calculates a Local Outlier Factor as the average ratio between its own density and the neighbours. By doing so, it can find local outliers that a global method might not find.
  
 {{:Sift_LOF_example.png}} {{:Sift_LOF_example.png}}
Line 54: Line 54:
 Where X is the vector difference of the point from the distribution mean, and S^-1 is the inverse of the variance-covariance matrix of the distribution. Where X is the vector difference of the point from the distribution mean, and S^-1 is the inverse of the variance-covariance matrix of the distribution.
  
-The square of the Mahalanobis Distance (d^2) follows a chi-square distribution, and as such we can use this as a test statistic: The Mahalanobis Distance Test.+The square of the Mahalanobis Distance (d<sup>2</sup>) follows a chi-square distribution, and as such we can use this as a test statistic: The Mahalanobis Distance Test.
  
 We produce this statistic for each data point being observed, and choose the null hypothesis to be that this point was drawn from the specified multi-variate normal distribution. We can reject this null hypothesis (at significance level alpha) if (i.e. the point is an outlier): We produce this statistic for each data point being observed, and choose the null hypothesis to be that this point was drawn from the specified multi-variate normal distribution. We can reject this null hypothesis (at significance level alpha) if (i.e. the point is an outlier):
Line 60: Line 60:
 {{:Sift_Mahalanobis_chi_square.png}} {{:Sift_Mahalanobis_chi_square.png}}
  
-Where X^2(chi^2) is the chi-square distribution with n dimensions, at significance level alpha.+Where X<sup>2</sup>(chi<sup>2</sup>) is the chi-square distribution with n dimensions, at significance level alpha.
  
-If we successfully reject the null hypothesis, we remove the outlier from the data, and calculate the statistic again on until we have calculated it on each data point. We can recalculate the new covariance, and continue this until no outliers are detected, or stop after X iterations have occurred (X being up to the user).+If we successfully reject the null hypothesis, we remove the outlier from the data, and calculate the statistic again or until we have calculated it on each data point. We can recalculate the new covariance, and continue this until no outliers are detected, or stop after X iterations have occurred (X being up to the user).
  
 ==== Squared Prediction Error (SPE) ==== ==== Squared Prediction Error (SPE) ====
Line 78: Line 78:
 Where Q represents the SPE, r<sub>i</sub> is the ith residual value (between the projection and the sample), x<sub>i</sub> is the sample, I is the identity matrix, and P<sub>l</sub> is the projection matrix onto l principal components. Where Q represents the SPE, r<sub>i</sub> is the ith residual value (between the projection and the sample), x<sub>i</sub> is the sample, I is the identity matrix, and P<sub>l</sub> is the projection matrix onto l principal components.
  
-The SPE is a direct+The SPE complements the Mahalanobis Distance, and is often used in conjunction with it to determine the outliers. 
 + 
 +The SPE follows a non-central chi-square distribution, and as such we can use this as a test statistic: The SPE Test. 
 + 
 +We produce this statistic for each data point being observed, and choose the null hypothesis to be that this point was drawn from the specified non-central chi-square distribution. We can reject this null hypothesis (at significance level alpha) if Q is greater than Q<sub>a</sub>(i.e. the point is an outlier): 
 + 
 +{{:Sift_Mahalanobis_noncentral_chi_square.png}} 
 + 
 +[[https://hrcak.srce.hr/file/117623|[2]]] 
 + 
 +If we successfully reject the null hypothesis, we remove the outlier from the data, and calculate the statistic again or until we have calculated it on each data point. We can recalculate the new covariance, and continue this until no outliers are detected, or stop after X iterations have occurred (X being up to the user).
  
 ===== Reference ===== ===== Reference =====
  
 Markus M. Breunig, Hans-Peter Kriegel, Raymond T. Ng, and Jörg Sander. 2000. LOF: identifying density-based local outliers. SIGMOD Rec. 29, 2 (June 2000), 93–104. https://doi.org/10.1145/335191.335388 Markus M. Breunig, Hans-Peter Kriegel, Raymond T. Ng, and Jörg Sander. 2000. LOF: identifying density-based local outliers. SIGMOD Rec. 29, 2 (June 2000), 93–104. https://doi.org/10.1145/335191.335388
 +
 **Abstract** **Abstract**
 +
 For many KDD applications, such as detecting criminal activities in E-commerce, finding the rare instances or the outliers, can be more interesting than finding the common patterns. Existing work in outlier detection regards being an outlier as a binary property. In this paper, we contend that for many scenarios, it is more meaningful to assign to each object a degree of being an outlier. This degree is called the local outlier factor (LOF) of an object. It is local in that the degree depends on how isolated the object is with respect to the surrounding neighborhood. We give a detailed formal analysis showing that LOF enjoys many desirable properties. Using real-world datasets, we demonstrate that LOF can be used to find outliers which appear to be meaningful, but can otherwise not be identified with existing approaches. Finally, a careful performance evaluation of our algorithm confirms we show that our approach of finding local outliers can be practical. For many KDD applications, such as detecting criminal activities in E-commerce, finding the rare instances or the outliers, can be more interesting than finding the common patterns. Existing work in outlier detection regards being an outlier as a binary property. In this paper, we contend that for many scenarios, it is more meaningful to assign to each object a degree of being an outlier. This degree is called the local outlier factor (LOF) of an object. It is local in that the degree depends on how isolated the object is with respect to the surrounding neighborhood. We give a detailed formal analysis showing that LOF enjoys many desirable properties. Using real-world datasets, we demonstrate that LOF can be used to find outliers which appear to be meaningful, but can otherwise not be identified with existing approaches. Finally, a careful performance evaluation of our algorithm confirms we show that our approach of finding local outliers can be practical.
 +
 +----
 +
 +Slišković, Dražen, Ratko Grbić, and Željko Hocenski. "Multivariate statistical process monitoring." Tehnički vjesnik 19.1 (2012): 33-41.
 +
 +**Abstract**
 +
 +Demands regarding production efficiency, product quality, safety levels and environment protection are continuously increasing in the process industry. The way to accomplish these demands is to introduce ever more complex automatic control systems which require more process variables to be measured and more advanced measurement systems. Quality and reliable measurements of process variables are the basis for the quality process control. Process equipment failures can significantly deteriorate production process and even cause production outage, resulting in high additional costs. This paper analyzes automatic fault detection and identification of process measurement equipment, i.e. sensors. Different statistical methods can be used for this purpose in a way that continuously acquired measurements are analyzed by these methods. In this paper, PCA and ICA methods are used for relationship modelling which exists between process variables while Hotelling's ( ), and (SPE) statistics are used for fault detection because they provide an indication of unusual variability within and outside normal process workspace. Contribution plots are used for fault identification. The algorithms for the statistical process monitoring based on PCA and ICA methods are derived and applied to the two processes of different complexity.Apart from that, their fault detection ability is mutually compared.
 +
 +
  
  
sift/principal_component_analysis/outlier_detection_for_pca.1724870772.txt.gz · Last modified: 2024/08/28 18:46 by wikisysop