Differences

This shows you the differences between two versions of the page.

--- sift:principal_component_analysis:outlier_detection_for_pca [2024/08/28 18:57] – wikisysop
+++ sift:principal_component_analysis:outlier_detection_for_pca [2024/08/28 19:04] (current) – [Reference] wikisysop
@@ Line 7: / Line 7: @@
 ==== Local Outlier Factor ====
-Local Outlier Factor (LOF) is an unsupervised method of finding outliers through a data points local density, introduced in 2000 by Breunig et al. [[[https://www.dbs.ifi.lmu.de/Publikationen/Papers/LOF.pdf|[1]]]]. It compares the local density of each point to its local neighbours, and calculates a Local Outlier Factor as the average ratio between its own density and the neighbours. By doing so, it can find local outliers that a global method might not find.
+Local Outlier Factor (LOF) is an unsupervised method of finding outliers through a data points local density, introduced in 2000 by Breunig et al. [[https://www.dbs.ifi.lmu.de/Publikationen/Papers/LOF.pdf|[1]]]. It compares the local density of each point to its local neighbours, and calculates a Local Outlier Factor as the average ratio between its own density and the neighbours. By doing so, it can find local outliers that a global method might not find.
 {{:Sift_LOF_example.png}}
@@ Line 62: / Line 62: @@
 Where X<sup>2</sup>(chi<sup>2</sup>) is the chi-square distribution with n dimensions, at significance level alpha.
-If we successfully reject the null hypothesis, we remove the outlier from the data, and calculate the statistic again on until we have calculated it on each data point. We can recalculate the new covariance, and continue this until no outliers are detected, or stop after X iterations have occurred (X being up to the user).
+If we successfully reject the null hypothesis, we remove the outlier from the data, and calculate the statistic again or until we have calculated it on each data point. We can recalculate the new covariance, and continue this until no outliers are detected, or stop after X iterations have occurred (X being up to the user).
 ==== Squared Prediction Error (SPE) ====
@@ Line 86: / Line 86: @@
 {{:Sift_Mahalanobis_noncentral_chi_square.png}}
+[[https://hrcak.srce.hr/file/117623|[2]]]
+If we successfully reject the null hypothesis, we remove the outlier from the data, and calculate the statistic again or until we have calculated it on each data point. We can recalculate the new covariance, and continue this until no outliers are detected, or stop after X iterations have occurred (X being up to the user).
-......................
+===== Reference =====
+Markus M. Breunig, Hans-Peter Kriegel, Raymond T. Ng, and Jörg Sander. 2000. LOF: identifying density-based local outliers. SIGMOD Rec. 29, 2 (June 2000), 93–104. https://doi.org/10.1145/335191.335388
-Where X<sup>2</sup>(chi<sup>2</sup>) is the chi-square distribution with n dimensions, at significance level alpha.
+**Abstract**
-If we successfully reject the null hypothesis, we remove the outlier from the data, and calculate the statistic again on until we have calculated it on each data point. We can recalculate the new covariance, and continue this until no outliers are detected, or stop after X iterations have occurred (X being up to the user).
+For many KDD applications, such as detecting criminal activities in E-commerce, finding the rare instances or the outliers, can be more interesting than finding the common patterns. Existing work in outlier detection regards being an outlier as a binary property. In this paper, we contend that for many scenarios, it is more meaningful to assign to each object a degree of being an outlier. This degree is called the local outlier factor (LOF) of an object. It is local in that the degree depends on how isolated the object is with respect to the surrounding neighborhood. We give a detailed formal analysis showing that LOF enjoys many desirable properties. Using real-world datasets, we demonstrate that LOF can be used to find outliers which appear to be meaningful, but can otherwise not be identified with existing approaches. Finally, a careful performance evaluation of our algorithm confirms we show that our approach of finding local outliers can be practical.
-===== Reference =====
+----
+Slišković, Dražen, Ratko Grbić, and Željko Hocenski. "Multivariate statistical process monitoring." Tehnički vjesnik 19.1 (2012): 33-41.
-Markus M. Breunig, Hans-Peter Kriegel, Raymond T. Ng, and Jörg Sander. 2000. LOF: identifying density-based local outliers. SIGMOD Rec. 29, 2 (June 2000), 93–104. https://doi.org/10.1145/335191.335388
 **Abstract**
-For many KDD applications, such as detecting criminal activities in E-commerce, finding the rare instances or the outliers, can be more interesting than finding the common patterns. Existing work in outlier detection regards being an outlier as a binary property. In this paper, we contend that for many scenarios, it is more meaningful to assign to each object a degree of being an outlier. This degree is called the local outlier factor (LOF) of an object. It is local in that the degree depends on how isolated the object is with respect to the surrounding neighborhood. We give a detailed formal analysis showing that LOF enjoys many desirable properties. Using real-world datasets, we demonstrate that LOF can be used to find outliers which appear to be meaningful, but can otherwise not be identified with existing approaches. Finally, a careful performance evaluation of our algorithm confirms we show that our approach of finding local outliers can be practical.
+Demands regarding production efficiency, product quality, safety levels and environment protection are continuously increasing in the process industry. The way to accomplish these demands is to introduce ever more complex automatic control systems which require more process variables to be measured and more advanced measurement systems. Quality and reliable measurements of process variables are the basis for the quality process control. Process equipment failures can significantly deteriorate production process and even cause production outage, resulting in high additional costs. This paper analyzes automatic fault detection and identification of process measurement equipment, i.e. sensors. Different statistical methods can be used for this purpose in a way that continuously acquired measurements are analyzed by these methods. In this paper, PCA and ICA methods are used for relationship modelling which exists between process variables while Hotelling's ( ), and (SPE) statistics are used for fault detection because they provide an indication of unusual variability within and outside normal process workspace. Contribution plots are used for fault identification. The algorithms for the statistical process monitoring based on PCA and ICA methods are derived and applied to the two processes of different complexity.Apart from that, their fault detection ability is mutually compared.