Outliers in a dataset are observations that differ considerably from the rest of the data. Outliers must be identified and handled in any data science project since they can have a substantial influence on numerous statistical approaches, such as means, standard deviations, and so on, as well as the performance of ML models. Outliers can occasionally reveal data flaws or anomalies. What are outliers? Outliers are observations or data points in statistics that differ considerably from the rest of the observations or data points in a dataset. Outliers are unusually high or low values in a feature or dataset. For instance, suppose you have a dataset with a feature height. The bulk of the values in this feature are between 4.5-6.5 feet, however one is 10 feet. This figure is an outlier since it is not just an extreme value but also an unachievable height. Other names of Outliers: Outliers are also known as aberrations, anomalous points, abnormalities, and so on. Outliers in a dataset must be detected and handled carefully since they can have a substantial influence on numerous statistical approaches, such as mean, variance, and so on, as well as the performance of ML models. If not appropriately accounted for, it can lead to misleading, inconsistent, and erroneous outcomes. Types of Outliers Univariate outliers: These are extreme values in one variable. For instance, in a distribution of ages, a value like 200 would be an outlier. Multivariate outliers: These are a combination of values in several variables. For example, in a dataset of height and weight, a combination of 5 feet height and 200 kg weight would be an outlier. Causes of Outliers Data Entry Errors: Human errors such as errors caused during data collection, recording, or entry can cause outliers in data. Measurement Error: It can be a result of faulty equipment or the result of experimenter error. Experimental Error: For example, in a controlled environment, an unforeseen factor might disrupt an experiment leading to anomalous results. Intentional Outlier: These are sometimes introduced to test detection methods. Sampling Errors: For instance, during sample collection or extraction, certain unusual samples might be picked. Natural Outlier: They don't necessarily represent any anomaly. For instance, in a class of students, one student may genuinely be extraordinarily tall or short. Detecting Outliers Visualization tools: Box plots, scatter plots, and histograms can be used to spot outliers. Statistical Tests: The Z-score or IQR (Interquartile Range) and Percentile Methods can be used to identify outliers. Machine Learning algorithms: There are algorithms like DBSCAN and Isolation Forest that can be used to detect outliers. Handling Outliers Removing the outlier: This is the most common method where all detected outliers are removed from the dataset. Transforming and binning values: Outliers can be transformed to bring them within a range. Techniques like log transformation or square root transformation can be used. Imputation: Outliers can also be replaced with mean, median, or mode values. Separate treatment: In some use-cases, it's beneficial to treat outliers separately rather than removing or imputing them. Robus Statistical Methods: Some of the statistical methods to analyze and model the data are less sensitive to outliers and provide more accurate results in the data. Conclusion Outliers in a dataset are observations that deviate dramatically from the rest of the data points. They might arise as a result of data gathering mistakes or abnormalities, or they can be real findings that are just infrequent or extraordinary. If outliers are not appropriately accounted for, they might produce misleading, inconsistent, and erroneous findings. As a result, identifying and dealing with outliers is critical in order to produce accurate and useful data analysis findings. Outliers may be detected using a variety of methods, including the percentile approach, IQR method, and z-score method. Outliers can be dealt with in a variety of methods, including removal, transformation, imputation, and so on.
