Explain in detail how dirty data can be detected

Dirty data refers to inaccurate, incomplete, inconsistent, or duplicate data within a dataset. It can arise from various sources such as human error during data entry, system glitches, outdated information, or issues during data migration or integration processes. Dirty data can significantly impact the quality and reliability of analyses, as it may lead to erroneous conclusions or decisions if not properly identified and corrected. Cleaning and preprocessing techniques are often employed to address dirty data before it is used for analysis or other purposes.

Visualizations during this phase can act as powerful tools to detect these inconsistencies.

1. Identifying Missing Values:

• Bar Charts: Simple bar charts can reveal columns with a high percentage of missing entries. Unexpectedly low bars compared to others might indicate missing data.
• Heatmaps: Heatmaps show the correlation between variables. Blank squares or entire rows/columns with no color can highlight missing values and potential data collection issues.

2. Spotting Outliers:

• Scatter Plots: Scatter plots show the relationship between two variables. Points far away from the main cluster might be outliers caused by typos or errors during data entry.
• Boxplots: Boxplots visualize the distribution of data. Points outside the whiskers (upper and lower bounds) could be outliers that need investigation.

3. Detecting Inconsistency:

• Histograms: Histograms show the frequency distribution of a single variable. Unexpected gaps, multiple peaks, or skewed distributions could indicate inconsistencies in data formatting or measurement units.
• Time Series Plots: When dealing with time-based data, sudden jumps or drops in a time series plot can signify missing entries or inconsistencies in data collection frequency.

4. Highlighting Duplicates:

• Frequency Tables: Creating frequency tables for categorical variables can reveal unexpected high frequencies for specific entries. This might indicate duplicate records.
• Cluster Analysis: Techniques like k-means clustering can group similar data points. If a cluster seems suspiciously large, it could contain duplicates.