# Detect and fix data issues

> For the complete documentation index, see [llms.txt](https://www.jetbrains.com/help/idea/2026.3/llms.txt).

IntelliJ IDEA can automatically analyze datasets loaded into a pandas DataFrame and displayed in a Jupyter Notebook cell. The IDE highlights potential problems such as missing values, outliers, duplicate rows, duplicate column names, or constant columns — making it easier to clean your data before modeling.

IntelliJ IDEA automatically analyzes any DataFrame that meets the following conditions:

* Contains not more than 100,000 rows.

* Contains not more than 200 columns.

* Is displayed using `df`, `df.head()`, `df[x:y]`, `df.loc(.. .)`, or `df.iloc(...)`.

If at least one issue is found, the ![](https://resources.jetbrains.com/help/img/idea/2026.3/app.general.inspectionsWarningIcon.svg) Dataset Issues button appears on the DataFrame toolbar.

## Fix data issues with AI

Procedure:

1. [Install and enable the
AI Assistant plugin](https://www.jetbrains.com/help/ai-assistant/installation-guide-ai-assistant.html).

2. Click the ![](https://resources.jetbrains.com/help/img/idea/2026.3/app.general.inspectionsWarningIcon.svg) Dataset Issues button in the DataFrame toolbar. The Dataset Quality Issues dialog opens.

![Issues button](https://resources.jetbrains.com/help/img/idea/2026.3/py_data_issues_button.png)

3. Select the issues that you want to fix and click the Fix with AI button.

![Dataset Quality Issues dialog](https://resources.jetbrains.com/help/img/idea/2026.3/py_data_issues_dialog.png)

A new code cell will be created below the DataFrame with the suggested changes.

> **Note:**
> This cell is not executed automatically. You can review and run it manually.

## Common issues and how to handle them

Procedure: Missing values

Some cells in your dataset may have no value, represented as `NaN` or `None`. This often happens during data collection, merging, or file import. If ignored, missing values can introduce bias into your analysis. Certain machine learning algorithms also cannot run on data that contain them.

For example, six empty cells were found in the `salary` column:

![Missing Values](https://resources.jetbrains.com/help/img/idea/2026.3/py_data_issues_missing_values.png)

> **Note:**
> You can check which columns contain missing data, using `df.isna().sum()`.

Typical fixes to handle this issue include:

* Removing rows or columns that have too many unknown values: ```PYTHON df = df.dropna() ```

* Replacing unknown values with a constant, such as `0` or `Unknown`: ```PYTHON df['Name'] = df['Name'].fillna('Unknown') ```

* Replacing unknown values with the mean, median, or mode of the column: ```PYTHON df = df.fillna(df.mean()) ```

* Predicting and replacing the missing values using a machine learning model, such as linear regression or k-nearest neighbors: ```PYTHON from sklearn.impute import KNNImputer imputer = KNNImputer(n_neighbors=5) df_imputed = pd.DataFrame(imputer.fit_transform(df), columns=df.columns) ```

> **Tip:**
> It is important to understand the reason for missing values and how it may affect your data. For more details, refer to [this article](https://blog.jetbrains.com/pycharm/2025/01/data-cleaning-in-data-science/#missing-values).
>
>
>
> Each replacement method also has its own advantages and disadvantages and should be chosen in the context of your dataset.

Procedure: Duplicate column names

A dataset may contain multiple columns with the same name, for example, after assigning column names manually or combining datasets. Duplicate column names can make column choice ambiguous and may lead to unexpected results during data processing.

For example, two columns with duplicate names were found in the dataset:

![Duplicate Column Names](https://resources.jetbrains.com/help/img/idea/2026.3/py_data_issues_duplicate_column_names.png)

> **Note:**
> You can identify the duplicates, using `df.columns[df.columns.duplicated()]`.

Typical fixes to handle this issue include:

* Renaming duplicate columns

* Removing duplicate columns if they contain redundant data: ```PYTHON df = df.loc[:, ~df.columns.duplicated()] ```

> **Tip:**
> Before removing or renaming duplicate columns, check their contents to make sure that no meaningful data is lost.

Procedure: Duplicate rows

Some rows in your dataset may be identical, for example, due to repeated imports or logging errors. If ignored, they can create bias and make statistical results unreliable.

For example, two duplicate rows were found in the dataset:

![Duplicate Rows](https://resources.jetbrains.com/help/img/idea/2026.3/py_data_issues_duplicate_rows.png)

> **Note:**
> You can identify the duplicates, using `df[df.duplicated()]`.

Typical fixes to handle this issue include:

* Removing all duplicate rows: ```PYTHON df = df.drop_duplicates() ```

* Removing all duplicates, except the first occurrence: ```PYTHON df = df.drop_duplicates(keep='first') ```

> **Tip:**
> If duplicates represent valid repeated events (for example, identical purchases), consider keeping them instead of removing.

Procedure: Outliers

Outliers are data points that differ significantly from most other values in the dataset. They can distort statistical results and affect machine learning models.

PyCharm detects outliers using the interquartile range (IQR) method. A value is considered an outlier if it falls below `Q1 - 1.5 × IQR` or above `Q3 + 1.5 × IQR`, where `Q1` and `Q3` are the first and third quartiles.

It is generally recommended to only remove outliers if you believe they belong to a different population from the one you’re attempting to target. See [this article](https://blog.jetbrains.com/pycharm/2025/01/data-cleaning-in-data-science/#addressing-outliers) for more details.

For example, outliers were found in the `salary` and `age` columns:

![Outliers](https://resources.jetbrains.com/help/img/idea/2026.3/py_data_issues_outliers.png)

Typical fixes to handle this issue include:

* Discovering extreme values using a statistical method, such as IQR: ```PYTHON Q1 = df["salary"].quantile(0.25) Q3 = df["salary"].quantile(0.75) IQR = Q3 - Q1 outliers = df[ (df["salary"] < Q1 - 1.5 * IQR) | (df["salary"] > Q3 + 1.5 * IQR) ] ```

* Removing rows with extreme values: ```PYTHON df = df[df['salary'] < 20000] ```

* Capping extreme values to a threshold: ```PYTHON df['salary'] = np.where(df['salary'] > 20000, 20000, df['salary']) ```

* Transforming values to reduce the effect of outliers (for example, by applying a logarithmic transformation): ```PYTHON df['salary_log'] = np.log1p(df['salary']) ```

> **Tip:**
> Visualize data distributions, using `df.boxplot()` or `df['column'].hist()` before and after removing or transforming outliers to confirm that the distributions of your data look suitable for your planned analysis or model.

Procedure: Constant columns

Some columns in your dataset may contain the same value in all the rows. Such constant columns often do not add any useful information to your analysis or models because they have no variability.

For example, the column `country` with the constant value `USA` was found:

![Constant Columns](https://resources.jetbrains.com/help/img/idea/2026.3/py_data_issues_constant_columns.png)

> **Note:**
> You can identify single-value columns by checking how many unique values each column contains:
>
>
>
>
> ```PYTHON
> single_value_cols = [col for col in df.columns if df[col].nunique() == 1]
> print(single_value_cols)
> ```

A typical fix to handle this issue is:

* Removing all single-value columns: ```PYTHON df = df.drop(columns=single_value_cols) ```

> **Tip:**
> Consider keeping the constant columns if they provide meaningful context.