Module 9 Lesson: Exploratory Data Analysis and Data Quality

Let us begin

Exploratory data analysis, or EDA, is how we get to know a dataset.

It is not random clicking. It is disciplined curiosity.

We ask:

• What is typical?
• What is unusual?
• What is missing?
• What is changing?
• What is related?
• What differs by group?
• What might be unreliable?

EDA helps us decide what is worth explaining later.

Start with the question

Question:

> Which course activities are connected with steady learner progress?

EDA plan:

1. Check data quality.
2. Inspect activity distributions.
3. Compare completed and paused enrolments.
4. Check weekly patterns.
5. Write careful observations.

The question gives structure.

Profile the data

Useful checks:

df.shape
df.dtypes
df.isna().sum()
df.duplicated().sum()
df.describe()

For categories:

df["status"].value_counts(dropna=False)

These checks tell you whether the data is ready for deeper exploration.

Inspect distributions

A distribution shows how values are spread.

Questions:

• What values are common?
• Are there extreme values?
• Are there many zeros?
• Is the data skewed?

Code:

df["minutes_spent"].describe()
df["minutes_spent"].quantile([0.25, 0.5, 0.75, 0.9, 0.99])

A histogram will come later in more detail. For now, the important habit is to inspect the spread before using averages alone.

Inspect relationships

Relationship questions compare two values.

Example:

> Do learners with more quiz attempts also tend to have more practice minutes?

Use summaries first:

df[["quiz_attempts", "practice_minutes"]].corr()

Correlation is only a clue. It does not prove cause.

Compare segments

Segment comparison looks across groups.

segment_summary = (
    df
    .groupby("status")
    .agg(
        learner_count=("learner_id", "nunique"),
        avg_minutes=("practice_minutes", "mean"),
        median_minutes=("practice_minutes", "median"),
    )
    .reset_index()
)

Use both average and median when outliers may matter.

Missingness patterns

Missing values may not be random.

Ask:

• Which columns are missing together?
• Does missingness differ by group?
• Is missing feedback more common for paused learners?

Example:

df["missing_rating"] = df["rating"].isna()

df.groupby("status")["missing_rating"].mean()

If missingness differs by group, final conclusions need caution.

Outliers

Outliers can be errors or meaningful cases.

Do not delete them automatically.

Investigate:

df.sort_values("practice_minutes", ascending=False).head(10)

Ask whether the values are possible, duplicated, or created by a tracking issue.

Writing findings

A good EDA finding has three parts:

1. Evidence.
2. Interpretation.
3. Limit.

Example:

> Completed enrolments had a higher median practice time than paused enrolments in this sample. This suggests practice time is a useful signal to inspect. It does not prove practice time caused completion, because motivation, schedule, and prior skill are not fully measured.

Takeaway

EDA is where we learn what the data can support. In the next module, we begin formal chart construction with Matplotlib.