Module 4 Lesson: Selecting, Filtering, Sorting, and Creating Columns

Let us begin

Most pandas work starts with ordinary table questions:

• Which columns do I need?
• Which rows match my rule?
• Which rows should appear first?
• What new column will make the analysis easier?

These are simple actions, but they are used every day.

Select columns

Select one column:

enrolments["status"]

Select multiple columns:

enrolments[["course_id", "learner_id", "status"]]

Use double brackets for multiple columns because you are passing a list of column names.

Use loc and iloc

loc selects by labels.

enrolments.loc[:, ["learner_id", "status"]]

iloc selects by position.

enrolments.iloc[0:5, 0:3]

In analysis work, loc is often easier to read because it uses names.

Filter rows

A filter uses a True or False rule.

completed = enrolments[enrolments["status"] == "completed"]

Multiple conditions need parentheses.

active_foundation = enrolments[
    (enrolments["status"] == "active") &
    (enrolments["course_level"] == "foundation")
]

Use:

• & for and;
• | for or;
• ~ for not.

Sort rows

Sort by one column:

enrolments.sort_values("start_date")

Sort descending:

enrolments.sort_values("practice_minutes", ascending=False)

Sort by multiple columns:

enrolments.sort_values(["course_id", "start_date"])

Sorting helps inspection. It can reveal unusual values, late dates, or top records.

Create derived columns

A derived column is created from existing data.

enrolments["hours_spent"] = enrolments["minutes_spent"] / 60

Every derived column needs a rule.

Rule:

> hours_spent equals minutes_spent divided by 60.

Do not create a column only because the calculation is available. Create it because it helps answer a question.

Conditional columns

Sometimes a new column depends on a rule.

import numpy as np

enrolments["practice_band"] = np.where(
    enrolments["minutes_spent"] >= 120,
    "high",
    "lower"
)

For more than two groups, use careful logic and test the output.

Avoid unclear chained assignment

This can be risky:

enrolments[enrolments["status"] == "active"]["flag"] = "review"

It may not update what you think it updates.

Prefer a clear loc assignment:

enrolments.loc[enrolments["status"] == "active", "flag"] = "review"

Mini worked example

Question:

> Which active enrolments have low practice time?

Code:

low_practice = enrolments[
    (enrolments["status"] == "active") &
    (enrolments["minutes_spent"] < 60)
].sort_values("minutes_spent")

low_practice["hours_spent"] = low_practice["minutes_spent"] / 60

Interpretation:

> This table does not prove that low practice causes non-completion. It only identifies active enrolments with lower recorded practice time for follow-up analysis.

Takeaway

Selecting, filtering, sorting, and creating columns are the daily tools of table analysis. In the next module, we clean messy data so these operations become more trustworthy.