Skip to course content
Free course

Data Analysis and Visualization with Python / Module 3

Module 3 lesson

Module 3 Lesson: Pandas Core Objects and Data Import

Let us begin

NumPy gave us arrays. pandas gives us labelled tables.

In pandas, the main object is a DataFrame. A DataFrame is like a spreadsheet table inside Python. It has rows, columns, labels, and types.

A Series is one labelled column or one labelled sequence.

You will use pandas for most tabular analysis in this course.

Import pandas

The common import is:

import pandas as pd

The alias pd is a community convention. It keeps code short and familiar.

Reading a CSV file

Many first analysis projects start with a CSV file.

import pandas as pd

enrolments = pd.read_csv("data/course_enrolments.csv")

Use a relative path. Do not use a local absolute path like /Users/... in course notebooks.

First inspection

After loading data, do not begin cleaning immediately. First inspect.

enrolments.head()
enrolments.shape
enrolments.columns

Ask:

  • How many rows are there?
  • How many columns are there?
  • What does one row mean?
  • Are the column names readable?
  • Are there obvious missing values?

Info and dtypes

enrolments.info()

This shows each column, non-null counts, and dtype.

Common dtypes:

  • int64: whole numbers;
  • float64: decimal numbers;
  • object or string: text-like values;
  • bool: True or False;
  • datetime64: dates and times;
  • category: repeated labels.

Do not panic if dtypes are not perfect at first. Inspection tells us what to fix.

Summary statistics

enrolments.describe()

This summarizes numeric columns.

For categories:

enrolments["status"].value_counts()

Use both. Numeric summaries and category counts answer different questions.

Missing values

enrolments.isna().sum()

This tells you how many missing values each column has.

A missing value is not automatically an error. It may mean:

  • the value was not collected;
  • the value does not apply;
  • the value is delayed;
  • the value was lost;
  • the value is intentionally blank.

Do not fill or drop missing values until you understand them.

Data dictionary

A data dictionary explains columns.

For each column, record:

  • column name;
  • plain-English meaning;
  • type;
  • example value;
  • allowed values or range;
  • missing-value rule;
  • notes.

The data dictionary turns a table from a mystery into a source you can reason about.

Export safely

When you save an output, do not overwrite raw data.

enrolments.to_csv("outputs/enrolments_inspected.csv", index=False)

Use a new file path. Keep the original file unchanged.

Mini worked example

courses = pd.read_csv("data/courses.csv")

print(courses.shape)
print(courses.dtypes)
print(courses.isna().sum())

courses["level"].value_counts()

A good first note might be:

> The table has one row per course. The level column is categorical. The planned_hours column should be numeric. No cleaning decision should be made until missing values and allowed levels are checked.

Takeaway

pandas helps us bring a table into Python, inspect it, and start documenting it. In the next module, we begin selecting rows, filtering data, sorting, and creating new columns.