In [1]:
            # Run this cell to set up packages for lecture.
from lec06_imports import *

Lecture 6 – Data Visualization 📈¶

DSC 10, Fall 2024¶

Announcements¶

  • Quiz 1 is today in Solis 104 during your assigned quiz session.
    • Check your email for a seating assignment and refer to this map to know where to sit.
    • 20 minute paper-based quiz with no aids allowed.
    • The quiz covers Lectures 1 through 4, or BPD 1-9 in the babypandas notes.
      • Study by doing practice problems, especially those from this week's discussion and previous quizzes.
      • Do as much as you can of Lab 1 (due tomorrow) and Homework 1 (due Sunday) before the quiz to develop fluency.
      • You should know all of the functions and methods covered in lecture such as abs, round, .upper() for strings, etc.
    • See more quiz announcements on Ed.

Agenda¶

  • Adjusting columns.
  • Why visualize?
  • Terminology.
  • Scatter plots.
  • Line plots.
  • Bar charts.

Aside: Keyboard shortcuts¶

There are several keyboard shortcuts built into Jupyter Notebooks designed to help you save time. To see them, either click the keyboard button in the toolbar above or hit the H key on your keyboard (as long as you're not actively editing a cell).

Particularly useful shortcuts:

Action Keyboard shortcut
Run cell + jump to next cell SHIFT + ENTER
Save the notebook CTRL/CMD + S
Create new cell above/below A/B
Delete cell DD

Adjusting columns¶

.count()¶

As before, we'll work with the states DataFrame. Notice the column names don't make sense after grouping with the .count() aggregation method.

In [8]:
states = bpd.read_csv('data/states.csv')
states = states.assign(Density=states.get('Population') / states.get('Land Area'))
states = states.set_index('State')
states
Out[8]:
Region Capital City Population Land Area Party Density
State
Alabama South Montgomery 5024279 50645 Republican 99.21
Alaska West Juneau 733391 570641 Republican 1.29
Arizona West Phoenix 7151502 113594 Republican 62.96
... ... ... ... ... ... ...
West Virginia South Charleston 1793716 24038 Republican 74.62
Wisconsin Midwest Madison 5893718 54158 Republican 108.82
Wyoming West Cheyenne 576851 97093 Republican 5.94

50 rows × 6 columns

In [9]:
states.groupby('Region').count()
Out[9]:
Capital City Population Land Area Party Density
Region
Midwest 12 12 12 12 12
Northeast 9 9 9 9 9
South 16 16 16 16 16
West 13 13 13 13 13

Adjusting columns with .assign, .drop, and .get¶

  1. To rename a column, use .assign to create a new column containing the same values as an existing column.
    • New columns are added on the right.
  1. Then use .drop(columns=list_of_column_labels) to drop any columns you no longer need.
    • Alternatively, use .get(list_of_column_labels) to keep only certain columns. The columns will appear in the order you specify, so this is also useful for reordering columns!

Two ways to .get¶

  1. Getting a single column name gives a Series.
  2. Getting a list of column names gives a DataFrame. (Even if the list has just one element!)
In [14]:
states.get('Capital City')
Out[14]:
State
Alabama          Montgomery
Alaska               Juneau
Arizona             Phoenix
                    ...    
West Virginia    Charleston
Wisconsin           Madison
Wyoming            Cheyenne
Name: Capital City, Length: 50, dtype: object
In [15]:
states.get(['Capital City', 'Party'])
Out[15]:
Capital City Party
State
Alabama Montgomery Republican
Alaska Juneau Republican
Arizona Phoenix Republican
... ... ...
West Virginia Charleston Republican
Wisconsin Madison Republican
Wyoming Cheyenne Republican

50 rows × 2 columns

In [16]:
states.get(['Capital City'])
Out[16]:
Capital City
State
Alabama Montgomery
Alaska Juneau
Arizona Phoenix
... ...
West Virginia Charleston
Wisconsin Madison
Wyoming Cheyenne

50 rows × 1 columns

Activity¶

Change the DataFrame states_by_region so that it only has one column, called Count, containing the number of states in each region.

In [18]:
states_by_region = states.groupby('Region').count()
states_by_region
Out[18]:
Capital City Population Land Area Party Density
Region
Midwest 12 12 12 12 12
Northeast 9 9 9 9 9
South 16 16 16 16 16
West 13 13 13 13 13

Why visualize?¶

Little Women¶

In Lecture 1, we were able to answer questions about the plot of Little Women without having to read the novel and without having to understand Python code. Some of those questions included:

  • Who is the main character?
  • Which pair of characters gets married in Chapter 35?

We answered these questions from a data visualization alone!

In [21]:
bpd.read_csv('data/lw_counts.csv').plot(x='Chapter');
No description has been provided for this image

Napoleon's March¶

"Probably the best statistical graphic ever drawn, this map by Charles Joseph Minard portrays the losses suffered by Napoleon's army in the Russian campaign of 1812." (source)

No description has been provided for this image

Why visualize?¶

  • Computers are better than humans at crunching numbers, but humans are better at identifying visual patterns.
  • Visualizations allow us to understand lots of data quickly – they make it easier to spot trends and communicate our results with others.
  • There are many types of visualizations; in this class, we'll look at scatter plots, line plots, bar charts, and histograms, but there are many others.
    • The right choice depends on the type of data.

Terminology¶

Individuals and variables¶

No description has been provided for this image
  • Individual (row): Person/place/thing for which data is recorded. Also called an observation.
  • Variable (column): Something that is recorded for each individual. Also called a feature.

Types of variables¶

There are two main types of variables:

  • Numerical: It makes sense to do arithmetic with the values.
  • Categorical: Values fall into categories, that may or may not have some order to them.

Note that here, "variable" does not mean a variable in Python, but rather it means a column in a DataFrame.

Examples of numerical variables¶

  • Salaries of NBA players 🏀.
    • Individual: An NBA player.
    • Variable: Their salary.
  • Company's annual profit 💰.
    • Individual: A company.
    • Variable: Its annual profit.
  • Flu shots administered per day 💉.
    • Individual: Date.
    • Variable: Number of flu shots administered on that date.

Examples of categorical variables¶

  • Movie genres 🎬.
    • Individual: A movie.
    • Variable: Its genre.
  • Zip codes 🏠.
    • Individual: US resident.
    • Variable: Zip code.
      • Even though they look like numbers, zip codes are categorical (arithmetic doesn't make sense).
  • Level of prior programming experience for students in DSC 10 🧑‍🎓.
    • Individual: Student in DSC 10.
    • Variable: Their level of prior programming experience, e.g. none, low, medium, or high.
      • There is an order to these categories!

Concept Check ✅ – Answer at cc.dsc10.com¶

Which of these is not a numerical variable?

A. Fuel economy in miles per gallon.

B. Number of quarters at UCSD.

C. College at UCSD (Sixth, Seventh, etc).

D. Bank account number.

E. More than one of these are not numerical variables.

Types of visualizations¶

The type of visualization we create depends on the kinds of variables we're visualizing.

  • Scatter plot: Numerical vs. numerical.
  • Line plot: Sequential numerical (time) vs. numerical.
  • Bar chart: Categorical vs. numerical.
  • Histogram: Numerical.
    • Will cover next time.

We may interchange the words "plot", "chart", and "graph"; they all mean the same thing.

Scatter plots¶

The data: exoplanets discovered by NASA 🪐¶

An exoplanet is a planet outside our solar system. NASA has discovered over 5,000 exoplanets so far in its search for signs of life beyond Earth. 👽

Column Contents

'Distance'| Distance from Earth, in light years. 'Magnitude'| Apparent magnitude, which measures brightness in such a way that brighter objects have lower values. 'Type'| Categorization of planet based on its composition and size. 'Year'| When the planet was discovered. 'Detection'| The method of detection used to discover the planet. 'Mass'| The ratio of the planet's mass to Earth's mass. 'Radius'| The ratio of the planet's radius to Earth's radius.

In [47]:
exo = bpd.read_csv('data/exoplanets.csv').set_index('Name')
exo
Out[47]:
Distance Magnitude Type Year Detection Mass Radius
Name
11 Comae Berenices b 304.0 4.72 Gas Giant 2007 Radial Velocity 6165.90 11.88
11 Ursae Minoris b 409.0 5.01 Gas Giant 2009 Radial Velocity 4684.81 11.99
14 Andromedae b 246.0 5.23 Gas Giant 2008 Radial Velocity 1525.58 12.65
... ... ... ... ... ... ... ...
YZ Ceti b 12.0 12.07 Terrestrial 2017 Radial Velocity 0.70 0.91
YZ Ceti c 12.0 12.07 Super Earth 2017 Radial Velocity 1.14 1.05
YZ Ceti d 12.0 12.07 Super Earth 2017 Radial Velocity 1.09 1.03

5043 rows × 7 columns

Scatter plots¶

  • What is the relationship between 'Distance' and 'Magnitude'?
In [50]:
exo.plot(kind='scatter', x='Distance', y='Magnitude');
No description has been provided for this image
  • Further planets have greater 'Magnitude' (meaning they are less bright), which makes sense.

  • The data appears curved because 'Magnitude' is measured on a logarithmic scale. A decrease of one unit in 'Magnitude' corresponds to a 2.5 times increase in brightness.

No description has been provided for this image

Scatter plots¶

  • Scatter plots visualize the relationship between two numerical variables.
  • To create one from a DataFrame df, use
df.plot(
    kind='scatter', 
    x=x_column_for_horizontal, 
    y=y_column_for_vertical
)
  • The resulting scatter plot has one point per row of df.
  • If you put a semicolon after a call to .plot, it will hide the weird text output that displays.

Zooming in 🔍¶

The majority of exoplanets are less than 10,000 light years away; if we'd like to zoom in on just these exoplanets, we can query before plotting.

In [55]:
exo[exo.get('Distance') < 10000].plot(kind='scatter', x='Distance', y='Magnitude');
No description has been provided for this image

Line plots 📉¶

Line plots¶

  • How has the 'Magnitude' of newly discovered exoplanets changed over time?
In [59]:
# There were multiple exoplanets discovered each year.
# What operation can we apply to this DataFrame so that there is one row per year?
exo
Out[59]:
Distance Magnitude Type Year Detection Mass Radius
Name
11 Comae Berenices b 304.0 4.72 Gas Giant 2007 Radial Velocity 6165.90 11.88
11 Ursae Minoris b 409.0 5.01 Gas Giant 2009 Radial Velocity 4684.81 11.99
14 Andromedae b 246.0 5.23 Gas Giant 2008 Radial Velocity 1525.58 12.65
... ... ... ... ... ... ... ...
YZ Ceti b 12.0 12.07 Terrestrial 2017 Radial Velocity 0.70 0.91
YZ Ceti c 12.0 12.07 Super Earth 2017 Radial Velocity 1.14 1.05
YZ Ceti d 12.0 12.07 Super Earth 2017 Radial Velocity 1.09 1.03

5043 rows × 7 columns

  • Let's calculate the average 'Magnitude' of all exoplanets discovered in each 'Year'.
In [61]:
exo.groupby('Year').mean()
Out[61]:
Distance Magnitude Mass Radius
Year
1995 50.00 5.45 146.20 13.97
1996 51.33 5.12 1020.67 13.09
1997 57.00 5.41 332.10 13.53
... ... ... ... ...
2021 1944.22 13.01 255.42 4.44
2022 508.61 10.62 943.16 6.77
2023 451.89 12.09 162.78 7.12

29 rows × 4 columns

In [62]:
exo.groupby('Year').mean().plot(kind='line', y='Magnitude');
No description has been provided for this image
  • It looks like the brightest planets were discovered first, which makes sense.

  • NASA's Kepler space telescope began its nine-year mission in 2009, leading to a boom in the discovery of exoplanets.

Line plots¶

  • Line plots show trends in numerical variables over time.
  • To create one from a DataFrame df, use
df.plot(
    kind='line', 
    x=x_column_for_horizontal, 
    y=y_column_for_vertical
)
  • To use the index as the x-axis, omit the x= argument.
    • This doesn't work for scatterplots, but it works for most other plot types.

Extra video on line plots¶

If you're curious how line plots work under the hood, watch this video we made a few quarters ago.

In [66]:
YouTubeVideo('glzZ04D1kDg')
Out[66]:

Bar charts 📊¶

Bar charts¶

  • How big are each of the different 'Type's of exoplanets, on average?
In [70]:
types = exo.groupby('Type').mean()
types
Out[70]:
Distance Magnitude Year Mass Radius
Type
Gas Giant 1096.40 10.30 2013.73 1472.39 12.74
Neptune-like 2189.02 13.52 2016.59 15.28 3.11
Super Earth 1916.26 13.85 2016.43 5.81 1.58
Terrestrial 1373.60 13.45 2016.37 1.62 0.85
In [71]:
types.plot(kind='barh', y='Radius');
No description has been provided for this image
In [72]:
types.plot(kind='barh', y='Mass');
No description has been provided for this image
  • It looks like the 'Gas Giant's are aptly named!
No description has been provided for this image

Bar charts¶

  • Bar charts visualize the relationship between a categorical variable and a numerical variable.
  • In a bar chart...
    • The thickness and spacing of bars is arbitrary.
    • The order of the categorical labels doesn't matter.
  • To create one from a DataFrame df, use
df.plot(
    kind='barh', 
    x=categorical_column_name, 
    y=numerical_column_name
)
  • The "h" in 'barh' stands for "horizontal".
    • It's easier to read labels this way.
  • Note that in the previous chart, we set y='Mass' even though mass is measured by x-axis length.

Bar charts and sorting¶

What are the most popular 'Detection' methods for discovering exoplanets?

In [77]:
# Count how many exoplanets are discovered by each detection method.
popular_detection = exo.groupby('Detection').count()
popular_detection
Out[77]:
Distance Magnitude Type Year Mass Radius
Detection
Astrometry 1 1 1 1 1 1
Direct Imaging 50 50 50 50 50 50
Disk Kinematics 1 1 1 1 1 1
... ... ... ... ... ... ...
Radial Velocity 1019 1019 1019 1019 1019 1019
Transit 3914 3914 3914 3914 3914 3914
Transit Timing Variations 23 23 23 23 23 23

11 rows × 6 columns

In [78]:
# Give columns more meaningful names and eliminate redundancy.
popular_detection = (popular_detection.assign(Count=popular_detection.get('Distance'))
                                      .get(['Count'])
                                      .sort_values(by='Count', ascending=False)
                    )
popular_detection
Out[78]:
Count
Detection
Transit 3914
Radial Velocity 1019
Direct Imaging 50
... ...
Astrometry 1
Disk Kinematics 1
Pulsar Timing 1

11 rows × 1 columns

In [79]:
# Notice that the bars appear in the opposite order relative to the DataFrame.
popular_detection.plot(kind='barh', y='Count');
No description has been provided for this image
In [80]:
# Change "barh" to "bar" to get a vertical bar chart. 
# These are harder to read, but the bars do appear in the same order as the DataFrame.
popular_detection.plot(kind='bar', y='Count');
No description has been provided for this image

Multiple plots on the same axes¶

Can we look at both the average 'Magnitude' and the average 'Radius' for each 'Type' at the same time?

In [82]:
types.get(['Magnitude', 'Radius']).plot(kind='barh');
No description has been provided for this image

How did we do that?

Overlaying plots¶

When calling .plot, if we omit the y=column_name argument, all other columns are plotted.

In [85]:
types
Out[85]:
Distance Magnitude Year Mass Radius
Type
Gas Giant 1096.40 10.30 2013.73 1472.39 12.74
Neptune-like 2189.02 13.52 2016.59 15.28 3.11
Super Earth 1916.26 13.85 2016.43 5.81 1.58
Terrestrial 1373.60 13.45 2016.37 1.62 0.85
In [86]:
types.plot(kind='barh');
No description has been provided for this image

Selecting multiple columns at once¶

Remember, to select multiple columns, use .get([column_1, ..., column_k]). This returns a DataFrame.

In [88]:
types
Out[88]:
Distance Magnitude Year Mass Radius
Type
Gas Giant 1096.40 10.30 2013.73 1472.39 12.74
Neptune-like 2189.02 13.52 2016.59 15.28 3.11
Super Earth 1916.26 13.85 2016.43 5.81 1.58
Terrestrial 1373.60 13.45 2016.37 1.62 0.85
In [89]:
types.get(['Magnitude', 'Radius'])
Out[89]:
Magnitude Radius
Type
Gas Giant 10.30 12.74
Neptune-like 13.52 3.11
Super Earth 13.85 1.58
Terrestrial 13.45 0.85
In [90]:
types.get(['Magnitude', 'Radius']).plot(kind='barh');
No description has been provided for this image

Summary¶

Summary¶

  • Visualizations make it easy to extract patterns from datasets.
  • There are two main types of variables: categorical and numerical.
  • The types of the variables we're visualizing inform our choice of which type of visualization to use.
  • Today, we looked at scatter plots, line plots, and bar charts.
  • Next time: histograms.