In [1]:
# Set up packages for lecture. Don't worry about understanding this code, but
# make sure to run it if you're following along.
import numpy as np
import babypandas as bpd
import pandas as pd

from matplotlib_inline.backend_inline import set_matplotlib_formats
import matplotlib.pyplot as plt
set_matplotlib_formats("svg")
plt.style.use('ggplot')
plt.rcParams['figure.figsize'] = (10, 5)

np.set_printoptions(threshold=20, precision=2, suppress=True)
pd.set_option("display.max_rows", 7)
pd.set_option("display.max_columns", 8)
pd.set_option("display.precision", 2)

from IPython.display import display, IFrame

def show_def():
    src = "https://docs.google.com/presentation/d/e/2PACX-1vRKMMwGtrQOeLefj31fCtmbNOaJuKY32eBz1VwHi_5ui0AGYV3MoCjPUtQ_4SB1f9x4Iu6gbH0vFvmB/embed?start=false&loop=false&delayms=60000&rm=minimal"
    width = 960 
    height = 569
    display(IFrame(src, width, height))

Lecture 8 – Functions and Apply¶

DSC 10, Spring 2023¶

Announcements¶

  • Lab 2 is due on Saturday 4/22 at 11:59PM.
  • Homework 2 is due on Tuesday 4/25 at 11:59PM.
    • Do Lab 2 before Homework 2.
  • Come to office hours for help! The schedule is here.
    • We're also offering 1-on-1 tutoring (separate from office hours) to anyone in the class! See this post on Ed for details on how to sign up.
  • Check out the new Diagrams page on the course website.
  • You can safely ignore this warning when you come across it in homeworks and labs.

Agenda¶

  • Functions.
  • Applying functions to DataFrames.
    • Example: Student names.

Reminder: Use the DSC 10 Reference Sheet. You can also use it on exams!

Functions¶

Defining functions¶

  • We've learned how to do quite a bit in Python:
    • Manipulate arrays, Series, and DataFrames.
    • Perform operations on strings.
    • Create visualizations.
  • But so far, we've been restricted to using existing functions (e.g. max, np.sqrt, len) and methods (e.g. .groupby, .assign, .plot).

Motivation¶

Suppose you drive to a restaurant 🥘 in LA, located exactly 100 miles away.

  • For the first 50 miles, you drive at 80 miles per hour.
  • For the last 50 miles, you drive at 60 miles per hour.
  • Question: What is your average speed throughout the journey?
  • 🚨 The answer is not 70 miles per hour! Remember, from Homework 1, you need to use the fact that $\text{speed} = \frac{\text{distance}}{\text{time}}$.
$$\text{average speed} = \frac{\text{total distance}}{\text{total time}} = \frac{50 + 50}{\text{time}_1 + \text{time}_2} \text{ miles per hour}$$

In segment 1, when you drove 50 miles at 80 miles per hour, you drove for $\frac{50}{80}$ hours:

$$\text{speed}_1 = \frac{\text{distance}_1}{\text{time}_1}$$
$$80 \text{ miles per hour} = \frac{50 \text{ miles}}{\text{time}_1} \implies \text{time}_1 = \frac{50}{80} \text{ hours}$$

Similarly, in segment 2, when you drove 50 miles at 60 miles per hour, you drove for $\text{time}_2 = \frac{50}{60} \text{ hours}$.

Then,

$$\text{average speed} = \frac{50 + 50}{\frac{50}{80} + \frac{50}{60}} \text{ miles per hour} $$
$$\begin{align*}\text{average speed} &= \frac{50}{50} \cdot \frac{1 + 1}{\frac{1}{80} + \frac{1}{60}} \text{ miles per hour} \\ &= \frac{2}{\frac{1}{80} + \frac{1}{60}} \text{ miles per hour} \end{align*}$$

Example: Harmonic mean¶

The harmonic mean ($\text{HM}$) of two positive numbers, $a$ and $b$, is defined as

$$\text{HM} = \frac{2}{\frac{1}{a} + \frac{1}{b}}$$

It is often used to find the average of multiple rates.

Finding the harmonic mean of 80 and 60 is not hard:

In [2]:
2 / (1 / 80 + 1 / 60)
Out[2]:
68.57142857142857

But what if we want to find the harmonic mean of 80 and 70? 80 and 90? 20 and 40? This would require a lot of copy-pasting, which is prone to error.

It turns out that we can define our own "harmonic mean" function just once, and re-use it multiple times.

In [3]:
def harmonic_mean(a, b):
    return 2 / (1 / a + 1 / b)
In [4]:
harmonic_mean(80, 60)
Out[4]:
68.57142857142857
In [5]:
harmonic_mean(20, 40)
Out[5]:
26.666666666666664

Note that we only had to specify how to calculate the harmonic mean once!

Functions¶

Functions are a way to divide our code into small subparts to prevent us from writing repetitive code. Each time we define our own function in Python, we will use the following pattern.

In [6]:
show_def()

Functions are "recipes"¶

  • Functions take in inputs, known as arguments, do something, and produce some outputs.
  • The beauty of functions is that you don't need to know how they are implemented in order to use them!
    • For instance, you've been using the function bpd.read_csv without knowing how it works.
    • This is the premise of the idea of abstraction in computer science – you'll hear a lot about this if you take DSC 20.
In [7]:
harmonic_mean(20, 40)
Out[7]:
26.666666666666664
In [8]:
harmonic_mean(79, 894)
Out[8]:
145.17163412127442
In [9]:
harmonic_mean(-2, 4)
Out[9]:
-8.0

Parameters and arguments¶

triple has one parameter, x.

In [10]:
def triple(x):
    return x * 3

When we call triple with the argument 5, within the body of harmonic_mean, x means 5.

In [11]:
triple(5)
Out[11]:
15

We can change the argument we call triple with – we can even call it with strings!

In [12]:
triple(7 + 8)
Out[12]:
45
In [13]:
triple('triton')
Out[13]:
'tritontritontriton'

Scope 🩺¶

The names you choose for a function’s parameters are only known to that function (known as local scope). The rest of your notebook is unaffected by parameter names.

In [14]:
def triple(x):
    return x * 3
In [15]:
triple(7)
Out[15]:
21

Since we haven't defined an x outside of the body of triple, our notebook doesn't know what x means.

In [16]:
x

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
/var/folders/ch/hyjw6whx3g9gshnp58738jc80000gp/T/ipykernel_82186/32546335.py in <module>
----> 1 x

NameError: name 'x' is not defined

We can define an x outside of the body of triple, but that doesn't change how triple works.

In [17]:
x = 15
In [18]:
# When triple(12) is called, you can pretend
# there's an invisible line inside the body of x
# that says x = 12.
# The x = 15 above is ignored.
triple(12)
Out[18]:
36

Functions can take 0 or more arguments¶

Functions can have any number of arguments. So far, we've created a function that takes two arguments, harmonic_mean, and a function that takes one argument, triple.

greeting takes no arguments!

In [19]:
def greeting():
    return 'Hi! 👋'
In [20]:
greeting()
Out[20]:
'Hi! 👋'

Functions don't run until you call them!¶

The body of a function is not run until you use (call) the function.

Here, we can define where_is_the_error without seeing an error message.

In [21]:
def where_is_the_error(something):
    '''You can describe your function within triple quotes. For example, this function 
    illustrates that errors don't occur until functions are executed (called).'''
    return (1 / 0) + something

It is only when we call where_is_the_error that Python gives us an error message.

In [22]:
where_is_the_error(5)

---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
/var/folders/ch/hyjw6whx3g9gshnp58738jc80000gp/T/ipykernel_82186/3423408763.py in <module>
----> 1 where_is_the_error(5)

/var/folders/ch/hyjw6whx3g9gshnp58738jc80000gp/T/ipykernel_82186/1703529954.py in where_is_the_error(something)
      2     '''You can describe your function within triple quotes. For example, this function 
      3     illustrates that errors don't occur until functions are executed (called).'''
----> 4     return (1 / 0) + something

ZeroDivisionError: division by zero

Example: first_name¶

Let's create a function called first_name that takes in someone's full name and returns their first name. Example behavior is shown below.

>>> first_name('Pradeep Khosla')
'Pradeep'

Hint: Use the string method .split.

General strategy for writing functions:

  1. First, try and get the behavior to work on a single example.
  2. Then, encapsulate that behavior inside a function.
In [23]:
'Pradeep Khosla'.split(' ')[0]
Out[23]:
'Pradeep'
In [24]:
def first_name(full_name):
    '''Returns the first name given a full name.'''
    return full_name.split(' ')[0]
In [25]:
first_name('Pradeep Khosla')
Out[25]:
'Pradeep'
In [26]:
# What if there are three names?
first_name('Chancellor Pradeep Khosla')
Out[26]:
'Chancellor'

Returning¶

  • The return keyword specifies what the output of your function should be, i.e. what a call to your function will evaluate to.
  • Most functions we write will use return, but using return is not strictly required.
    • If you want to be able to save the output of your function to a variable, you must use return!
  • Be careful: print and return work differently!
In [27]:
def pythagorean(a, b):
    '''Computes the hypotenuse length of a triangle with legs a and b.'''
    c = (a ** 2 + b ** 2) ** 0.5
    print(c)
In [28]:
x = pythagorean(3, 4)

5.0
In [29]:
# No output – why?
x
In [30]:
# Errors – why?
x + 10

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/var/folders/ch/hyjw6whx3g9gshnp58738jc80000gp/T/ipykernel_82186/3305400239.py in <module>
      1 # Errors – why?
----> 2 x + 10

TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'
In [31]:
def better_pythagorean(a, b):
    '''Computes the hypotenuse length of a triangle with legs a and b, 
       and actually returns the result.
    '''
    c = (a ** 2 + b ** 2) ** 0.5
    return c
In [32]:
x = better_pythagorean(3, 4)
x
Out[32]:
5.0
In [33]:
x + 10
Out[33]:
15.0

Returning¶

Once a function executes a return statement, it stops running.

In [34]:
def motivational(quote):
    return 0
    print("Here's a motivational quote:", quote)
In [35]:
motivational('Fall seven times and stand up eight.')
Out[35]:
0

Applying functions to DataFrames¶

DSC 10 student data¶

The DataFrame roster contains the names and lecture sections of all students enrolled in DSC 10 this quarter. The first names are real, while the last names have been anonymized for privacy.

In [36]:
roster = bpd.read_csv('data/roster-anon.csv')
roster
Out[36]:
name section
0 Derrick Gernlq 1PM
1 Tommy Vbpsht 12PM
2 Grace Smgsmb 12PM
... ... ...
273 Norah Pcqynf 12PM
274 Harry Jwofgg 1PM
275 Zhe Ltynpn 1PM

276 rows × 2 columns

Example: Common first names¶

What is the most common first name among DSC 10 students? (Any guesses?)

In [37]:
roster
Out[37]:
name section
0 Derrick Gernlq 1PM
1 Tommy Vbpsht 12PM
2 Grace Smgsmb 12PM
... ... ...
273 Norah Pcqynf 12PM
274 Harry Jwofgg 1PM
275 Zhe Ltynpn 1PM

276 rows × 2 columns

  • Problem: We can't answer that right now, since we don't have a column with first names. If we did, we could group by it.
  • Solution: Use our function that extracts first names on every element of the 'name' column.

Using our first_name function¶

Somehow, we need to call first_name on every student's 'name'.

In [38]:
roster
Out[38]:
name section
0 Derrick Gernlq 1PM
1 Tommy Vbpsht 12PM
2 Grace Smgsmb 12PM
... ... ...
273 Norah Pcqynf 12PM
274 Harry Jwofgg 1PM
275 Zhe Ltynpn 1PM

276 rows × 2 columns

In [39]:
roster.get('name').iloc[0]
Out[39]:
'Derrick Gernlq'
In [40]:
first_name(roster.get('name').iloc[0])
Out[40]:
'Derrick'
In [41]:
first_name(roster.get('name').iloc[1])
Out[41]:
'Tommy'

Ideally, there's a better solution than doing this hundreds of times...

.apply¶

  • To apply the function func_name to every element of column 'col' in DataFrame df, use
df.get('col').apply(func_name)
  • The .apply method is a Series method.
    • Important: We use .apply on Series, not DataFrames.
    • The output of .apply is also a Series.
  • Pass just the name of the function – don't call it!
    • Good ✅: .apply(first_name).
    • Bad ❌: .apply(first_name()).
In [42]:
roster.get('name')
Out[42]:
0      Derrick Gernlq
1        Tommy Vbpsht
2        Grace Smgsmb
            ...      
273      Norah Pcqynf
274      Harry Jwofgg
275        Zhe Ltynpn
Name: name, Length: 276, dtype: object
In [43]:
roster.get('name').apply(first_name)
Out[43]:
0      Derrick
1        Tommy
2        Grace
        ...   
273      Norah
274      Harry
275        Zhe
Name: name, Length: 276, dtype: object

Example: Common first names¶

In [44]:
roster = roster.assign(
    first=roster.get('name').apply(first_name)
)
roster
Out[44]:
name section first
0 Derrick Gernlq 1PM Derrick
1 Tommy Vbpsht 12PM Tommy
2 Grace Smgsmb 12PM Grace
... ... ... ...
273 Norah Pcqynf 12PM Norah
274 Harry Jwofgg 1PM Harry
275 Zhe Ltynpn 1PM Zhe

276 rows × 3 columns

Now that we have a column containing first names, we can find the distribution of first names.

In [45]:
name_counts = (
    roster
    .groupby('first')
    .count()
    .sort_values('name', ascending=False)
    .get(['name'])
)
name_counts
Out[45]:
name
first
Ryan 6
Andrew 4
Grace 3
... ...
Jared 1
Jasnoor 1
Zixuan 1

250 rows × 1 columns

Activity¶

Below:

  • Create a bar chart showing the number of students with each first name, but only include first names shared by at least two students.
  • Determine the proportion of students in DSC 10 who have a first name that is shared by at least two students.

Hint: Start by defining a DataFrame with only the names in name_counts that appeared at least twice. You can use this DataFrame to answer both questions.


✅ Click here to see the solutions after you've tried it yourself. 

shared_names = name_counts[name_counts.get('name') >= 2]

# Bar chart.
shared_names.sort_values('name').plot(kind='barh', y='name');

# Proportion = # students with a shared name / total # of students.
shared_names.get('name').sum() / roster.shape[0]
In [46]:
...
Out[46]:
Ellipsis
In [47]:
...
Out[47]:
Ellipsis

.apply works with built-in functions, too!¶

In [48]:
name_counts.get('name')
Out[48]:
first
Ryan       6
Andrew     4
Grace      3
          ..
Jared      1
Jasnoor    1
Zixuan     1
Name: name, Length: 250, dtype: int64
In [49]:
# Not necessarily meaningful, but doable.
name_counts.get('name').apply(np.log)
Out[49]:
first
Ryan       1.79
Andrew     1.39
Grace      1.10
           ... 
Jared      0.00
Jasnoor    0.00
Zixuan     0.00
Name: name, Length: 250, dtype: float64

Aside: Resetting the index¶

In name_counts, first names are stored in the index, which is not a Series. This means we can't use .apply on it.

In [50]:
name_counts.index
Out[50]:
Index(['Ryan', 'Andrew', 'Grace', 'Ethan', 'Aaron', 'Krishna', 'Tara',
       'Danielle', 'Daniel', 'Jacob',
       ...
       'Hunter', 'Hyunwoo', 'Ibrahim', 'Isabel', 'Isaiah', 'James', 'Janayra',
       'Jared', 'Jasnoor', 'Zixuan'],
      dtype='object', name='first', length=250)
In [51]:
name_counts.index.apply(max)

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/var/folders/ch/hyjw6whx3g9gshnp58738jc80000gp/T/ipykernel_82186/1905262767.py in <module>
----> 1 name_counts.index.apply(max)

AttributeError: 'Index' object has no attribute 'apply'

To help, we can use .reset_index() to turn the index of a DataFrame into a column, and to reset the index back to the default of 0, 1, 2, 3, and so on.

In [52]:
# What is the max of an individual string?
name_counts.reset_index().get('first').apply(max)
Out[52]:
0      y
1      w
2      r
      ..
247    r
248    s
249    x
Name: first, Length: 250, dtype: object

Example: Shared first names and sections¶

  • Suppose you're one of the $\approx$16\% of students in DSC 10 who has a first name that is shared with at least one other student.
  • Let's try and determine whether someone in your lecture section shares the same first name as you.
    • For example, maybe 'Grace Smgsmb' wants to see if there's another 'Grace' in their section.
In [53]:
roster
Out[53]:
name section first
0 Derrick Gernlq 1PM Derrick
1 Tommy Vbpsht 12PM Tommy
2 Grace Smgsmb 12PM Grace
... ... ... ...
273 Norah Pcqynf 12PM Norah
274 Harry Jwofgg 1PM Harry
275 Zhe Ltynpn 1PM Zhe

276 rows × 3 columns

Strategy:

  1. Which section is 'Grace Smgsmb' in?
  2. How many people in that section have a first name of 'Grace'?
In [54]:
which_section = (roster[roster.get('name') == 'Grace Smgsmb'].get('section').iloc[0])
which_section
Out[54]:
'12PM'
In [55]:
section_cond = roster.get('section') == which_section # A Boolean Series!
first_cond = roster.get('first') == 'Grace' # A Boolean Series!
how_many = roster[section_cond & first_cond].shape[0]
how_many
Out[55]:
3

Another function: shared_first_and_section¶

Let's create a function named shared_first_and_section. It will take in the full name of a student and return the number of students in their section with the same first name and section (including them).

Note: This is the first function we're writing that involves using a DataFrame within the function – this is fine!

In [56]:
def shared_first_and_section(name):
    # First, find the row corresponding to that full name in roster.
    # We're assuming that full names are unique.
    row = roster[roster.get('name') == name]
    
    # Then, get that student's first name and section.
    first = row.get('first').iloc[0]
    section = row.get('section').iloc[0]
    
    # Now, find all the students with the same first name and section.
    shared_info = roster[(roster.get('first') == first) & (roster.get('section') == section)]
    
    # Return the number of such students.
    return shared_info.shape[0]
In [57]:
shared_first_and_section('Grace Smgsmb')
Out[57]:
3
In [58]:
shared_first_and_section('Ryan Uklbnk')
Out[58]:
4

Now, let's add a column to roster that contains the values returned by shared_first_and_section.

In [59]:
roster = roster.assign(shared=roster.get('name').apply(shared_first_and_section))
roster
Out[59]:
name section first shared
0 Derrick Gernlq 1PM Derrick 1
1 Tommy Vbpsht 12PM Tommy 1
2 Grace Smgsmb 12PM Grace 3
... ... ... ... ...
273 Norah Pcqynf 12PM Norah 1
274 Harry Jwofgg 1PM Harry 1
275 Zhe Ltynpn 1PM Zhe 1

276 rows × 4 columns

Let's find all of the students who are in a section with someone that has the same first name as them.

In [60]:
roster[(roster.get('shared') >= 2)].sort_values('shared', ascending=False)
Out[60]:
name section first shared
98 Ryan Uklbnk 1PM Ryan 4
38 Ryan Dyrncc 1PM Ryan 4
223 Andrew Mplcin 12PM Andrew 4
... ... ... ... ...
3 Danielle Jhpshv 1PM Danielle 2
78 Jacob Jhcoau 1PM Jacob 2
34 Ashley Pkqzzd 1PM Ashley 2

29 rows × 4 columns

We can narrow this down to a particular lecture section if we'd like.

In [61]:
one_section_only = (
    roster[(roster.get('shared') >= 2) & 
           (roster.get('section') == '12PM')]
    .sort_values('shared', ascending=False)
)
one_section_only
Out[61]:
name section first shared
269 Andrew Dpkpan 12PM Andrew 4
66 Andrew Mdvbuw 12PM Andrew 4
223 Andrew Mplcin 12PM Andrew 4
... ... ... ... ...
225 Brandon Jtzrkj 12PM Brandon 2
247 Andrea Bauodp 12PM Andrea 2
101 Ryan Vztixv 12PM Ryan 2

17 rows × 4 columns

For instance, the above is telling us that there are 4 Andrews, 2 Brandons, 2 Andreas, and 2 Ryans in the 12PM section of DSC 10.

In [62]:
# All of the names shared by multiple students in the 12PM section.
one_section_only.get('first').unique()
Out[62]:
array(['Andrew', 'Grace', 'Ethan', 'Brandon', 'Ryan', 'Andrea', 'Aaron'],
      dtype=object)

Sneak peek¶

While the DataFrames on the previous slide contain the info we were looking for, they're not organized very conveniently. For instance, there are four rows containing the fact that there are 4 Andrews in the 12PM lecture section.

Wouldn't it be great if we could create a DataFrame like the one below? We'll see how on Friday!

section first count
0 12PM Andrew 4
1 1PM Danielle 2
2 12PM Ryan 2
3 1PM Ryan 4
4 12PM Grace 3

Activity¶

Find the longest first name in the class that is shared by at least two students in the same section.

Hint: You'll have to use both .assign and .apply.


✅ Click here to see the answer after you've tried it yourself. 

with_len = roster.assign(name_len=roster.get('first').apply(len))
with_len[with_len.get('shared') >= 2].sort_values('name_len')
In [63]:
...
Out[63]:
Ellipsis

Summary, next time¶

Summary¶

  • Functions are a way to divide our code into small subparts to prevent us from writing repetitive code.
  • The .apply method allows us to call a function on every single element of a Series, which usually comes from .getting a column of a DataFrame.

Next time¶

More advanced DataFrame manipulations!