In [1]:
from dsc80_utils import *

Lecture 11 – Regular Expressions and Text Features¶

DSC 80, Winter 2024¶

Announcements 📣¶

  • Lab 6 is due on Wednesday, February 21st at 5PM (no slip days!).

    • Remember that next Monday is a holiday.

  • Project 3 is out! In it, you'll implement an N-Gram language model. We'll start covering relevant topics for it today.

    • The checkpoint is due on Thursday, February 22nd.
    • The full project is due on Thursday, February 29th.

  • Midterm Exam regrades are due tomorrow.

    • We've given back partial credit on a few questions, like 1e, bringing the mean up ~1%.

  • If at least 80% of the class fills out the Mid-Quarter Survey by Saturday at 11:59PM, everyone will earn 2 extra points (2.5%) on the Midterm Exam!

Agenda 📆¶

  • Most of today's lecture will be about regular expressions. Good resources:
    • regex101.com, a helpful site to have open while writing regular expressions.
    • Python re library documentation and how-to.
      • The "how-to" is great, read it!
    • regex "cheat sheet" (taken from here).
    • These are all on the resources tab of the course website as well.
  • With remaining time, we'll start discussing text features.
    • How can we use strings as inputs in linear regression, for example?

Exercise

Consider the following HTML document, which represents a webpage containing the top few songs with the most streams on Spotify today in Canada.

<head>
    <title>3*Canada-2022-06-04</title>
</head>
<body>
    <h1>Spotify Top 3 - Canada</h1>
    <table>
        <tr class='heading'>
            <th>Rank</th>
            <th>Artist(s)</th> 
            <th>Song</th>
        </tr>
        <tr class=1>
            <td>1</td>
            <td>Harry Styles</td> 
            <td>As It Was</td>
        </tr>
        <tr class=2>
            <td>2</td>
            <td>Jack Harlow</td> 
            <td>First Class</td>
        </tr>
        <tr class=3>
            <td>3</td>
            <td>Kendrick Lamar</td> 
            <td>N95</td>
        </tr>
    </table>
</body>

Part 4: Complete the implementation of the function top_nth, which takes in a positive integer n and returns the name of the n-th ranked song in the HTML document. For instance, top_nth(2) should evaluate to "First Class" (n=1 corresponds to the top song).

Note: Your implementation should work in the case that the page contains more than 3 songs.

def top_nth(n):
    return soup.find("tr", attrs=__(a)__).find_all("td")__(b)__

Motivation¶

In [2]:
contact = '''
Thank you for buying our expensive product!

If you have a complaint, please send it to complaints@compuserve.com or call (800) 867-5309.

If you are happy with your purchase, please call us at (800) 123-4567; we'd love to hear from you!

Due to high demand, please allow one-hundred (100) business days for a response.
'''

Who called? 📞¶

  • Goal: Extract all phone numbers from a piece of text, assuming they are of the form '(###) ###-####'.
In [3]:
print(contact)

Thank you for buying our expensive product!

If you have a complaint, please send it to complaints@compuserve.com or call (800) 867-5309.

If you are happy with your purchase, please call us at (800) 123-4567; we'd love to hear from you!

Due to high demand, please allow one-hundred (100) business days for a response.

  • We can do this using the same string methods we've come to know and love.

  • Strategy:

    • Split by spaces.
    • Check if there are any consecutive "words" where:
      • the first "word" looks like an area code, like '(678)'.
      • the second "word" looks like the last 7 digits of a phone number, like '999-8212'.

Let's first write a function that takes in a string and returns whether it looks like an area code.

In [4]:
def is_possibly_area_code(s):
    '''Does `s` look like (678)?'''
    return (len(s) == 5 and
            s.startswith('(') and
            s.endswith(')') and
            s[1:4].isnumeric())
In [5]:
is_possibly_area_code('(123)')
Out[5]:
True
In [6]:
is_possibly_area_code('(99)')
Out[6]:
False

Let's also write a function that takes in a string and returns whether it looks like the last 7 digits of a phone number.

In [7]:
def is_last_7_phone_number(s):
    '''Does `s` look like 999-8212?'''
    return len(s) == 8 and s[0:3].isnumeric() and s[3] == '-' and s[4:].isnumeric()
In [8]:
is_last_7_phone_number('999-8212')
Out[8]:
True
In [9]:
is_last_7_phone_number('534 1100')
Out[9]:
False

Finally, let's split the entire text by spaces, and check whether there are any instances where pieces[i] looks like an area code and pieces[i+1] looks like the last 7 digits of a phone number.

In [10]:
# Removes punctuation from the end of each string.
pieces = [s.rstrip('.,?;"\'') for s in contact.split()]

for i in range(len(pieces) - 1):
    if is_possibly_area_code(pieces[i]):
        if is_last_7_phone_number(pieces[i+1]):
            print(pieces[i], pieces[i+1])

(800) 867-5309
(800) 123-4567

Is there a better way?¶

  • This was an example of pattern matching.
  • It can be done with string methods, but there is often a better approach: regular expressions.
In [11]:
print(contact)

Thank you for buying our expensive product!

If you have a complaint, please send it to complaints@compuserve.com or call (800) 867-5309.

If you are happy with your purchase, please call us at (800) 123-4567; we'd love to hear from you!

Due to high demand, please allow one-hundred (100) business days for a response.
In [12]:
import re
re.findall(r'\(\d{3}\) \d{3}-\d{4}', contact)
Out[12]:
['(800) 867-5309', '(800) 123-4567']

🤯

Basic regular expressions¶

Regular expressions¶

  • A regular expression, or regex for short, is a sequence of characters used to match patterns in strings.
    • For example, \(\d{3}\) \d{3}-\d{4} describes a pattern that matches US phone numbers of the form '(XXX) XXX-XXXX'.
    • Think of regex as a "mini-language" (formally: they are a grammar for describing a language).
  • Pros: They are very powerful and are widely used (virtually every programming language has a module for working with them).
  • Cons: They can be hard to read and have many different "dialects."

Writing regular expressions¶

  • You will ultimately write most of your regular expressions in Python, using the re module. We will see how to do so shortly.

  • However, a useful tool for designing regular expressions is regex101.com.

  • We will use it heavily during lecture; you should have it open as we work through examples. If you're trying to revisit this lecture in the future, you'll likely want to watch the podcast.

Literals¶

  • A literal is a character that has no special meaning.

  • Letters, numbers, and some symbols are all literals.

  • Some symbols, like ., *, (, and ), are special characters.

  • *Example*: The regex hey matches the string 'hey'. The regex he. also matches the string 'hey'.

Regex building blocks 🧱¶

The four main building blocks for all regexes are shown below (table source, inspiration).

operation order of op. example matches ✅ does not match ❌
concatenation 3 AABAAB 'AABAAB' every other string
or 4 AA|BAAB 'AA', 'BAAB' every other string
closure
(zero or more)		 2 AB*A 'AA', 'ABBBBBBA' 'AB', 'ABABA'
parentheses 1 A(A|B)AAB
(AB)*A		 'AAAAB', 'ABAAB'
'A', 'ABABABABA'		 every other string
'AA', 'ABBA'

Note that |, (, ), and * are special characters, not literals. They manipulate the characters around them.

*Example (or, parentheses)*:

  • What does DSC 30|80 match?
  • What does DSC (30|80) match?

*Example (closure, parentheses)*:

  • What does blah* match?
  • What does (blah)* match?

Exercise

Write a regular expression that matches 'billy', 'billlly', 'billlllly', etc.

  • First, think about how to match strings with any even number of 'l's, including zero 'l's (i.e. 'biy').
  • Then, think about how to match only strings with a positive even number of 'l's.

✅ Click here to see the answer after you've tried it yourself at regex101.com. bi(ll)*y will match any even number of 'l's, including 0.

To match only a positive even number of 'l's, we'd need to first "fix into place" two 'l's, and then follow that up with zero or more pairs of 'l's. This specifies the regular expression bill(ll)*y.

Exercise

Write a regular expression that matches 'billy', 'billlly', 'biggy', 'biggggy', etc.

Specifically, it should match any string with a positive even number of 'l's in the middle, or a positive even number of 'g's in the middle.


✅ Click here to see the answer after you've tried it yourself at regex101.com.

Possible answers: bi(ll(ll)*|gg(gg)*)y or bill(ll)*y|bigg(gg)*y.


Note, bill(ll)*|gg(gg)*y is not a valid answer! This is because "concatenation" comes before "or" in the order of operations. This regular expression would match strings that match bill(ll)*, like 'billll', OR strings that match gg(gg)*y, like 'ggy'.

Intermediate regex¶

More regex syntax¶

operation example matches ✅ does not match ❌
wildcard .U.U.U. 'CUMULUS'
'JUGULUM'		 'SUCCUBUS'
'TUMULTUOUS'
character class [A-Za-z][a-z]* 'word'
'Capitalized'		 'camelCase'
'4illegal'
at least one bi(ll)+y 'billy'
'billlllly'		 'biy'
'bily'
between $i$ and $j$ occurrences m[aeiou]{1,2}m 'mem'
'maam'
'miem'		 'mm'
'mooom'
'meme'

., [, ], +, {, and } are also special characters, in addition to |, (, ), and *.

*Example (character classes, at least one): [A-E]+ is just shortform for `(A|B|C|D|E)(A|B|C|D|E)`.

*Example (wildcard)*:

  • What does . match?
  • What does he. match?
  • What does ... match?

*Example (at least one, closure)*:

  • What does 123+ match?
  • What does 123* match?

*Example (number of occurrences)*: What does tri{3,5} match? Does it match 'triiiii'?

*Example (character classes, number of occurrences)*: What does [1-6a-f]{3}-[7-9E-S]{2} match?

Exercise

Write a regular expression that matches any lowercase string has a repeated vowel, such as 'noon', 'peel', 'festoon', or 'zeebraa'.


✅ Click here to see the answer after you've tried it yourself at regex101.com.

One answer: [a-z]*(aa|ee|ii|oo|uu)[a-z]*


This regular expression matches strings of lowercase characters that have 'aa', 'ee', 'ii', 'oo', or 'uu' in them anywhere. [a-z]* means "zero or more of any lowercase characters"; essentially we are saying it doesn't matter what letters come before or after the double vowels, as long as the double vowels exist somewhere.

Exercise

Write a regular expression that matches any string that contains both a lowercase letter and a number, in any order. Examples include 'billy80', '80!!billy', and 'bil8ly0'.


✅ Click here to see the answer after you've tried it yourself at regex101.com.

One answer: (.*[a-z].*[0-9].*)|(.*[0-9].*[a-z].*)


We can break the above regex into two parts – everything before the |, and everything after the |.

The first part, .*[a-z].*[0-9].*, matches strings in which there is at least one lowercase character and at least one digit, with the lowercase character coming first.

The second part, .*[0-9].*[a-z].*, matches strings in which there is at least one lowercase character and at least one digit, with the digit coming first.

Note, the .* between the digit and letter classes is needed in the event the string has non-digit and non-letter characters.

This is the kind of task that would be easier to accomplish with regular Python string methods.

Even more regex syntax¶

operation example matches ✅ does not match ❌
escape character ucsd\.edu 'ucsd.edu' 'ucsd!edu'
beginning of line ^ark 'ark two'
'ark o ark'		 'dark'
end of line ark$ 'dark'
'ark o ark'		 'ark two'
zero or one cat? 'ca'
'cat'		 'cart' (matches 'ca' only)
built-in character classes* \w+
\d+		 'billy'
'231231'		 'this person'
'858 people'
character class negation [^a-z]+ 'KINGTRITON551'
'1721$$'		 'porch'
'billy.edu'

**Note*: in Python's implementation of regex,

  • \d refers to digits.
  • \w refers to alphanumeric characters ([A-Z][a-z][0-9]_). Whenever we say "alphanumeric" in an assignment, we're referring to \w!
  • \s refers to whitespace.
  • \b is a word boundary.

*Example (escaping)*:

  • What does he. match?
  • What does he\. match?
  • What does (858) match?
  • What does \(858\) match?

*Example (anchors)*:

  • What does 858-534 match?
  • What does ^858-534 match?
  • What does 858-534$ match?

*Example (built-in character classes)*:

  • What does \d{3} \d{3}-\d{4} match?
  • What does \bcat\b match? Does it find a match in 'my cat is hungry'? What about 'concatenate' or 'kitty cat'?

Remember, in Python's implementation of regex,

  • \d refers to digits.
  • \w refers to alphanumeric characters ([A-Z][a-z][0-9]_). Whenever we say "alphanumeric" in an assignment, we're referring to \w!
  • \s refers to whitespace.
  • \b is a word boundary.

Exercise

Write a regular expression that matches any string that:

  • is between 5 and 10 characters long, and
  • is made up of only vowels (either uppercase or lowercase, including 'Y' and 'y'), periods, and spaces.

Examples include 'yoo.ee.IOU' and 'AI.I oey'.


✅ Click here to see the answer after you've tried it yourself at regex101.com.

One answer: ^[aeiouyAEIOUY. ]{5,10}$


Key idea: Within a character class (i.e. [...]), special characters do not generally need to be escaped.

Regex in Python¶

re in Python¶

The re package is built into Python. It allows us to use regular expressions to find, extract, and replace strings.

In [13]:
import re

re.search takes in a string regex and a string text and returns the location and substring corresponding to the first match of regex in text.

In [14]:
re.search('AB*A', 
          'here is a string for you: ABBBA. here is another: ABBBBBBBA')
Out[14]:
<re.Match object; span=(26, 31), match='ABBBA'>

re.findall takes in a string regex and a string text and returns a list of all matches of regex in text. You'll use this most often.

In [15]:
re.findall('AB*A', 
           'here is a string for you: ABBBA. here is another: ABBBBBBBA')
Out[15]:
['ABBBA', 'ABBBBBBBA']

re.sub takes in a string regex, a string repl, and a string text, and replaces all matches of regex in text with repl.

In [16]:
re.sub('AB*A', 
       'billy', 
       'here is a string for you: ABBBA. here is another: ABBBBBBBA')
Out[16]:
'here is a string for you: billy. here is another: billy'

Raw strings¶

When using regular expressions in Python, it's a good idea to use raw strings, denoted by an r before the quotes, e.g. r'exp'.

In [17]:
re.findall('\bcat\b', 'my cat is hungry')
Out[17]:
[]
In [18]:
re.findall(r'\bcat\b', 'my cat is hungry')
Out[18]:
['cat']
In [19]:
# Huh?
print('\bcat\b')

cat

Capture groups¶

  • Surround a regex with ( and ) to define a capture group within a pattern.
  • Capture groups are useful for extracting relevant parts of a string.
In [20]:
re.findall(r'\w+@(\w+)\.edu', 
           'my old email was billy@notucsd.edu, my new email is notbilly@ucsd.edu')
Out[20]:
['notucsd', 'ucsd']
  • Notice what happens if we remove the ( and )!
In [21]:
re.findall(r'\w+@\w+\.edu', 
           'my old email was billy@notucsd.edu, my new email is notbilly@ucsd.edu')
Out[21]:
['billy@notucsd.edu', 'notbilly@ucsd.edu']
  • Earlier, we also saw that parentheses can be used to group parts of a regex together. When using re.findall, all groups are treated as capturing groups.
In [22]:
# A regex that matches strings with two of the same vowel followed by 3 digits
# We only want to capture the digits, but...
re.findall(r'(aa|ee|ii|oo|uu)(\d{3})', 'eeoo124')
Out[22]:
[('oo', '124')]

Example: Log parsing¶

Web servers typically record every request made of them in the "logs".

In [23]:
s = '''132.249.20.188 - - [24/Feb/2023:12:26:15 -0800] "GET /my/home/ HTTP/1.1" 200 2585'''

Let's use our new regex syntax (including capturing groups) to extract the day, month, year, and time from the log string s.

In [24]:
exp = '\[(.+)\/(.+)\/(.+):(.+):(.+):(.+) .+\]'
re.findall(exp, s)
Out[24]:
[('24', 'Feb', '2023', '12', '26', '15')]

While above regex works, it is not very specific. It works on incorrectly formatted log strings.

In [25]:
other_s = '[adr/jduy/wffsdffs:r4s4:4wsgdfd:asdf 7]'
re.findall(exp, other_s)
Out[25]:
[('adr', 'jduy', 'wffsdffs', 'r4s4', '4wsgdfd', 'asdf')]

The more specific, the better!¶

  • Be as specific in your pattern matching as possible – you don't want to match and extract strings that don't fit the pattern you care about.
    • .* matches every possible string, but we don't use it very often.
  • A better date extraction regex:
\[(\d{2})\/([A-Z]{1}[a-z]{2})\/(\d{4}):(\d{2}):(\d{2}):(\d{2}) -\d{4}\]

- `\d{2}` matches any 2-digit number.
- `[A-Z]{1}` matches any single occurrence of any uppercase letter.
- `[a-z]{2}` matches any 2 consecutive occurrences of lowercase letters.
- Remember, special characters (`[`, `]`, `/`) need to be escaped with `\`.
In [26]:
s
Out[26]:
'132.249.20.188 - - [24/Feb/2023:12:26:15 -0800] "GET /my/home/ HTTP/1.1" 200 2585'
In [27]:
new_exp = '\[(\d{2})\/([A-Z]{1}[a-z]{2})\/(\d{4}):(\d{2}):(\d{2}):(\d{2}) -\d{4}\]'
re.findall(new_exp, s)
Out[27]:
[('24', 'Feb', '2023', '12', '26', '15')]

A benefit of new_exp over exp is that it doesn't capture anything when the string doesn't follow the format we specified.

In [28]:
other_s
Out[28]:
'[adr/jduy/wffsdffs:r4s4:4wsgdfd:asdf 7]'
In [29]:
re.findall(new_exp, other_s)
Out[29]:
[]

Exercise

^\w{2,5}.\d*\/[^A-Z5]{1,}

Select all strings below that contain any match with the regular expression above.

  • "billy4/Za"
  • "billy4/za"
  • "DAI_s2154/pacific"
  • "daisy/ZZZZZ"
  • "bi_/_lly98"
  • "!@__!14/atlantic"

Limitations of regular expressions¶

Writing a regular expression is like writing a program.

  • You need to know the syntax well.
  • They can be easier to write than to read.
  • They can be difficult to debug.

Regular expressions are terrible at certain types of problems. Examples:

  • Anything involving counting (same number of instances of a and b).
  • Anything involving complex structure (palindromes).
  • Parsing highly complex text structure (HTML, for instance).

Text features¶

No description has been provided for this image

Review: Regression and features¶

  • In DSC 40A, our running example was to use regression to predict a data scientist's salary, given their GPA, years of experience, and years of education.
  • After minimizing empirical risk to determine optimal parameters, $w_0^*, \dots, w_3^*$, we made predictions using:
$$\text{predicted salary} = w_0^* + w_1^* \cdot \text{GPA} + w_2^* \cdot \text{experience} + w_3^* \cdot \text{education}$$
  • GPA, years of experience, and years of education are features – they represent a data scientist as a vector of numbers.
    • e.g. Your feature vector may be [3.5, 1, 7].
  • This approach requires features to be numerical.

Moving forward¶

Suppose we'd like to predict the sentiment of a piece of text from 1 to 10.

  • 10: Very positive (happy).
  • 1: Very negative (sad, angry).

Example:

  • Input: "DSC 80 is a pretty good class."

  • Output: 7.

  • We can frame this as a regression problem, but we can't directly use what we learned in 40A, because here our inputs are text, not numbers.

Text features¶

  • Big question: How do we represent a text document as a feature vector of numbers?
  • If we can do this, we can:
    • use a text document as input in a regression or classification model (in a few lectures).
    • quantify the similarity of two text documents (today).

Example: San Diego employee salaries¶

  • Transparent California publishes the salaries of all City of San Diego employees.
  • Let's look at the 2022 data.
In [30]:
salaries = pd.read_csv('https://transcal.s3.amazonaws.com/public/export/san-diego-2022.csv')
salaries['Employee Name'] = salaries['Employee Name'].str.split().str[0] + ' Xxxx'
In [31]:
salaries.head()
Out[31]:
Employee Name Job Title Base Pay Overtime Pay ... Year Notes Agency Status
0 Mara Xxxx City Attorney 227441.53 0.00 ... 2022 NaN San Diego FT
1 Todd Xxxx Mayor 227441.53 0.00 ... 2022 NaN San Diego FT
2 Terence Xxxx Assistant Police Chief 227224.32 0.00 ... 2022 NaN San Diego FT
3 Esmeralda Xxxx Police Sergeant 124604.40 162506.54 ... 2022 NaN San Diego FT
4 Marcelle Xxxx Assistant Retirement Administrator 279868.04 0.00 ... 2022 NaN San Diego FT

5 rows × 13 columns

Aside on privacy and ethics¶

  • Even though the data we downloaded is publicly available, employee names still correspond to real people.
  • Be careful when dealing with PII (personably identifiable information).
    • Only work with the data that is needed for your analysis.
    • Even when data is public, people have a reasonable right to privacy.
  • Remember to think about the impacts of your work outside of your Jupyter Notebook.

Goal: Quantifying similarity¶

  • Our goal is to describe, numerically, how similar two job titles are.
  • For instance, our similarity metric should tell us that 'Deputy Fire Chief' and 'Fire Battalion Chief' are more similar than 'Deputy Fire Chief' and 'City Attorney'.
  • Idea: Two job titles are similar if they contain shared words, regardless of order. So, to measure the similarity between two job titles, let's count the number of words they share in common.
  • Before we do this, we need to be confident that the job titles are clean and consistent – let's explore.

Exploring job titles¶

In [32]:
jobtitles = salaries['Job Title']
jobtitles.head()
Out[32]:
0                         City Attorney
1                                 Mayor
2                Assistant Police Chief
3                       Police Sergeant
4    Assistant Retirement Administrator
Name: Job Title, dtype: object

How many employees are in the dataset? How many unique job titles are there?

In [33]:
jobtitles.shape[0], jobtitles.nunique()
Out[33]:
(12831, 611)

What are the most common job titles?

In [34]:
jobtitles.value_counts().iloc[:100]
Out[34]:
Police Officer Ii               1082
Police Sergeant                  311
Fire Fighter Ii                  306
                                ... 
Public Works Supervisor           29
Project Assistant                 29
Associate Engineer - Traffic      29
Name: Job Title, Length: 100, dtype: int64
In [ ]:
jobtitles.value_counts().iloc[:10].sort_values().plot(kind='barh')

Are there any missing job titles?

In [36]:
jobtitles.isna().sum()
Out[36]:
0

Fortunately, no.

Canonicalization¶

Remember, our goal is ultimately to count the number of shared words between job titles. But before we start counting the number of shared words, we need to consider the following:

  • Some job titles may have punctuation, like '-' and '&', which may count as words when they shouldn't.
    • 'Assistant - Manager' and 'Assistant Manager' should count as the same job title.
  • Some job titles may have "glue" words, like 'to' and 'the', which (we can argue) also shouldn't count as words.
    • 'Assistant To The Manager' and 'Assistant Manager' should count as the same job title.
  • If we just want to focus on the titles themselves, then perhaps roman numerals should be removed: that is, 'Police Officer Ii' and 'Police Officer I' should count as the same job title.

Let's address the above issues. The process of converting job titles so that they are always represented the same way is called canonicalization.

Punctuation¶

Are there job titles with unnecessary punctuation that we can remove?

  • To find out, we can write a regular expression that looks for characters other than letters, numbers, and spaces.

  • We can use regular expressions with the .str methods we learned earlier in the quarter just by using regex=True.

In [37]:
# Uses character class negation.
jobtitles.str.contains(r'[^A-Za-z0-9 ]', regex=True).sum()
Out[37]:
922
In [38]:
jobtitles[jobtitles.str.contains(r'[^A-Za-z0-9 ]', regex=True)].head()
Out[38]:
137          Park & Recreation Director
248     Associate Engineer - Mechanical
734     Associate Engineer - Electrical
882        Associate Engineer - Traffic
1045         Associate Engineer - Civil
Name: Job Title, dtype: object

It seems like we should replace these pieces of punctuation with a single space.

"Glue" words¶

Are there job titles with "glue" words in the middle, such as 'Assistant to the Manager'?

To figure out if any titles contain the word 'to', we can't just do the following, because it will evaluate to True for job titles that have 'to' anywhere in them, even if not as a standalone word.

In [39]:
# Why are we converting to lowercase?
jobtitles.str.lower().str.contains('to').sum()
Out[39]:
1577
In [40]:
jobtitles[jobtitles.str.lower().str.contains('to')]
Out[40]:
0                             City Attorney
4        Assistant Retirement Administrator
8                  Retirement Administrator
                        ...                
12778                        Test Monitor I
12812                           Custodian I
12826              Word Processing Operator
Name: Job Title, Length: 1577, dtype: object

Instead, we need to look for 'to' separated by word boundaries.

In [41]:
jobtitles.str.lower().str.contains(r'\bto\b', regex=True).sum()
Out[41]:
10
In [42]:
jobtitles[jobtitles.str.lower().str.contains(r'\bto\b', regex=True)]
Out[42]:
1638              Assistant To The Chief Operating Officer
2183                  Principal Assistant To City Attorney
2238                             Assistant To The Director
                               ...                        
6594     Confidential Secretary To Chief Operating Officer
6832                       Confidential Secretary To Mayor
11028                      Confidential Secretary To Mayor
Name: Job Title, Length: 10, dtype: object

We can look for other filler words too, like 'the' and 'for'.

In [43]:
jobtitles[jobtitles.str.lower().str.contains(r'\bthe\b', regex=True)]
Out[43]:
1638    Assistant To The Chief Operating Officer
2238                   Assistant To The Director
5609                   Assistant To The Director
6544                   Assistant To The Director
Name: Job Title, dtype: object
In [44]:
jobtitles[jobtitles.str.lower().str.contains(r'\bfor\b', regex=True)]
Out[44]:
3449     Assistant For Community Outreach
6889     Assistant For Community Outreach
10810    Assistant For Community Outreach
Name: Job Title, dtype: object

We should probably remove these "glue" words.

Roman numerals (e.g. "Ii")¶

Lastly, let's try and identify job titles that have roman numerals at the end, like 'i' (1), 'ii' (2), 'iii' (3), or 'iv' (4). As before, we'll convert to lowercase first.

In [45]:
jobtitles[jobtitles.str.lower().str.contains(r'\bi+v?\b', regex=True)]
Out[45]:
5                   Police Officer Ii
10                  Police Officer Ii
48       Fire Prevention Inspector Ii
                     ...             
12822           Clerical Assistant Ii
12828               Police Officer Ii
12830                Police Officer I
Name: Job Title, Length: 6087, dtype: object

Let's get rid of those numbers, too.

Fixing punctuation and removing "glue" words and roman numerals¶

Let's put the preceeding three steps together and canonicalize job titles by:

  • converting to lowercase,
  • removing each occurrence of 'to', 'the', and 'for',
  • replacing each non-letter/digit/space character with a space,
  • replacing each sequence of roman numerals – either 'i', 'ii', 'iii', or 'iv' at the end with nothing, and
  • replacing each sequence of multiple spaces with a single space.
In [46]:
jobtitles = (
    jobtitles
    .str.lower()
    .str.replace(r'\bto\b|\bthe\b|\bfor\b', '', regex=True)
    .str.replace(r'[^A-Za-z0-9 ]', ' ', regex=True)
    .str.replace(r'\bi+v?\b', '', regex=True)
    .str.replace(r' +', ' ', regex=True)               # ' +' matches 1 or more occurrences of a space.
    .str.strip()                                       # Removes leading/trailing spaces if present.
)
In [47]:
jobtitles.sample(5)
Out[47]:
2011       program manager
8150        police officer
10000    library assistant
9457        utility worker
7341        utility worker
Name: Job Title, dtype: object
In [48]:
(jobtitles == 'police officer').sum()
Out[48]:
1378

Possible issue: inconsistent representations¶

Another possible issue is that some job titles may have inconsistent representations of the same word (e.g. 'Asst.' vs 'Assistant').

In [49]:
jobtitles[jobtitles.str.contains('asst')].value_counts()
Out[49]:
Series([], Name: Job Title, dtype: int64)
In [50]:
jobtitles[jobtitles.str.contains('assistant')].value_counts().head()
Out[50]:
library assistant            385
assistant engineer civil     297
clerical assistant           111
assistant center director     50
assistant chemist             45
Name: Job Title, dtype: int64

The 2020 salaries dataset had several of these issues, but fortunately they appear to be fixed for us in the 2022 dataset (thanks, Transparent California).

Bag of words 💰¶

Text similarity¶

Recall, our idea is to measure the similarity of two job titles by counting the number of shared words between the job titles. How do we actually do that, for all of the job titles we have?

A counts matrix¶

Let's create a "counts" matrix, such that:

  • there is 1 row per job title,
  • there is 1 column per unique word that is used in job titles, and
  • the value in row title and column word is the number of occurrences of word in title.

Such a matrix might look like:

senior lecturer teaching professor assistant associate
senior lecturer 1 1 0 0 0 0
assistant teaching professor 0 0 1 1 1 0
associate professor 0 0 0 1 0 1
senior assistant to the assistant professor 1 0 0 1 2 0
  • Then, we can make statements like:
    • "assistant teaching professor" is more similar to "associate professor" than to "senior lecturer".
  • Next time!

Summary, next time¶

Summary¶

  • Regular expressions are used to match and extract patterns from text.
  • You don't need to force yourself to "memorize" regex syntax – refer to the resources in the Agenda section of the lecture and on the Resources tab of the course website.
  • Also refer to the three tables of syntax in the lecture:
    • Regex building blocks.
    • More regex syntax.
    • Even more regex syntax.
  • Note: You don't always have to use regular expressions! If Python/pandas string methods work for your task, you can still use those.
    • Play Regex Golf to practice! 🏌️
  • pandas .str methods can use regular expressions; just set regex=True.
  • One way to turn texts, like 'deputy fire chief', into feature vectors, is to count the number of occurrences of each word in the text, ignoring order. This is done using the bag of words model.

Next time¶

  • Implementing the bag of words model.
  • TF-IDF: an improvement on bag of words that considers the importance of each word.