import pandas as pd
import numpy as np
import os

import util

import matplotlib.pyplot as plt
plt.style.use('seaborn-white')
plt.rc('figure', dpi=100, figsize=(10, 5))
plt.rc('font', size=12)

Lecture 14 – HTTP Basics

DSC 80, Spring 2022

Announcements

• Midterm Exam grades are released! See #935 for details.
  • Regrades are due next Thursday.
• Project 2 is due tomorrow at 11:59PM.
• Lab 5 is due on Monday, May 2nd at 11:59PM.

Agenda

• Recap: Imputation.
• Introduction to HTTP.
• Making HTTP requests.
  • See the course notes for more details.

Recap: Imputation

Example: Heights 🧍📏

heights = pd.read_csv(os.path.join('data', 'heights.csv'))
heights = (
    heights
    .rename(columns={'childHeight': 'child', 'childNum': 'number'})
    .drop('midparentHeight', axis=1)
)
heights.head()

	family	father	mother	children	number	gender	child
0	1	78.5	67.0	4	1	male	73.2
1	1	78.5	67.0	4	2	female	69.2
2	1	78.5	67.0	4	3	female	69.0
3	1	78.5	67.0	4	4	female	69.0
4	2	75.5	66.5	4	1	male	73.5

np.random.seed(42) # So that we get the same results each time (for lecture)
heights_mcar = util.make_mcar(heights, 'child', pct=0.50)
heights_mcar.head()

	family	father	mother	children	number	gender	child
0	1	78.5	67.0	4	1	male	73.2
1	1	78.5	67.0	4	2	female	69.2
2	1	78.5	67.0	4	3	female	NaN
3	1	78.5	67.0	4	4	female	NaN
4	2	75.5	66.5	4	1	male	73.5

Mean imputation

The 'child' column has missing values.

• If 'child' is MCAR, then fill in each of the missing values using the mean of the observed values.

• If 'child' is MAR dependent on a categorical column, then fill in each of the missing values using the mean of the observed values in each category. For instance, if 'child' is MAR dependent on 'gender', we can fill in:
  • missing female 'child' heights with the observed mean for female children, and
  • missing male 'child' heights with the observed mean for male children.

• If 'child' is MAR dependent on a numerical column, then bin the numerical column to make it categorical, then follow the procedure above. See Lab 5, Question 3!

• Mean imputation, when done correctly, creates a distribution whose mean is an unbiased estimate of the true distribution's mean, but whose variance is an underestimate of the true variance.

heights_mcar_mfilled = heights_mcar.fillna(heights_mcar['child'].mean())
heights_mcar_mfilled['child'].head()

0    73.200000
1    69.200000
2    66.640685
3    66.640685
4    73.500000
Name: child, dtype: float64

plt.hist([heights['child'], heights_mcar['child'].dropna(), heights_mcar_mfilled['child']])
plt.legend(['full data', 'missing (mcar)', 'imputed']);

Probabilistic imputation

The 'child' column has missing values.

• If 'child' is MCAR, then fill in each of the missing values with randomly selected observed 'child' heights.
  • For instance, if there are 5 missing 'child' values, pick 5 of the not-missing 'child' values.
• If 'child' is MAR dependent on a categorical column, sample from the observed values separately for each category.

# Figure out the number of missing values
num_null = heights_mcar['child'].isna().sum()

# Sample that number of values from the observed dataset
fill_values = heights_mcar['child'].dropna().sample(num_null, replace=True)

# Find the positions where values in heights_mcar are missing
fill_values.index = heights_mcar.loc[heights_mcar['child'].isna()].index

# Fill in the missing values
heights_mcar_dfilled = heights_mcar.fillna({'child': fill_values.to_dict()})  # fill the vals

plt.hist([heights['child'], heights_mcar['child'], heights_mcar_dfilled['child']], density=True);
plt.legend(['full data','missing (mcar)', 'distr imputed']);

No spikes!

Observations

• With probabilistic imputation, the missing values were filled in with observed values in the dataset.

• If a value was never observed in the dataset, it will never be used to fill in a missing value.

  • For instance, if the observed heights were 68, 69, and 69.5 inches, we will never fill a missing value with 68.5 inches even though it's a perfectly reasonable height.

• Solution? Create a histogram (with np.histogram) to bin the data, then sample from the histogram. See Lab 5, Question 4.

Randomness

• Unlike mean imputation, probabilistic imputation is random – each time you run the cell in which imputation is performed, the results could be different.
• Multiple imputation: Generate multiple imputed datasets and aggregate the results!
  • Similar to bootstrapping.

Multiple imputation

Steps:

1. Start with observed and incomplete data.

2. Create several imputed versions of the data through a probabilistic procedure.

   • The imputed datasets are identical for the observed data entries.
   • They differ in the imputed values.
   • The differences reflect our uncertainty about what value to impute.

3. Then, estimate the parameters of interest for each imputed dataset.

   • For instance, the mean, standard deviation, median, etc.
4. Finally, pool the parameter estimates into one estimate.

Let's try this procedure out on the heights_mcar dataset.

heights_mcar.head()

	family	father	mother	children	number	gender	child
0	1	78.5	67.0	4	1	male	73.2
1	1	78.5	67.0	4	2	female	69.2
2	1	78.5	67.0	4	3	female	NaN
3	1	78.5	67.0	4	4	female	NaN
4	2	75.5	66.5	4	1	male	73.5

# This function implements the 3-step process we studied earlier
def create_imputed(col):
    num_null = col.isna().sum()
    fill_values = col.dropna().sample(num_null, replace=True)
    fill_values.index = col.loc[col.isna()].index
    return col.fillna(fill_values.to_dict())

Each time we run the following cell, it generates a new imputed version of the 'child' column.

create_imputed(heights_mcar['child']).head()

0    73.2
1    69.2
2    67.7
3    66.0
4    73.5
Name: child, dtype: float64

Let's run the above procedure 100 times.

mult_imp = pd.concat([create_imputed(heights_mcar['child']).rename(k) for k in range(100)], axis=1)
mult_imp.head()

	0	1	2	3	4	5	6	7	8	9	...	90	91	92	93	94	95	96	97	98	99
0	73.2	73.2	73.2	73.2	73.2	73.2	73.2	73.2	73.2	73.2	...	73.2	73.2	73.2	73.2	73.2	73.2	73.2	73.2	73.2	73.2
1	69.2	69.2	69.2	69.2	69.2	69.2	69.2	69.2	69.2	69.2	...	69.2	69.2	69.2	69.2	69.2	69.2	69.2	69.2	69.2	69.2
2	64.5	72.0	67.0	69.0	69.0	70.0	69.0	67.0	67.0	64.0	...	65.7	65.0	65.7	70.7	73.0	65.0	66.0	63.0	71.0	69.0
3	63.5	64.5	61.7	67.2	61.0	64.0	66.0	56.0	60.0	62.0	...	67.2	64.5	68.0	62.0	64.0	62.0	66.0	65.5	69.0	64.5
4	73.5	73.5	73.5	73.5	73.5	73.5	73.5	73.5	73.5	73.5	...	73.5	73.5	73.5	73.5	73.5	73.5	73.5	73.5	73.5	73.5

5 rows × 100 columns

Let's plot some of the imputed columns above.

# Random sample of 15 imputed columns
mult_imp.sample(15, axis=1).plot(kind='kde', alpha=0.5, legend=False);

Let's look at the distribution of means across the imputed columns.

mult_imp.mean().plot(kind='hist', bins=20, ec='w', density=True);

Summary of imputation techniques

See the end of Lecture 13 for a detailed summary of all imputation techniques that we've seen so far.

Introduction to HTTP

Collecting data

• Often, the data you need doesn't exist in "clean" .csv files.
• Solution: collect your own data!
  • Design and administer your own survey or run an experiment.
  • Find related data on the internet.

Data on the internet

• The internet contains massive amounts of historical record.
  • News stories provide a record of world events.
  • Social networks and commerce sites provide an insight into human behavior.
• For most questions you can think of, the data to answer the question exists somewhere on the internet.

Collecting data from the internet

• There are two ways to programmatically access data on the internet.
  • Through an API.
  • By scraping.
• We will discuss the differences between both approaches next lecture, but for now, the important part is that they both use HTTP.

HTTP

• HTTP stands for Hypertext Transfer Protocol.
  • It was developed in 1989 by Tim Berners-Lee (and friends).
• It is a request-response protocol.
  • Protocol = set of rules.
• HTTP allows...
  • computers to talk to each other over a network.
  • devices to fetch data from "web servers".
• The "S" in HTTPS stands for "secure".

UCSD was a node in ARPANET, the predecessor to the modern internet (source).

The request-response model

HTTP follows the request-response model.

• A request is made by the client.
• A response is returned by the server.

• Example: YouTube 🎥.
  • Your phone's web browser, a client, makes an HTTP request to view a video.
  • The server, YouTube, is a computer that is sitting somewhere else.
  • The server returns a response that contains the video.

Request methods

• The request methods you will use most often are GET and POST.

  • GET is used to request data from a specified resource.

  • POST is used to send data to the server.

    • e.g. uploading a photo to Instagram or entering credit card information on Amazon.

• See Mozilla's web docs for a detailed list of request methods.

Example GET request

Below is an example GET HTTP request made by a browser when accessing datascience.ucsd.edu.

GET / HTTP/1.1
Host: datascience.ucsd.edu
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36
Connection: keep-alive
Accept-Language: en-US,en;q=0.9

• The first line (GET / HTTP/1.1) is called the "request line", and the lines afterwards are called "header fields". We could also provide a "body" after the header fields.
• To see HTTP requests in Google Chrome, follow these steps.

Example GET response

The response below was generated by executing the request on the previous slide.

HTTP/1.1 200 OK
Date: Fri, 29 Apr 2022 02:54:41 GMT
Server: Apache
Link: <https://datascience.ucsd.edu/wp-json/>; rel="https://api.w.org/"
Link: <https://datascience.ucsd.edu/wp-json/wp/v2/pages/2427>; rel="alternate"; type="application/json"
Link: <https://datascience.ucsd.edu/>; rel=shortlink
Content-Type: text/html; charset=UTF-8

<!DOCTYPE html>
<html lang="en-US">
<head>
    <meta charset="UTF-8">
    <link rel="profile" href="https://gmpg.org/xfn/11">
    <style media="all">img.wp-smiley,img.emoji{display:inline !important;border:none
...

Consequences of the request-response model

• When a request is sent to view content on a webpage, the server must:
  • process your request (i.e. prepare data for the response).
  • send content back to the client in its response.
• Remember, servers are computers.
  • Someone has to pay to keep these computers running.
  • This means that every time you access a website, someone has to pay.

Example: istheshipstuck.com

Read Inside a viral website, an account of what it's like to run a site that gained 50 million+ views in 5 days.

Making HTTP requests

Making HTTP requests

There are (at least) two ways to make HTTP requests:

• From the command line, with curl.
• From Python, with the requests package.

Making HTTP requests using curl

curl is a command-line tool that sends HTTP requests, like a browser.

1. The client, curl, sends a HTTP request.
2. The request contains a method (e.g. GET or POST).
3. The HTTP server responds with
   • a status line, indicating if things went well,
   • response headers, and
   • (usually) a response body, containing the requested data.

Example: GET requests via curl

• By default, curl issues a GET request.
• Remember, you can run command-line commands in a Jupyter Notebook by placing a ! before them.

curl -v https://httpbin.org/html
# (`-v` is short for verbose)

• After running the command, go to https://httpbin.org/html in your browser. What do you notice?

!curl -v https://httpbin.org/html

Queries in a GET request

• In order to request more specific information, we can include a query string in the URL.
• ? begins a query. For instance,

https://www.google.com/search?q=ucsd+dsc+80+hard&client=safari

• This method works well when sending small amounts of data.
• We will use a similiar technique when working with APIs next lecture.

Example: POST requests via curl

• When using curl, -d is short for POST.
• Below is an example curl POST request that sends 'King Triton' as the parameter 'name'.

curl -d 'name=King Triton' https://httpbin.org/post

!curl -d 'name=King Triton' https://httpbin.org/post

• Run the cell below. Notice the difference?

!curl -d 'name=King Triton' https://youtube.com

Making HTTP requests using requests

• requests is a Python package that allows you to use Python to interact with the internet!
• There are other packages that work similarly (e.g. urllib), but requests is arguably the easiest to use.

import requests

Example: GET requests via requests

To access the source code of the UCSD home page, all we need to run is the following:

text = requests.get('https://ucsd.edu').text

url = 'https://ucsd.edu'
resp = requests.get(url)

resp is now a Response object.

resp

<Response [200]>

The text attribute of resp is a string that containing the entire response.

type(resp.text)

str

len(resp.text)

43692

print(resp.text[:1000])

<!DOCTYPE html>
<html lang="en">
  <head>
  
  

 





    <meta charset="utf-8"/>
    <meta content="IE=edge" http-equiv="X-UA-Compatible"/>
    <meta content="width=device-width, initial-scale=1" name="viewport"/>
    <title>University of California San Diego</title>
    <meta content="University of California, San Diego" name="ORGANIZATION"/>
    <meta content="index,follow,noarchive" name="robots"/>
    <meta content="UCSD" name="SITE"/>
    <meta content="University of California San Diego" name="PAGETITLE"/>
    <meta content="The University California San Diego is one of the world's leading public research universities, located in beautiful La Jolla, California" name="DESCRIPTION"/>
    <link href="favicon.ico" rel="icon"/>


    
  




<!-- Site-specific CSS files -->
    
  <link href="https://www.ucsd.edu/_resources/css/vendor/brix_sans.css" rel="stylesheet" type="text/css"/>
  
  <!-- CSS complied from style overrides -->
  <link href="https://www.ucsd.edu/_resources/css/s

The url attribute contains the URL that we accessed.

resp.request.url

'https://ucsd.edu/'

Example: POST requests via requests

post_response = requests.post('https://httpbin.org/post',
                              data={'name': 'King Triton'})
post_response

<Response [200]>

print(post_response.text)

{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "name": "King Triton"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate, br", 
    "Content-Length": "16", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.26.0", 
    "X-Amzn-Trace-Id": "Root=1-626b9412-7d96f8fc5ad980b61f34ae79"
  }, 
  "json": null, 
  "origin": "70.95.172.151", 
  "url": "https://httpbin.org/post"
}

HTTP status codes

• When we request data from a website, the server includes an HTTP status code in the response.
• The most common status code is 200, which means there were no issues.
• Other times, you will see a different status code, describing some sort of event or error.
  • e.g. 404: page not found; 500: internal server error.
  • The first digit of a status describes its general "category".
• See https://httpstat.us for a list of all HTTP status codes.
  • It also has example sites for each status code.
  • For example, https://httpstat.us/404 returns a 404.

r = requests.get('https://httpstat.us/503')
print(r.status_code)

503

r.text

'503 Service Unavailable'

Successful requests ✅

• You can check if a request was successful using the ok attribute, which returns a bool.
  • If a status is in the 200s, then it is successful.
• Unsuccessful requests can be re-tried, depending on the issue.
  • Wait a little, then try the request again.
  • You can even re-try requests programmatically (e.g. using a loop). If rate of requests is too high, slow down requests between each retry (e.g. using time.sleep).
• See the course notes for more examples.

status_codes = [200, 201, 403, 404, 503]

for code in status_codes:
    r = requests.get(f'https://httpstat.us/{code}')
    print(f'{code} ok: {r.ok}')

200 ok: True
201 ok: True
403 ok: False
404 ok: False
503 ok: False

• The raise_for_status request method raises an exception when the status code is not-ok.

requests.get('https://httpstat.us/400').raise_for_status()

---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
/var/folders/pd/w73mdrsj2836_7gp0brr2q7r0000gn/T/ipykernel_8477/2094305732.py in <module>
----> 1 requests.get('https://httpstat.us/400').raise_for_status()

~/opt/anaconda3/lib/python3.9/site-packages/requests/models.py in raise_for_status(self)
    951 
    952         if http_error_msg:
--> 953             raise HTTPError(http_error_msg, response=self)
    954 
    955     def close(self):

HTTPError: 400 Client Error: Bad Request for url: https://httpstat.us/400

The data formats of the internet

The internet currently relies on two key data formats – HTML and JSON.

• The response body of a GET request is usually either JSON (when using an API) or HTML (when accessing a webpage).
• The response body of a POST request is usually JSON.

JSON

• JSON stands for JavaScript Object Notation.
• It is a lightweight format for storing and transferring data.
• It is:
  • very easy for computers to read and write.
  • moderately easy for programmers to read and write by hand.
  • meant to be generated and parsed.
• Most modern languages have an interface for working with JSON objects.
  • JSON objects resemble Python dictionaries (but are not the same!).
• JSON is slowly replacing XML, another text-based format for sending data from/to servers.

JSON data types

• string: anything inside double quotes.
• number: any number (no difference between ints and floats).
• boolean: true and false.
• array: anything wrapped in [].
• null: JSON's empty value, denoted by null.
• object: a collection of key-value pairs (like dictionaries).
  • Keys must be strings, values can be anything (even other objects).

See json-schema.org for more details.

Example JSON object

See data/family.json.

import json

f = open(os.path.join('data', 'family.json'), 'r')
family_tree = json.load(f)

family_tree

{'name': 'Grandma',
 'age': 94,
 'children': [{'name': 'Dad',
   'age': 60,
   'children': [{'name': 'Me', 'age': 23}, {'name': 'Brother', 'age': 21}]},
  {'name': 'My Aunt',
   'children': [{'name': 'Cousin 1', 'age': 34},
    {'name': 'Cousin 2',
     'age': 36,
     'children': [{'name': 'Cousin 2 Jr.', 'age': 2}]}]}]}

family_tree['children'][0]['children'][0]['age']

23

Aside: eval

• eval, which stands for "evaluate", is a function built into Python.
• It takes in a string containing a Python expression and evaluates it in the current context.

x = 4

eval('x + 5')

9

• It seems like eval can do the same thing that json.load does...

f = open(os.path.join('data', 'family.json'), 'r')
eval(f.read())

{'name': 'Grandma',
 'age': 94,
 'children': [{'name': 'Dad',
   'age': 60,
   'children': [{'name': 'Me', 'age': 23}, {'name': 'Brother', 'age': 21}]},
  {'name': 'My Aunt',
   'children': [{'name': 'Cousin 1', 'age': 34},
    {'name': 'Cousin 2',
     'age': 36,
     'children': [{'name': 'Cousin 2 Jr.', 'age': 2}]}]}]}

• But you should never use eval. The next slide demonstrates why.

eval gone wrong

• Observe what happens when we use eval on a string representation of a JSON object:

f_other = open(os.path.join('data', 'evil_family.json'))
eval(f_other.read())

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/var/folders/pd/w73mdrsj2836_7gp0brr2q7r0000gn/T/ipykernel_8477/3392341705.py in <module>
      1 f_other = open(os.path.join('data', 'evil_family.json'))
----> 2 eval(f_other.read())

<string> in <module>

~/Desktop/80/private/lectures/sp22/lec14/util.py in err()
    132 # For JSON evaluation example
    133 def err():
--> 134     raise ValueError('i just deleted all your files lol 😂')

ValueError: i just deleted all your files lol 😂

• Oh no! Since evil_family.json, which could have been downloaded from the internet, contained malicious code, we now lost all of our files.
• This happened because eval evaluates all parts of the input string as if it were Python code. You never need to do this – instead, use the json library.
  • json.load loads a JSON file from a file.
  • json.loads loads a JSON file from a string.

f_other = open(os.path.join('data', 'evil_family.json'))
s = f_other.read()
s

'{\n    "name": "Grandma",\n    "age": 94,\n    "children": [\n        {\n        "name": util.err(),\n        "age": 60,\n        "children": [{"name": "Me", "age": 23}, \n                     {"name": "Brother", "age": 21}]\n        },\n        {\n        "name": "My Aunt",\n        "children": [{"name": "Cousin 1", "age": 34}, \n                     {"name": "Cousin 2", "age": 36, "children": \n                        [{"name": "Cousin 2 Jr.", "age": 2}]\n                     }\n                    ]\n        }\n    ]\n}'

json.loads(s)

---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
/var/folders/pd/w73mdrsj2836_7gp0brr2q7r0000gn/T/ipykernel_8477/1938830664.py in <module>
----> 1 json.loads(s)

~/opt/anaconda3/lib/python3.9/json/__init__.py in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    344             parse_int is None and parse_float is None and
    345             parse_constant is None and object_pairs_hook is None and not kw):
--> 346         return _default_decoder.decode(s)
    347     if cls is None:
    348         cls = JSONDecoder

~/opt/anaconda3/lib/python3.9/json/decoder.py in decode(self, s, _w)
    335 
    336         """
--> 337         obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    338         end = _w(s, end).end()
    339         if end != len(s):

~/opt/anaconda3/lib/python3.9/json/decoder.py in raw_decode(self, s, idx)
    353             obj, end = self.scan_once(s, idx)
    354         except StopIteration as err:
--> 355             raise JSONDecodeError("Expecting value", s, err.value) from None
    356         return obj, end

JSONDecodeError: Expecting value: line 6 column 17 (char 84)

• Since util.err() is not a string in JSON (there are no quotes around it), json.loads is not able to parse it as a JSON object.
• This "safety check" is intentional.

Handling unfamiliar data

• Never trust data from an unfamiliar site.
• Never use eval on "raw" data that you didn't create!
• The JSON data format needs to be parsed, not evaluated as a dictionary.
  • It was designed with safety in mind!

Summary, next time

Summary

• HTTP is the protocol the internet uses for transferring information.
• Clients can make GET HTTP requests to ask for information and POST HTTP requests to send information.
• Servers send responses with the desired information.
• We can use curl in the command-line or the requests Python package to make HTTP requests.
• The two main file formats used for storing information on the internet are HTML and JSON.
• JSON objects resemble Python dictionaries, but they are not quite the same. Use the json package to parse them, not eval.
• Next time: Using HTTP to make API requests and scrape the web. Parsing HTML files.