```
from dsc80_utils import *
```

```
# Used for plotting examples.
def create_kde_plotly(df, group_col, group1, group2, vals_col, title=''):
fig = ff.create_distplot(
hist_data=[df.loc[df[group_col] == group1, vals_col], df.loc[df[group_col] == group2, vals_col]],
group_labels=[group1, group2],
show_rug=False, show_hist=False,
colors=['#ef553b', '#636efb'],
)
return fig.update_layout(title=title)
```

## 📣 Announcements 📣¶

- Good job on Lab 3!
- Project 2 checkpoint due tomorrow.
- Lab 4 due Monday.
- Midterm exam will happen next week on Thurs Nov 2.

## 📝 Midterm Exam¶

- Thurs, Nov 2 from 3:30-4:50pm in WLH 2005.
- Pen and paper only. No calculators, phones, or watches allowed.
- You are allowed to bring one double-sided 8.5" x 11" sheet of handwritten notes.
- No reference sheet given, unlike DSC 10!

- We will display clarifications and the time remaining during the exam.

## 📆 Agenda¶

- Why do data go missing?
- Mechanisms: Missing by design; Not Missing at Random; Missing Completely at Random; Missing at Random
- Deciding between MCAR and MAR using a hypothesis test

**Additional resources:**

- Permutation testing:
- Extra lecture notebook: Fast Permutation Tests.
- Great visualization.

- Missingness mechanisms:

## Review: Hypothesis testing¶

(Question 6 from DSC 10 Su22 Final)

Every year, the American Kennel Club holds a Photo Contest for dogs. Eric wants to know whether **toy dogs win disproportionately more often than other kinds of dogs**. He has collected a sample of 500 dogs that have won the Photo Contest. In his sample, 200 dogs were toy dogs.

Eric also knows the distribution of dog kinds in the population:

Suppose he has the following null and alternative hypotheses:

- Null: The proportion of toy dogs that win is 0.3.
- Alternative: The proportion of toy dogs that win is greater than 0.3.

Select all the test statistics that Eric can use to conduct his hypothesis.

- The proportion of toy dogs in his sample.
- The number of toy dogs in his sample.
- The absolute difference of the sample proportion of toy dogs and 0.3.
- The absolute difference of the sample proportion of toy dogs and 0.5.
- The TVD between his sample and the population.

## Missingness mechanisms¶

### Imperfect data¶

When studying a problem, we are interested in understanding the

**true model**in nature.The data generating process is the "real-world" version of the model, that generates the data that we observe.

The recorded data is

**supposed**to "well-represent" the data generating process, and subsequently the true model.Example: Consider the upcoming Midterm Exam.

- The exam is meant to be a
**model**of your**true**knowledge of DSC 80 concepts. - The data generating process should give us a sense of your true knowledge, but is influenced by the specific questions on the exam, your preparation for the exam, whether or not you are sick on the day of the exam, etc.
- The recorded data consists of the final answers you write on the exam page.

- The exam is meant to be a

### Imperfect data¶

**Problem 1:**Your data is not representative, i.e. you have a poor access frame and/or collected a poor sample.- If the exam only asked questions about
`pivot_table`

, that would not give us an accurate picture of your understanding of DSC 80!

- If the exam only asked questions about
**Problem 2:**Some of the entries are missing.- If you left some questions blank, why?

We will focus on the second problem.

### Types of missingness¶

There are four key ways in which values can be missing. It is important to distinguish between these types since in some cases, we can **impute** (fill in) the missing data.

**Missing by design (MD)**.**Not missing at random (NMAR)**.- Also called "non-ignorable" (NI).

**Missing at random (MAR)**.**Missing completely at random (MCAR)**.

### Missing by design (MD)¶

Values in a column are missing by design if:

- the designers of the data collection process
**intentionally decided to not collect data in that column**, - because it can be recovered from other columns.

- the designers of the data collection process
If you can determine whether a value is missing solely using other columns, then the data is missing by design.

- For example:
`'Age4'`

is missing if and only if`'Number of People'`

is less than 4.

- For example:

- Refer to this StackExchange link for more examples.

### Missing by design¶

**Example**: `'Car Type?'`

and `'Car Colour?'`

are missing if and only if `'Own a car?'`

is `'No'`

.

### Other types of missingness¶

Not missing at random (NMAR).

- The chance that a value is missing
**depends on the actual missing value**!

- The chance that a value is missing
Missing completely at random (MCAR).

- The chance that a value is missing is
**completely independent**of- other columns, and
- the actual missing value.

- The chance that a value is missing is
Missing at random (MAR).

- The chance that a value is missing
**depends on other columns**, but**not**the actual missing value itself. - If a column is MAR, then it is MCAR
**when conditioned on some set of other columns**.

- The chance that a value is missing

### Mom... the dog ate my data! 🐶¶

Consider the following (contrived) example:

- We survey 100 people for their favorite color and birth month.
- We write their answers on index cards.
- On the left side, we write colors.
- On the right side, we write birth months 📆.

- A bad dog takes the top 10 cards from the stack and chews off the right side (birth months).
- Now ten people are missing birth months!

### Discussion Question¶

We are now missing birth months for the first 10 people we surveyed. What is the missingness mechanism for birth months if:

- Cards were sorted by favorite color?
- Cards were sorted by birth month?
- Cards were shuffled?

Remember:

**Not missing at random (NMAR)**: The chance that a value is missing**depends on the actual missing value**!**Missing at random (MAR)**: The chance that a value is missing**depends on other columns**, but**not**the actual missing value itself.**Missing completely at random (MCAR)**: The chance that a value is missing is**completely independent**of other columns and the actual missing value.

### Discussion Question, solved¶

If cards were sorted by favorite color, then:

- The fact that a card is missing a month is
**related to the favorite color**. - Since the missingness depends on another column, we say values are
**missing at random (MAR)**.- The missingness doesn't depend on the actual missing values – if we fix a particular color, early months are no more likely to be missing than later months.

- The fact that a card is missing a month is
If cards were sorted by birth month, then:

- The fact that a card is missing a month is
**related to the missing month**. - Since the missingness depends on the actual missing values – early months are more likely to be missing than later months – we say values are
**not missing at random (NMAR)**.

- The fact that a card is missing a month is
If cards were shuffled, then:

- The fact that a card is missing a month is
**related to nothing**. - Since the missingness depends on nothing, we say values are
**missing completely at random (MCAR)**.

- The fact that a card is missing a month is

### The real world is messy! 🌎¶

- In our contrived example, the distinction between NMAR, MAR, and MCAR was relatively clear.
- However, in more practical examples, it can be hard to distinguish between types of missingness.
- Domain knowledge is often needed to understand
**why**values might be missing.

### Not missing at random (NMAR)¶

Data is NMAR if the chance that a value is missing

**depends on the actual missing value**!- It could
*also*depend on other columns.

- It could
Another term for NMAR is "non-ignorable" – the fact that data is missing is data in and of itself that we cannot ignore.

**Example:**On an employment survey, people with really high incomes may be less likely to report their income.- If we
**ignore**missingness and compute the mean salary, our result will be**biased**low!

- If we
**Example:**A person doesn't take a drug test because they took drugs the day before.When data is NMAR, we must reason about why the data is missing using domain expertise on the data generating process – the other columns in our data won't help.

### Missing completely at random (MCAR)¶

Data is MCAR if the chance that a value is missing is

**completely independent**of other columns and the actual missing value.**Example:**After the Midterm Exam, I accidentally spill boba on the top of the stack. Assuming that the exams are in a random order, the exam scores that are lost due to this still will be MCAR. (Hopefully this doesn't happen!)

### Missing at random (MAR)¶

Data is MAR if the chance that a value is missing

**depends on other columns**, but**not**the actual missing value itself.**Example:**People who work in the service industry may be less likely to report their income.- If you look at service industry workers only, there is no pattern to the missingness of income (MCAR).
- If you look at corporate workers only, there is no pattern to the missingness of income (MCAR).
- Within each industry, missingness is MCAR, but overall, it is MAR, since the
**missingness of income depends on industry**.

**Example:**An elementary school teacher keeps track of the health conditions of each student in their class. One day, a student doesn't show up for a test because they are at the hospital.- The fact that their test score is missing has nothing to do with the test score itself.
- But the teacher could have predicted that the score would have been missing given the other information they had about the student.

### Isn't everything NMAR? 🤔¶

- You can argue that many of these examples are NMAR, by arguing that the missingness depends on the value of the data that is missing.
- For example, if a student is hospitalized, they may have lots of health problems and may not have spent much time on school, leading to their test scores being worse.

- Fair point, but with that logic
*almost everything is NMAR*. - What we really care about is
**the main reason data is missing**. - If the other columns
**mostly**explain the missing value and missingness, treat it as MAR. **In other words, accounting for potential confounding variables makes NMAR data more like MAR – big part of experimental design.**

### Flowchart¶

A good strategy is to assess missingness in the following order.

**Missing by design (MD)**

*Can I determine the missing value exactly by looking at the other columns?*🤔

**Not missing at random (NMAR)**

*Is there a good reason why the missingness depends on the values themselves?*🤔

**Missing at random (MAR)**

*Do other columns tell me anything about the likelihood that a value is missing?*🤔

**Missing completely at random (MCAR)**

*The missingness must not depend on other columns or the values themselves.*😄

### Discussion Question¶

In each of the following examples, decide whether the missing data are likely to be MD, NMAR, MAR, or MCAR:

- A table for a medical study has columns for
`'gender'`

and`'age'`

..`'age'`

has missing values - Measurements from the Hubble Space Telescope are
**dropped during transmission**. - A table has a single column,
`'self-reported education level'`

,**which contains missing values**. - A table of grades contains three columns,
`'Version 1'`

,`'Version 2'`

, and`'Version 3'`

.**$\frac{2}{3}$ of the entries in the table are**`NaN`

.

### Why do we care again?¶

- If a dataset contains missing values, it is likely not an accurate picture of the data generating process.
- By identifying missingness mechanisms, we can best
**fill in**missing values, to gain a better understanding of the DGP.

## Formal definitions¶

We won't spend much time on these in lecture, but you may find them helpful.

### Formal definition: MCAR¶

Suppose we have:

- A dataset $Y$ with observed values $Y_{obs}$ and missing values $Y_{mis}$.
- A parameter $\psi$ that represents all relevant information that is not part of the dataset.

Data is **missing completely at random** (MCAR) if

That is, adding information about the dataset doesn't change the likelihood data is missing!

### Formal definition: MAR¶

Suppose we have:

- A dataset $Y$ with observed values $Y_{obs}$ and missing values $Y_{mis}$.
- A parameter $\psi$ that represents all relevant information that is not part of the dataset.

Data is **missing at random** (MCAR) if

That is, MAR data is **actually MCAR**, **conditional** on $Y_{obs}$.

### Formal definition: NMAR¶

Suppose we have:

- A dataset $Y$ with observed values $Y_{obs}$ and missing values $Y_{mis}$.
- A parameter $\psi$ that represents all relevant information that is not part of the dataset.

Data is **not missing at random** (NMAR) if

cannot be simplified. That is, in NMAR data, **missingness is dependent on the missing value** itself.

## Assessing missingness through data¶

### Assessing missingness through data¶

- Suppose I believe that the missingness mechanism of a column is NMAR, MAR, or MCAR.
- I've ruled out missing by design (a good first step).

- Can I check whether this is true, by looking at the data?

### Assessing NMAR¶

We can't determine if data is NMAR just by looking at the data, as whether or not data is NMAR depends on the

**unobserved data**.To establish if data is NMAR, we must:

**reason about the data generating process**, or- collect more data.

**Example:**Consider a dataset of survey data of students' self-reported happiness. The data contains PIDs and happiness scores; nothing else. Some happiness scores are missing.**Are happiness scores likely NMAR?**

### Assessing MAR¶

- Data are MAR if the missingness only depends on
**observed**data. - After reasoning about the data generating process, if you establish that data is not NMAR, then it must be either MAR or MCAR.
- The more columns we have in our dataset, the "weaker the NMAR effect" is.
- Adding more columns -> controlling for more variables -> moving from NMAR to MAR.
**Example:**With no other columns, income in a census is NMAR. But once we look at location, education, and occupation, incomes are closer to being MAR.

### Deciding between MCAR and MAR¶

For data to be MCAR, the chance that values are missing should not depend on any other column or the values themselves.

**Example:**Consider a dataset of phones, in which we store the screen size and price of each phone.**Some prices are missing.**

Phone | Screen Size | Price |
---|---|---|

iPhone 14 | 6.06 | 999 |

Galaxy Z Fold 4 | 7.6 | NaN |

OnePlus 9 Pro | 6.7 | 799 |

iPhone 13 Pro Max | 6.68 | NaN |

If prices are MCAR, then

**the distribution of screen size should be the same**for:- phones whose prices are missing, and
- phones whose prices aren't missing.

**We can use a permutation test to decide between MAR and MCAR!**We are asking the question, did these two samples come from the same underlying distribution?

### Discussion Question¶

In each of the following examples, decide whether the missing data are likely to be MD, NMAR, MAR, or MCAR:

- A table for a medical study has columns for
`'gender'`

and`'age'`

..`'age'`

has missing values - Measurements from the Hubble Space Telescope are
**dropped during transmission**. - A table has a single column,
`'self-reported education level'`

,**which contains missing values**. - A table of grades contains three columns,
`'Version 1'`

,`'Version 2'`

, and`'Version 3'`

.**$\frac{2}{3}$ of the entries in the table are**`NaN`

.

### Why do we care again?¶

- If a dataset contains missing values, it is likely not an accurate picture of the data generating process.
- By identifying missingness mechanisms, we can best
**fill in**missing values, to gain a better understanding of the DGP.

## Formal definitions¶

We won't spend much time on these in lecture, but you may find them helpful.

### Formal definition: MCAR¶

Suppose we have:

- A dataset $Y$ with observed values $Y_{obs}$ and missing values $Y_{mis}$.
- A parameter $\psi$ that represents all relevant information that is not part of the dataset.

Data is **missing completely at random** (MCAR) if

That is, adding information about the dataset doesn't change the likelihood data is missing!

### Formal definition: MAR¶

Suppose we have:

- A dataset $Y$ with observed values $Y_{obs}$ and missing values $Y_{mis}$.
- A parameter $\psi$ that represents all relevant information that is not part of the dataset.

Data is **missing at random** (MCAR) if

That is, MAR data is **actually MCAR**, **conditional** on $Y_{obs}$.

### Formal definition: NMAR¶

Suppose we have:

- A dataset $Y$ with observed values $Y_{obs}$ and missing values $Y_{mis}$.
- A parameter $\psi$ that represents all relevant information that is not part of the dataset.

Data is **not missing at random** (NMAR) if

cannot be simplified. That is, in NMAR data, **missingness is dependent on the missing value** itself.

## Assessing missingness through data¶

### Assessing missingness through data¶

- Suppose I believe that the missingness mechanism of a column is NMAR, MAR, or MCAR.
- I've ruled out missing by design (a good first step).

- Can I check whether this is true, by looking at the data?

### Assessing NMAR¶

We can't determine if data is NMAR just by looking at the data, as whether or not data is NMAR depends on the

**unobserved data**.To establish if data is NMAR, we must:

**reason about the data generating process**, or- collect more data.

**Example:**Consider a dataset of survey data of students' self-reported happiness. The data contains PIDs and happiness scores; nothing else. Some happiness scores are missing.**Are happiness scores likely NMAR?**

### Assessing MAR¶

- Data are MAR if the missingness only depends on
**observed**data. - After reasoning about the data generating process, if you establish that data is not NMAR, then it must be either MAR or MCAR.
- The more columns we have in our dataset, the "weaker the NMAR effect" is.
- Adding more columns -> controlling for more variables -> moving from NMAR to MAR.
**Example:**With no other columns, income in a census is NMAR. But once we look at location, education, and occupation, incomes are closer to being MAR.

### Deciding between MCAR and MAR¶

For data to be MCAR, the chance that values are missing should not depend on any other column or the values themselves.

**Example**: Consider a dataset of phones, in which we store the screen size and price of each phone.**Some prices are missing.**

Phone | Screen Size | Price |
---|---|---|

iPhone 14 | 6.06 | 999 |

Galaxy Z Fold 4 | 7.6 | NaN |

OnePlus 9 Pro | 6.7 | 799 |

iPhone 13 Pro Max | 6.68 | NaN |

If prices are MCAR, then

**the distribution of screen size should be the same**for:- phones whose prices are missing, and
- phones whose prices aren't missing.

**We can use a permutation test to decide between MAR and MCAR!**We are asking the question, did these two samples come from the same underlying distribution?

### Deciding between MCAR and MAR¶

Suppose you have a DataFrame with columns named $\text{col}_1$, $\text{col}_2$, ..., $\text{col}_k$, and want to test whether values in $\text{col}_X$ are MCAR. To test whether $\text{col}_X$'s missingness is independent of all other columns in the DataFrame:

For $i = 1, 2, ..., k$, where $i \neq X$:

Look at the distribution of $\text{col}_i$ when $\text{col}_X$ is missing.

Look at the distribution of $\text{col}_i$ when $\text{col}_X$ is not missing.

Check if these two distributions are the same. (What do we mean by "the same"?)

If so, then $\text{col}_X$'s missingness doesn't depend on $\text{col}_i$.

If not, then $\text{col}_X$ is MAR dependent on $\text{col}_i$.

If all pairs of distribution were the same, then $\text{col}_X$ is MCAR.

### Example: Heights¶

- Let's load in Galton's dataset containing the heights of adult children and their parents (which you may have seen in DSC 10).
- The dataset does not contain any missing values – we will
**artifically introduce missing values**such that the values are MCAR, for illustration.

```
heights = pd.read_csv('data/midparent.csv')
heights = heights.rename(columns={'childHeight': 'child'})
heights = heights[['father', 'mother', 'gender', 'child']]
heights.head()
```

father | mother | gender | child | |
---|---|---|---|---|

0 | 78.5 | 67.0 | male | 73.2 |

1 | 78.5 | 67.0 | female | 69.2 |

2 | 78.5 | 67.0 | female | 69.0 |

3 | 78.5 | 67.0 | female | 69.0 |

4 | 75.5 | 66.5 | male | 73.5 |

Proof that there aren't currently any missing values in `heights`

:

```
heights.isna().sum()
```

father 0 mother 0 gender 0 child 0 dtype: int64

We have three numerical columns – `'father'`

, `'mother'`

, and `'child'`

. Let's visualize them simultaneously.

```
fig = px.scatter_matrix(heights.drop(columns=['gender']))
fig
```