Background: Undergraduate
Undergraduate in Psychology
- Statistics
- Experiment Design
- Cognitive Theory
- Neurology
- Humans
Background: Honours
Psychophysics: illusory contours in 3D
Phil Grove
Background: Undergraduate
Christina Lee
If every psychologist in the world delivered gold standard smoking cessation therapy, the rate of smoking would still increase. You need to change policy to make change. To make effective policy, you need to have good data, and do good statistics.
Background: PhD
- "Ah, statistics, everything is black and white!
- "There's always an answer"
- "data in, answer out"
Background: PhD
- Data is really messy
- Missing values are frustrating
- How to Explore data?
Background: PhD - But in psych
- Focus on experiment design
- No focus on exploring data
- Exploring data felt...wrong?
- But it was so critical.
(My personal) motivation
Focus on building a bridge across a river. Less focus on how it is built, and the tools used.
visdat
published under "visdat: Visualising Whole Data Frames" in the Journal of Open Source Software
visdat::vis_dat(airquality)
visdat::vis_miss(airquality)
naniar
Tierney, NJ. Cook, D. "Expanding tidy data principles to facilitate missing data exploration, visualization and assessment of imputations." [Pre-print]
naniar::gg_miss_var(airquality)
naniar::gg_miss_upset(airquality)
All of Australia
...And New Zealand
... And Afghanistan and Albania
And the rest?
And the rest?
Problems:
- Overplotting
- We don't see the individuals
- We could look at 153 individual plots, but this doesn't help.
Does transparency help?
Does transparency + a model help?
This is still useful
But we forget about the individuals
How do we get the most out of this plot?
Introducing brolgar: brolgar.njtierney.com
- browsing
- over
- longitudinal data
- graphically, and
- analytically, in
- r
Longitudinal data as a time series 
heights <- as_tsibble(heights, index = year, key = country, regular = FALSE)- index: Your time variable
- key: Variable(s) defining individual groups (or series)
1. + 2. determine distinct rows in a tsibble.
(From Earo Wang's talk: Melt the clock)
Longitudinal data as a time series
Key Concepts:
Record important time series information once, and use it many times in other places
- We add information about index + key
- Index = Year
- Key = Country
Problem #1: How do I look at some of the data?
Look at only a sample of the data:
Sample n rows with sample_n()
sample_n_keys() to sample ... keys
Look at many subsamples
Look at many subsamples
Look at many subsamples
facet_sample(): See more individuals
ggplot(heights, aes(x = year, y = height_cm, group = country)) + geom_line()
facet_sample(): See more individuals
facet_strata(): See all individuals
Can we re-order these facets in a meaningful way?
facet_strata(along = -year): see all individuals along some variable
Problem #2: How do I find interesting observations?
Problem #2: How do I find interesting observations?
Identify features: summarise down to one observation
Identify features: summarise down to one observation
Identify features: summarise down to one observation
Identify important features and decide how to filter
Identify important features and decide how to filter
Join this feature back to the data
Join this feature back to the data
🎉 Countries with smallest and largest max height
Identify features: one per key 
heights %>% features(height_cm, feat_five_num)## # A tibble: 153 x 6## country min q25 med q75 max## <chr> <dbl> <dbl> <dbl> <dbl> <dbl>## 1 Afghanistan 161. 164. 167. 168. 168.## 2 Albania 168. 168. 170. 170. 170.## 3 Algeria 166. 168. 169 170. 171.## 4 Angola 159. 160. 167. 168. 169.## 5 Argentina 167. 168. 168. 170. 174.## 6 Armenia 164. 166. 169. 172. 172.## # … with 147 more rowsfeatures: Summaries that are aware of data structure
features: Summaries that are aware of data structure
heights %>% features(height_cm, #<< # variable we want to summarise feat_five_num) #<< # feature to calculatefeatures: Summaries that are aware of data structure
heights %>% features(height_cm, #<< # variable we want to summarise feat_five_num) #<< # feature to calculate## # A tibble: 153 x 6## country min q25 med q75 max## <chr> <dbl> <dbl> <dbl> <dbl> <dbl>## 1 Afghanistan 161. 164. 167. 168. 168.## 2 Albania 168. 168. 170. 170. 170.## 3 Algeria 166. 168. 169 170. 171.## 4 Angola 159. 160. 167. 168. 169.## 5 Argentina 167. 168. 168. 170. 174.## 6 Armenia 164. 166. 169. 172. 172.## # … with 147 more rowsfeatures: what is feat_five_num?
features: what is feat_five_num?
feat_five_num## function(x, ...) {## list(## min = b_min(x, ...),## q25 = b_q25(x, ...),## med = b_median(x, ...),## q75 = b_q75(x, ...),## max = b_max(x, ...)## )## }## <bytecode: 0x7fb6bc8fcba0>## <environment: namespace:brolgar>features: what is feat_five_num?
features: MANY more features in feasts
Such as:
feat_acf: autocorrelation-based featuresfeat_stl: STL (Seasonal, Trend, and Remainder by LOESS) decomposition
Problem #3: How do I understand my statistical model
Look at some subsamples?
gg_heights_fit + facet_sample()
Look at all subsamples?
gg_heights_fit + facet_strata()
Look at all subsamples arranged by residuals?
gg_heights_fit + facet_strata(along = -res)
Look at the predictions with the data?
set.seed(2019-11-13)heights_sample <- heights_aug %>% sample_n_keys(size = 9) %>% #<< sample the data ggplot(aes( x = year, y = pred, group = country)) + geom_line() + facet_wrap(~country)heights_sample
Look at the predictions with the data?
heights_sample + geom_point( aes( y = height_cm #<< add the original data ))
Show data by joining it with predictions, to explore the spread of predictions.
left_join(heights_near, heights_aug, by = "country") %>% ggplot(aes(x = year, y = pred, group = country, colour = stat)) + geom_line() + geom_point(aes(y = height_cm)) + facet_wrap(~country)