§ Tidy Data
- Tidy data is a standard way of mapping the meaning of a dataset to its structure. A dataset is messy or tidy depending on how rows, columns and tables are matched up with observations, variables and types. In tidy data:
1. Each variable forms a column. 2. Each observation forms a row. 3. Each type of observational unit forms a table.
While the order of variables and observations does not affect analysis, a good ordering makes it easier to scan the raw values. One way of organising variables is by their role in the analysis: are values fixed by the design of the data collection, or are they measured during the course of the experiment? Fixed variables describe the experimental design and are known in advance. Computer scientists often call fixed variables dimensions, and statisticians usually denote them with subscripts on random variables. Measured variables are what we actually measure in the study. Fixed variables should come first, followed by measured variables, each ordered so that related variables are contiguous. Rows can then be ordered by the first variable, breaking ties with the second and subsequent (fixed) variables. This is the convention adopted by all tabular displays in this paper.
§ Messy 1: Column headers are values, not variable names
- eg. columns are
religion |<$10k |$10-20k |$20-30k |$30-40k |$40-50k |$50-75k.
- melt dataset to get
molten stacked data.
§ Messy 2: Multiple variables stored in one column
- This often manifests after melting.
- eg. columns are
country | year | m014 | m1524 | .. | f014 | f1524...
- columns represent both sex and age ranges. After metling, we get a single column
sexage with entries like
- The data is still molten , so we should reshape it before it sets into tidy columnlar data. We do this by splitting the column into two, one for
age and one for
§ Messy 3: Variables are stored in both rows and columns
id year month element d1 d2 d3 d4 d5 ...
MX17004 2010 1 tmax — — — — — — — —
MX17004 2010 1 tmin — — — — — — — —
MX17004 2010 2 tmax — 27.3 24.1 — — — — —
MX17004 2010 2 tmin — 14.4 14.4 — — — — —
MX17004 2010 3 tmax — — — — 32.1 — — —
MX17004 2010 3 tmin — — — — 14.2 — — —
MX17004 2010 4 tmax — — — — — — — —
MX17004 2010 4 tmin — — — — — — — —
MX17004 2010 5 tmax — — — — — — — —
MX17004 2010 5 tmin — — — — — — — —
- Some variables are in individual columns (id, year, month)
- Some variables are spread across columns (day is spread as d1–d31)
- Some variables are smearted across rows (eg.
tmax/tmin). TODO: what does this mean, really?
- First, tidy by collating into
id date element value
MX17004 2010-01-30 tmax 27.8
MX17004 2010-01-30 tmin 14.5
MX17004 2010-02-02 tmax 27.3
MX17004 2010-02-02 tmin 14.4
MX17004 2010-02-03 tmax 24.1
MX17004 2010-02-03 tmin 14.4
MX17004 2010-02-11 tmax 29.7
MX17004 2010-02-11 tmin 13.4
MX17004 2010-02-23 tmax 29.9
MX17004 2010-02-23 tmin 10.7
- Dataset above is still molten. Must reshape along
element to get two columns for
min. This gives:
id date tmax tmin
MX17004 2010-01-30 27.8 14.5
MX17004 2010-02-02 27.3 14.4
MX17004 2010-02-03 24.1 14.4
MX17004 2010-02-11 29.7 13.4
MX17004 2010-02-23 29.9 10.7
MX17004 2010-03-05 32.1 14.2
MX17004 2010-03-10 34.5 16.8
MX17004 2010-03-16 31.1 17.6
MX17004 2010-04-27 36.3 16.7
MX17004 2010-05-27 33.2 18.2
- Months with less than 31 days have structural missing values for the last day(s) of the month.
- The element column is not a variable; it stores the names of variables.
§ Multiple types in one table:
§ data manipulation, relationship to
- Data transformation in R for data science
mutate() adds new variables that are functions of existing variables
select() picks variables based on their names.
filter() picks cases based on their values.
summarise() reduces multiple values down to a single summary.
arrange() changes the ordering of the rows.
- Most of R's visualization ecosystem is tidy by default.
ggplot are all tidy.
- Most modelling tools work best with tidy datasets.
§ Questions about performance benching in terms of tidy
- Is runs of a program at different performance levels like
O3 to be stored as separate columns? Or as a categorical column called "optimization level" with entries stored in separate rows of
- If we go by the tidy rule "Each variable forms a column", then this suggests that
optimization level is a variable.
- Then the tidy rule
Each observation forms a row. makes us use rows like
[foo.test | opt-level=O1 | ] and
[foo.test | opt-level=O2 | ].
- Broader question: what is the tidy rule for categorical column?
- However, in the tidy data paper, Table 12, it is advocated to have two columns for
tmax instead of having a column called
element with choices
tmax. So it seems to be preferred that if one has a categorical variable, we make its observations into columns.
- This suggests that I order my bench data as
[foo.test | O1-runtime=_ | O2-runtime=_ | O3-runtime=_ ].