Week 01
- Week 01
Q1: IMPLICIT CASTING
When we combine two different data types we trigger an implicit cast.
x <- c(1,2,3)
y <- c("a","b","c")
c( x, y )
[1] "1" "2" "3" "a" "b" "c"
The challenge with casting is the loss of data or information.
x <- c(1,"a",2,"b")
x
[1] "1" "a" "2" "b"
as.numeric(x)
[1] 1 NA 2 NA
Warning message:
NAs introduced by coercion
R tries to be clever by invoking a casting rule that minimizes data loss. For example, when we combine numeric and character vectors we will implicitly recast the object as a character vector.
Q1: Default Data Types
Consider the following cases. Explain why the defaults were selected in each case.
x.numeric <- 1:3
x.character <- LETTERS[1:3]
f <- factor( LETTERS[24:26] )
x
[1] 1 2 3
x.character
[1] "A" "B" "C"
f
[1] X Y Z
Levels: X Y Z
# CASE A
z <- c( x.numeric, x.character )
[1] "1" "2" "3" "A" "B" "C"
class( z )
[1] "character"
# CASE B
z <- c( x.character, f )
[1] "A" "B" "C" "1" "2" "3"
class( z )
[1] "character"
# CASE C
z <- c( x.numeric, f )
[1] 1 2 3 1 2 3
class( z )
[1] "integer"
Which case is most likely to introduce errors in your analysis? Explain why and present an example.
Q2: NUMBERS
Computers by default treat numbers as continuous (can take on negative or positive values AND decimals). Integers are a subclass because they are continuous numbers (negative or positive) that always have NO decimals. As a result they are easier to represent in memory.
Integers, for example, require 32 bits of memory, meaning a ‘shelf’ in the computer storage that has space or slots for 32 bits: [ 01 02 03 ... 30 31 31 ]
A bit is the smallest unit of memory in a computer that takes on values of 0 or 1, so a 32 bit number would really look like: [01000101100001100010001011000101]
If you do the math, there are 2^32 ways to combine 32 zeros and ones, and if you divide that by two you get the possible range of a regular integer: -2,147,483,648 to +2,147,483,648.
x <- 10 # regular number
y <- 10L # integer
class(x)
# "numeric"
typeof(x)
# "double"
class(y)
# "integer"
typeof(y)
# "integer"
identical(x, y)
# FALSE
x == y
# TRUE
The class() function tells you what type of object it is (a label that is meaningful to the programer). The typeof() function tells you how R stores it in memory (an internal construct).
Integers are stored as integers. Numbers (numeric objects) are stored as doubles. What is a double? It literally means, continous numbers require twice as much space in memory as integers. Since integers require 32 bits, doubles require 64 bits. More on this in the Numeric Precision section below.
Why are they equal but not identical? Similar to previous examples, to compare two things with the == operator they need to be the same class. So x and y are recast implicitly (or “coerced”) before comparisons are made.
The identical(x,y) operation instead asks, are these objects exactly the same in every important way? That includes:
- value
- length
- storage type
- class
- attributes
And importantly, there is NO coercion before comparing. So identical() is more strict than ==.
Now consider this example:
x <- 10.0
y <- 10
typeof(x)
typeof(y)
x == y
identical(x, y)
Q2-A: Number Casting
Can you guess what each will return?
What about implicit casting? If the goal is to preserve as much information as possible, which numeric type will R choose, integer or double?
x <- 10.0
y <- 10L
class( c(x,y) )
Would this make it easier to guess?
x <- 10.96
y <- 10L
class( c(x,y) )
Which results in a loss of information?
as.integer(x)
as.numeric(y)
Q3: FACTORS
Q3-A: Ordering Levels
By default factors created in R will order levels (categories) alphabetically.
In many cases levels have a meaningful order other than alphabetization.
For example, days of the week, months of the year, etc.
How can you alter this factor so that it correctly orders the days of the week?
x <- c("MON","TUE","WED","THUR","FRI","SAT","SUN")
vec <- sample( x, 100, replace=TRUE )
f <- factor( vec )
table( f )
f
FRI MON SAT SUN THUR TUE WED
14 18 14 9 16 16 13
Side note: in R the term levels describes categories in a categorical variables (a factor).
In regression, we use the term levels to describe numerical dosages of a treatment, such as milligrams of caffeine in a heart rate study.
In regression, the term factor is also used synonymously with explanatory variable or covariate. For example, what other factors might explain elevated heart rates in patients? Age could be one factor, race could be another factor. In this context factor is a cause, not a categorical variable.
Make a mental note that factor and level are both precise technical terms that have different meanings in computer science and statistics.
Q3-B: Empty Levels
How can we drop empty levels from a factor?
f2 <- f[ f %in% c("MON","TUE","WED","THUR","FRI") ]
table( f2 )
# f2
# FRI MON SAT SUN THUR TUE WED
# 16 15 0 0 14 13 12
Q3-C: Counting Zeros
What about cases where counts of zeros are important? What if we wanted to note here days that events did not occur?
Create a new factor that will report levels that do not occur in a sample.
x <- c("TUE","WED","FRI","SUN")
vec <- sample( x, 20, replace=TRUE )
f3 <- factor( vec )
table( f3 )
# HAVE
# f3
# FRI SUN TUE WED
# 6 3 4 7
# WANT
# f3
# MON TUE WED THUR FRI SAT SUN
# 0 4 7 0 6 0 3
Q4: COMPARISON OF SETS
Recall the structure of IF STATEMENTS:
if( logical statement )
{ code that only executes if TRUE }
It’s important to note that the logical statement must return a SINGLE T/F value. If you use a logical vector as the argument in the IF condition then it will use the first T/F values in the vector and ignore the rest (which is problematic in many instances).
The logical EQUALS operator == compares elements in two vectors in order. It will return the same number of T/F as length(x).
If you want to test whether two vectors are the same use the identical() function, which always returns a single T/F value:
x <- c("A","B","C")
y <- c("A","B","C")
x == y
[1] TRUE TRUE TRUE
identical( x, y )
[1] TRUE
# incorrect
if( x == y )
{ some code }
# correct
if( identical( x, y ) )
{ some code }
Q4-A: Ignore Order
Perhaps you want to run a chunk of code only IF two vectors contain the same elements, but order is irrelevant.
Note that in most cases X and Y will represent vectors of observations. The order and position of vectors is extremely important. Changing the order or a vector will corrupt the data.
x <- c("tom","nancy","sara")
y <- c("male","female","female")
d <- data.frame( name=x, gender=y )
In this case X and Y represents sets of elements, so the positions are not important.
Update this case to compare the two sets X and Y to see if they contain the same elements while ignoring order.
x <- c("A","B","C")
y <- c("B","A","C")
identical( x, y )
[1] FALSE
Q4-B: Ignore Vector Length
Similar to the case above, we want to compare these two sets to ensure they contain the same elements. We don’t care about how many times each element occurs, just that the two sets are the same.
How can we ignore the number of elements here?
x <- c("A","B","A","C")
y <- c("B","C","B","A","C")
identical( x, y )
[1] FALSE
Q4-C: Data Types
This is an interesting case because the logical operator == will consider these vectors to be identical, but the identical() function will not.
x <- c(1,2,3)
y <- c("1","2","3")
x == y
[1] TRUE TRUE TRUE
identical( x, y )
[1] FALSE
The reason for this is that logical operators will implicitly cast data types in order to compare two objects that have different data types. You saw this last week:
5 > 10
[1] FALSE
# implicitly casts both as character vectors
5 > "10"
[1] TRUE
How would you adapt the code here so it compares the two vectors and ignores the data type.
x <- c(1,2,3)
y <- c("1","2","3")
identical( x, y )
Does you solution also work with this example?
x <- c(01,02,03)
y <- c("01","02","03")
Q4-D: Comparisons with Missing Values
Missing values are important in statistics and data analytics, but they pose some challenges for computer logic.
Note the behavior of logical operations here:
# missing info for both individuals
x <- c("A",NA,"C")
y <- c("A",NA,"C")
x == y
[1] TRUE NA TRUE
identical( x, y )
[1] TRUE
# missing info for one individual
x <- c("A","B","C")
y <- c("A",NA,"C")
x == y
[1] TRUE NA TRUE
identical( x, y )
[1] FALSE
Write a statement that compares two vectors while ignoring missing cases.
Does your code also work with the following cases?
x <- c("A","A","B",NA,"C","C")
y <- c("A",NA,"B","B","C","C")
x <- c("A","A","B","B","C","C")
y <- c("A",NA,"B",NA,"C","C")
Q5: APPROXIMATE MATCHES
In this example vectors represent sets of traits of pairs of individuals in a study.
- race (white/minority)
- gender (male/female)
- college degree? (college/highschool)
We want to identify people that are similar but not necessarily identical.
Create a function that compares the two individuals and returns TRUE if they are the same on at least two traits, and false if they only match on one or zero traits.
compare_pairs <- function( x, y )
{
# your code here
}
x1 <- c("white","female","college")
y1 <- c("white","female","college")
compare_pairs( x1, y1 )
# SHOULD BE TRUE
x2 <- c("minority","female","college")
y2 <- c("white","female","college")
compare_pairs( x2, y2 )
# SHOULD BE TRUE
x3 <- c("minority","female","college")
y3 <- c("white","male","college")
compare_pairs( x3, y3 )
# SHOULD BE FALSE
x4 <- c("minotity","female","college")
y4 <- c("white","male","high school")
compare_pairs( x4, y4 )
# SHOULD BE FALSE
Q6: NUMERIC PRECISION
A numeric vector is a generic category for vectors of numbers, but computers have different rules for storing integers versus decimals. The rules determine how much memory is allocated for each object.
Each element in an integer vector occupies 4 bytes of memory. Each byte contains 8 bits. So each element in an integer vector requires 32 bits of memory.
The word bit is short for “binary digit”, the smallest unit of storage in a computer and the fundamental building block of mathematics and code in computer science. A bit is a single position in memory that can be in one of two states - either 1 (on) or 0 (off). All data in computers is encoded using bits.
Try out binary number conversion to see an example of how it works: Numeric to Binary Converter
How many different integer values can we represent with 4 bytes of memory then?
Each byte is considered to have 8 bits in this context. Since there are 4 bytes, that means 4 × 8 bits = 32 bits are available for storing a number.
Therefore, each 4-byte portion of memory can handle 2³² = 4 294 967 296 representations. Typically, the ranges of integers supported are either:
- 0 .. 4 294 967 295 if you want only non-negative integer values and do not wish to “waste” a bit to indicate a sign (positive versus negative).
- −2 147 483 648 .. 2 147 483 647, where one of the bits is used to indicate sign as opposed to allowing for greater positive values.
Vectors of decimal numbers are called doubles because the computer allocates 8 bytes or twice as much memory for each element as allocated for elements in an integer vector. So more precise numbers are more “expensive” in computational terms. Modern computers typically allocate more than 8 bytes per element for doubles and it varies by operating system.
Q6-A: Default Numeric Type
Based upon these examples, what are the rules R applies for numeric casting when combining integers and doubles? Does this rule optimize performance (smaller objects = faster computing time), or information integrity (preventing loss of precision)?
> x <- sample( 1:10, 100, replace=TRUE )
> x
[1] 1 6 2 9 9 7 6 3 6 10 2 9 5 3 10
[16] 2 7 5 3 3 1 6 1 9 9 3 10 2 6 8
[31] 2 3 1 9 10 6 10 1 6 8 1 10 5 4 2
[46] 1 10 8 10 3 8 7 4 7 5 1 9 2 9 6
[61] 3 8 1 7 2 6 9 9 1 3 4 5 2 4 3
[76] 6 5 7 4 6 7 4 2 9 1 6 3 3 6 2
[91] 1 6 3 1 8 7 6 4 6 8
> typeof(x)
[1] "integer"
> object.size(x)
448 bytes
>
> x <- as.double(x)
> x
[1] 1 6 2 9 9 7 6 3 6 10 2 9 5 3 10
[16] 2 7 5 3 3 1 6 1 9 9 3 10 2 6 8
[31] 2 3 1 9 10 6 10 1 6 8 1 10 5 4 2
[46] 1 10 8 10 3 8 7 4 7 5 1 9 2 9 6
[61] 3 8 1 7 2 6 9 9 1 3 4 5 2 4 3
[76] 6 5 7 4 6 7 4 2 9 1 6 3 3 6 2
[91] 1 6 3 1 8 7 6 4 6 8
> typeof(x)
[1] "double"
> object.size(x)
848 bytes
>
>
> x <- sample( 1:10, 100, replace=TRUE )
> typeof(x)
[1] "integer"
> object.size(x)
448 bytes
>
> z <- c( x, 1 )
> z
[1] 10 8 5 5 6 7 5 1 5 7 3 10 2 9 5
[16] 4 5 7 9 1 6 7 10 7 10 5 1 8 3 5
[31] 2 4 5 4 2 9 1 1 7 4 3 5 4 9 7
[46] 3 1 2 5 8 9 1 3 8 6 1 1 8 4 4
[61] 6 10 6 6 8 6 6 3 3 7 1 1 9 9 5
[76] 8 10 3 10 5 1 8 5 4 10 4 3 4 1 5
[91] 2 4 7 4 2 4 6 9 4 10 1
> typeof(z) # THIS IS UNEXPECTED
[1] "double"
> object.size(z)
856 bytes
>
> z <- c( x, 1L ) # ADD AN "L" TO FORCE THE INTEGER DATA TYPE
> z
[1] 8 6 5 9 7 1 6 1 6 10 1 6 1 5 10 9 9 9
[19] 6 2 7 5 10 8 2 4 2 9 5 2 6 8 8 8 2 2
[37] 8 6 2 7 6 2 6 8 8 2 4 8 7 7 10 3 8 8
[55] 10 1 1 2 1 8 9 9 6 6 8 4 6 5 6 1 5 5
[73] 6 1 9 6 7 8 7 4 9 3 9 4 4 5 1 4 5 3
[91] 1 6 7 5 4 2 10 7 1 6 1
> typeof(z)
[1] "integer"
> object.size(z)
456 bytes
# STORES EXTRA ZEROS BUT DOESN'T PRINT THEM:
# MEMORY USE ISN'T CAPTURED WELL BY PRINT
> z <- c( x, 1.00 )
> z
[1] 10 8 5 5 6 7 5 1 5 7 3 10 2 9 5
[16] 4 5 7 9 1 6 7 10 7 10 5 1 8 3 5
[31] 2 4 5 4 2 9 1 1 7 4 3 5 4 9 7
[46] 3 1 2 5 8 9 1 3 8 6 1 1 8 4 4
[61] 6 10 6 6 8 6 6 3 3 7 1 1 9 9 5
[76] 8 10 3 10 5 1 8 5 4 10 4 3 4 1 5
[91] 2 4 7 4 2 4 6 9 4 10 1
> typeof(z)
[1] "double"
> object.size(z)
856 bytes
>
# NOW IT PRINTS SIGNIFICANT DIGITS OF ALL
# FOR CONSISTENCY WHEN ONE DECIMAL IS INTRODUCED
> z <- c( x, 1.01 )
> z
[1] 10.00 8.00 5.00 5.00 6.00 7.00 5.00
[8] 1.00 5.00 7.00 3.00 10.00 2.00 9.00
[15] 5.00 4.00 5.00 7.00 9.00 1.00 6.00
[22] 7.00 10.00 7.00 10.00 5.00 1.00 8.00
[29] 3.00 5.00 2.00 4.00 5.00 4.00 2.00
[36] 9.00 1.00 1.00 7.00 4.00 3.00 5.00
[43] 4.00 9.00 7.00 3.00 1.00 2.00 5.00
[50] 8.00 9.00 1.00 3.00 8.00 6.00 1.00
[57] 1.00 8.00 4.00 4.00 6.00 10.00 6.00
[64] 6.00 8.00 6.00 6.00 3.00 3.00 7.00
[71] 1.00 1.00 9.00 9.00 5.00 8.00 10.00
[78] 3.00 10.00 5.00 1.00 8.00 5.00 4.00
[85] 10.00 4.00 3.00 4.00 1.00 5.00 2.00
[92] 4.00 7.00 4.00 2.00 4.00 6.00 9.00
[99] 4.00 10.00 1.01
> typeof(z)
[1] "double"
> object.size(z)
856 bytes
>
> z <- c( x, 1.001 )
> z
[1] 8.000 6.000 5.000 9.000 7.000 1.000 6.000 1.000
[9] 6.000 10.000 1.000 6.000 1.000 5.000 10.000 9.000
[17] 9.000 9.000 6.000 2.000 7.000 5.000 10.000 8.000
[25] 2.000 4.000 2.000 9.000 5.000 2.000 6.000 8.000
[33] 8.000 8.000 2.000 2.000 8.000 6.000 2.000 7.000
[41] 6.000 2.000 6.000 8.000 8.000 2.000 4.000 8.000
[49] 7.000 7.000 10.000 3.000 8.000 8.000 10.000 1.000
[57] 1.000 2.000 1.000 8.000 9.000 9.000 6.000 6.000
[65] 8.000 4.000 6.000 5.000 6.000 1.000 5.000 5.000
[73] 6.000 1.000 9.000 6.000 7.000 8.000 7.000 4.000
[81] 9.000 3.000 9.000 4.000 4.000 5.000 1.000 4.000
[89] 5.000 3.000 1.000 6.000 7.000 5.000 4.000 2.000
[97] 10.000 7.000 1.000 6.000 1.001
Note that the print function (which is called implicitly by typing the object name) will truncate zeros at the end of numbers up to the last meaningful digit. As a result, you cannot determine whether a numeric vector is an integer by printing it out because integers and doubles will often appear the same when printed. You need the typeof() function to check the data type for numeric vectors.
Q6-B: Memory PRECISION Errors
Computers only allocate a certain amount of memory for a number so at some point they will need to truncate the value to store it in memory.
Explain why the following might happen:
x <- 6.001
x
[1] 6.001
x <- 6.0000000000000000000000000000000001
x
[1] 6
6 == 6.001
[1] FALSE
6 == 6.0000000000000000000000000000000001
[1] TRUE
Consider this example.
# 1.051
# 1.052
1.1 equals 1.1
1.05 equals 1.05
1.051 does NOT equal 1.052
If we were to compare two numbers that differ by a tiny amount, how many decimals can we add before we encounter a rounding error? Stated differently, how far can we go before we get TRUE?
4.0 == 4.1
4.00 == 4.01
4.000 == 4.001
Similarly:
101 == 100
1001 == 1000
10001 == 10000
Q6-C: Memory REPRESENTATION Errors
Computers can only store information in binary representations (strings of 1’s and 0’s). As a result, these can all be represented EXACTLY:
10
10.5
10.25
10.125
because they are sums of powers of two. But
0.1
is not, and thus it cannot be represented EXACTLY.
Internally it is approximately
0.10000000000000000555...
Likewise
0.1 + 0.2
returns
0.30000000000000004
at full precision (the value stored in memory), even though R will normally print just:
0.3
This is one of the most unexpected and somewhat shocking errors you can encounter in computer science:
x <- 0.5 - 0.3
y <- 0.3 - 0.1
x
[1] 0.2
y
[1] 0.2
x == y
[1] FALSE
# FALSE on most machines
identical( x, y )
[1] FALSE
To see what is happening here print the difference of the variables:
x - y
[1] 2.775558e-17
These are called binary bit encoding problems or more often floating point errors.
It turns out that numbers with decimals are hard to represent in computer memory, so very tiny rounding errors can be introduced in calculations. They are not noticed unless the numbers are compared at the smallest scale.
The tiny difference was introduced by converting decimal numbers to their binary representation as a string of 0’a and 1’s in ther computer’s memory while doing the mathematical calculations behind the scenes.
This tiny tiny number representation problem will tyically only pose a problem in logical statements, but often get ironed out before they ever cause an issue. For example:
x <- 0.5 - 0.3
y <- 0.3 - 0.1
x == y
# FALSE
sum(x,y) == 0.4
# TRUE
(x+y) == 0.4
# TRUE
Since R can store numbers precisely up until about 15 significant digits and the bit encoding error is smaller than that it gets washed out when R truncates the number around the 15th decimal zero.
The problem ONLY arises if the math introduces a binary bit encoding error, which is typically not the case:
x <- 5 - 3
y <- 3 - 1
x == y
[1] TRUE
x <- 0.1 + 0.1
y <- 0.0 + 0.2
x
[1] 0.2
y
[1] 0.2
x == y
[1] TRUE
THE FIX: The formal and robust solution is to use the all.equal() function when comparing two numbers. This comparison operator accounts for potential floating point errors JUST IN CASE you would be so unlucky to encounter them.
USE THIS APPROACH IN YOUR CODE:
x <- 0.5 - 0.3
y <- 0.3 - 0.1
all.equal( x, y )
[1] TRUE
Explain why these also work:
x <- 0.5 - 0.3
y <- 0.3 - 0.1
x2 <- round( x, 1 )
y2 <- round( y, 1 )
x2
[1] 0.2
y2
[1] 0.2
x2 == y2
[1] TRUE
x3 <- as.character( x )
y3 <- as.character( y )
x3
[1] "0.2"
y3
[1] "0.2"
x3 == y3
[1] TRUE
Q6-D: Precision vs Representation Errors
Explain the difference between a truncation or coerced rounding error:
6.000000000000000000000000000001 == 6
# TRUE
And a floating point or binary bit error:
x <- 0.5 - 0.3
y <- 0.3 - 0.1
x == y
# FALSE
You can fix some truncation errors by telling the computer to use special large number formats. If you double the memory slots, for example, you double the number of significant digits you can store before hitting a truncation problem.
Can you solve representation problems in the same way? Why or why not?
Q7: COUNTING SUBSTRINGS
CHALLENGE QUESTION
In all of the examples above we were comparing two things.
# is it a nine?
x <- c( 1, 9, 10, 19, 99, 09 )
x == 9
[1] FALSE TRUE FALSE FALSE FALSE TRUE
Recall that logical statements can be used to count things.
# is it a nine?
x <- c( 1, 9, 10, 19, 99, 09 )
sum( x == 9 )
[1] 2
Q7-A: How would you count all nine’s in the vector?
For example, 19 contains one nine, 99 contains two nines.
Q7-B: How would you count all of the elements of X that CONTAIN a nine?
For example, 19 contains a nine.
Q7-C: Count all 17’s in a vector X containing all integers from 1 to 2,000.
x <- 1:2000
How many times does “17” occur in the vector?
For example, the number 117 contains a 17. The number 170 also contains a 17.