intervalaverage.Rdintervalaverage takes values recorded over
non-overlapping intervals and averages them to defined intervals, possibly within
groups (individuals/monitors/locations/etc). This function could be used to take averages over long
intervals
of values measured over short intervals and/or to take short "averages" of values measured over
longer intervals (ie, downsample without smoothing). Measurement intervals and averaging intervals need
not align. In the event that an averaging interval contains more than one measurement interval,
a weighted average is calculated (ie each measurement is weighted on the duration of its interval's
overlap with the averaging period interval).
intervalaverage(
x,
y,
interval_vars,
value_vars,
group_vars = NULL,
required_percentage = 100,
skip_overlap_check = FALSE,
verbose = FALSE
)a data.table containing values measured over intervals. see
interval_vars parameter
for how to specify interval columns and value_vars for how to
specify value columns.
intervals in x must must be completely non-overlapping within
groups defined by group_vars. if group_vars is specified (non-NULL), x must
also contain columns specified in group_vars.
a data.table object containing intervals over which averages of x values should be computed.
averaging intervals in y, unlike measurement intervals in x, may be overlapping within groups.
if group_vars is specified (non-NULL), y must contains those group_vars column names
(and this would allow different averaging periods for each group)
a length-2 character vector of column names in both x and y.
These column names specify columns in x and y that define
closed (inclusive) starting and ending intervals. The column name
specifying the lower-bound column must be specified first.
these columns in x and y must all be of the same class and either be integer or IDate.
The interval_vars character vector cannot be named. This is reserved for future use allowing
different interval_vars column names in x and y.
a character vector of column names in x. This specifies
the columns to be averaged.
A character vector of column names in both x and y.
The interaction of
these variables define groups in which averages of x values will be taken.
specifying subjects/monitors/locations within which to take averages.
By default this is NULL, in which case averages are taken over the entire x
dataset for each y period.
The group_vars character vector cannot be named. This is reserved for future use allowing
different interval_vars column names in x and y.
This percentage of the duration of each (possibly group-specific)
y interval must be observed
and nonmissing for a specific value_var in x in order for the return table to
contain a nonmissing average of the value_var for that y interval. If the percentage
of the nonmissing value_var observations is less than required_percentage an NA will be be returned
for that average.
The default is 100, meaning that if any portion of a y interval is either not recorded or
missing in x, then the corresponding return row will contain a an NA for the average of that
value_var.
by default, FALSE. setting this to TRUE will skip internal checks to make sure x intervals are non-overlapping within groups defined by group_vars. intervals in x must be non-overlapping, but you may want to skip this check if you've already checked this because it is computationally intensive for large datasets.
include printed timing information? by default, FALSE
returns a data.table object.
Rows of the return data.table correspond to intervals from y. i.e, the number
of rows of the return will be the number of rows of y.
Columns of the returned data.table are as follows:
grouping variables as specified in group_vars
interval columns corresponding to intervals in y. These columns are named the
same they were in x and y and as specified in interval_vars
value variable columns from x, averaged to periods in y.
named the same as they were in x
yduration: the length of the interval (ie as a count) specified in y
xduration: the total length of the intervals (ie as a count)
from x that fall into this interval from y. this will be equal to
yduration if x is comprehensive for (ie, fully covers) this interval from y.
nobs_<value_vars>: for each value_var specified, this is the count of
times (or whatever the units of the intervals are)
where there are non-missing values in x that fall into the specified interval from y. this will be
equal to xduration if the value_var contains no NA values over the y
interval. If there are NAs in value variables, then nobs_<value_vars>
will be different from xduration and won't necessarily be all the same
for each value_var.
xminstart: For each returned interval (ie the intervals from Y) the minimum of the
start intervals represented in x. If the start of the earliest x interval is less than the start
of the y interval, the minimum of the y interval is returned. Note, this is the minimum start
time in x matching with the y interval whether or not any value_vars were missing or not for that start time.
If you need non-missing minimum start times, you could remove NA intervals from
x prior to calling intervalaverage (this would need to be done separately for each value_var).
xmaxend: similar to xminstart but the maximum of the end intervals represented in x.
Again, this does not pay attention to whether the interval in x had non-missing value_vars.
maxgap_<value_vars> For each value_var specified, this is the length of the largest
concurrent run of times (or whatever the units of intervals are) for which that value variable is missing
(either structurally missing ie--no intervals at all in x--or missing with an NA or any combination thereof).
All intervals are treated as closed (ie inclusive of the start and end values in interval_vars)
x and y are not copied but rather passed by reference to function internals but the order of these data.tables is restored on function completion or error,
When required_percentage is less than 100, xminstart and xmaxend may be useful to determine whether an average meets specified coverage requirements in terms of not just percent of missingness but whether values are represented through the range of the y interval
maxgap variables may be useful in determing whether an average from incomplete data may be representative of the true unknowable average (because of missingness) beyond what's already accounted for based counts/percentage of non-missing values. Large runs of missing values over time (or whatever the units of intervals are) mean missingness is not equally distributed through the period. However, note that this gap information is simply returned and does not affect whether the average value is returned as NA or not--you'll need to apply your own criteria using this (these) maxgap variables.
x <- data.table(start=seq(1L,by=7L,length=6),
end=seq(7L,by=7L,length=6),
pm25=c(10,12,8,14,22,18))
y <- data.table(start=seq(3L,by=7L,length=6),
end=seq(9L,by=7L,length=6))
z <- intervalaverage(x,y,interval_vars=c("start","end"),
value_vars=c("pm25"))
#also see vignette for more extensive examples