---
title: "zoo Quick Reference"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{zoo Quick Reference}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteDepends{zoo,tseries,tinyplot}
  %\VignetteKeywords{irregular time series, daily data, weekly data, returns}
  %\VignettePackage{zoo}
---

This vignette gives a brief overview of (some of) the functionality contained
in `zoo` including several nifty code snippets when dealing
with (daily) financial data. It was originally written by Ajay Shah.
For a more complete overview of the
package's functionality and extensibility see 
Zeileis and Grothendieck (2005, _Journal of Statistical Software_, [doi:10.18637/jss.v014.i06](https://doi.org/10.18637/jss.v014.i06)).
(contained as `vignette("zoo", package = "zoo")` in the package),
the manual pages and the reference card.  

```{r preliminaries, include=FALSE}
library("zoo")
library("tseries")
online <- FALSE ## if set to FALSE the local copy of
                ## is used instead of get.hist.quote()
options(prompt = "R> ")
Sys.setenv(TZ = "GMT")
suppressWarnings(RNGversion("3.5.0"))
```

Some of the examples below assume the availability of various data files
which are all provided in the package. An easy way to make the available
is to set the working directory as follows:

```{r setwd, eval=FALSE}
library("zoo")
setwd(file.path(system.file(package = "zoo"), "doc"))
```


## Read a series from a text file

To read in data in a text file, `read.table()` and associated
functions can 
be used as usual with `zoo()` being called subsequently.
The convenience function `read.zoo` is a simple wrapper to these
functions that assumes the index is in the first column of the file
and the remaining columns are data.

Data in `demo1.txt`, where each row looks like

```
   23 Feb 2005|43.72
```

can be read in via
```{r read.zoo}
Sys.setlocale("LC_TIME", "C")
inrusd <- read.zoo("demo1.txt", sep = "|", format="%d %b %Y")
```
The `format` argument causes the first column to be transformed
to an index of class `"Date"`. The `Sys.setlocale` is set to
`"C"` here to assure that the English month abbreviations are read
correctly. (It is unnecessary if the system is already set to a locale
that understands English month abbreviations.)

The data in `demo2.txt` look like

```
   Daily,24 Feb 2005,2055.30,4337.00
```

and requires more attention because of the format of
the first column.
```{r read.table}
tmp <- read.table("demo2.txt", sep = ",")
z <- zoo(tmp[, 3:4], as.Date(as.character(tmp[, 2]), format="%d %b %Y"))
colnames(z) <- c("Nifty", "Junior")
```

## Query dates

To return all dates corresponding to a series
`index(z)` or equivalently 
```{r extract dates}
time(z)
```
can be used. The first and last date can be obtained by
```{r start and end}
start(z)
end(inrusd)
```

## Convert back into a plain matrix

To strip off the dates and just return a plain vector/matrix
`coredata` can be used
```{r convert to plain matrix}
plain <- coredata(z)
str(plain)
```

## Union and intersection

Unions and intersections of series can be computed by `merge`. The
intersection are those days where both series have time points:
```{r intersection}
m <- merge(inrusd, z, all = FALSE)
```
whereas the union uses all dates and fills the gaps where one
series has a time point but the other does not 
with `NA`s (by default):
```{r union}
m <- merge(inrusd, z)
```

`cbind(inrusd, z)` is almost equivalent to the `merge`
call, but may lead to inferior naming in some situations 
hence `merge` is preferred

To combine a series with its lag, use
```{r merge with lag}
merge(inrusd, lag(inrusd, -1))
```

## Visualization

By default, the `plot()` method generates a graph for each
series in `m`

```{r plotting1,fig.height=8,fig.width=6}
plot(m)
```


but several series can also be plotted in a single window.

```{r plotting2,fig.height=4,fig.width=6}
plot(m[, 2:3], plot.type = "single", col = c("red", "blue"), lwd = 2)
```

In addition to the plain base graphics
[plot()](https://zeileis.codeberg.page/zoo/man/plot.zoo.html) there are methods
[xyplot()](https://zeileis.codeberg.page/zoo/man/xyplot.zoo.html) for `lattice`,
[autoplot()](https://zeileis.codeberg.page/zoo/man/ggplot2.zoo.html) for `ggplot2`, and
[tinyplot()](https://zeileis.codeberg.page/zoo/man/tinyplot.zoo.html). All of these
provide many additional graphics capabilities, especially including facetting for
multivariate series. For example, using the `tinyplot` package one can do:


```{r tinyplot, eval=requireNamespace("tinyplot", quietly = TRUE), fig.height=6, fig.width=6}
library("tinyplot")
tinyplot(z, facet = Series ~ 1, theme = "clean2",
  palette = "Dark 3", lwd = 2, legend = FALSE)
```


## Select (a few) observations

Selections can be made for a range of dates of interest
```{r select range of dates}
window(z, start = as.Date("2005-02-15"), end = as.Date("2005-02-28"))
```
and also just for a single date
```{r select one date}
m[as.Date("2005-03-10")]
```

## Handle missing data

Various methods for dealing with `NA`s are available, including
linear interpolation
```{r impute NAs by interpolation}
interpolated <- na.approx(m)
```
`last observation carried forward',
```{r impute NAs by LOCF}
m <- na.locf(m)
m
```
and others.

## Prices and returns

To compute log-difference returns in %, the following
convenience function is defined
```{r compute returns function}
prices2returns <- function(x) 100*diff(log(x))
```
which can be used to convert all columns (of prices) into returns.
```{r column-wise returns}
r <- prices2returns(m)
```

A 10-day rolling window standard deviations (for all columns) can
be computed by
```{r rolling standard deviations}
rollapply(r, 10, sd)
```

To go from a daily series to the series of just the last-traded-day of each month
`aggregate` can be used
```{r last day of month}
prices2returns(aggregate(m, as.yearmon, tail, 1))
```

Analogously, the series can be aggregated to the last-traded-day of each week
employing a convenience function `nextfri` that computes for each `"Date"`
the next friday.
```{r last day of week}
nextfri <- function(x) 7 * ceiling(as.numeric(x-5+4) / 7) + as.Date(5-4)
prices2returns(aggregate(na.locf(m), nextfri, tail, 1))
```

Here is a similar example of `aggregate` where we define `to4sec`
analogously to `nextfri` in order to aggregate the `zoo` object `zsec` every 4 seconds.
```{r four second mark}
zsec <- structure(1:10, index = structure(c(1234760403.968, 1234760403.969, 
1234760403.969, 1234760405.029, 1234760405.029, 1234760405.03, 
1234760405.03, 1234760405.072, 1234760405.073, 1234760405.073
), class = c("POSIXt", "POSIXct"), tzone = ""), class = "zoo")

to4sec <- function(x) as.POSIXct(4*ceiling(as.numeric(x)/4), origin = "1970-01-01")
aggregate(zsec, to4sec, tail, 1)
```
Here is another example using the same `zsec` zoo object but this time
rather than aggregating we truncate times to the second using the last data
value for each such second. For large objects this will be much faster
than using `aggregate.zoo` .

```{r one second grid}
# tmp is zsec with time discretized into one second bins
tmp <- zsec
st <- start(tmp)
Epoch <- st - as.numeric(st)
time(tmp) <- as.integer(time(tmp) + 1e-7) + Epoch

# find index of last value in each one second interval
ix <- !duplicated(time(tmp), fromLast = TRUE)

# merge with grid 
merge(tmp[ix], zoo(, seq(start(tmp), end(tmp), "sec")))

# Here is a function which generalizes the above:

intraday.discretise <- function(b, Nsec) {
 st <- start(b)
 time(b) <- Nsec * as.integer(time(b)+1e-7) %/% Nsec + st -
 as.numeric(st)
 ix <- !duplicated(time(b), fromLast = TRUE)
 merge(b[ix], zoo(, seq(start(b), end(b), paste(Nsec, "sec"))))
}

intraday.discretise(zsec, 1)

```

## Query Yahoo! Finance

When connected to the internet, Yahoo! Finance can be easily queried using
the `get.hist.quote` function in
```{r tseries}
library("tseries")
```

```{r data handling if offline, includ=FALSE}
if(online) {
  msft <- get.hist.quote(instrument = "MSFT", start = "2004-01-01", end = "2004-12-31")
  msft2 <- get.hist.quote(instrument = "MSFT", start = "2004-01-01", end = "2004-12-31",
    compression = "m", quote = "Close")
  save(msft, msft2, file = "msft2004.rda")
} else {
  load("msft2004.rda")
}
```

From version 0.9-30 on, `get.hist.quote` by default returns `"zoo"` series with
a `"Date"` attribute (in previous versions these had to be transformed from `"ts"`
`by hand').

A daily series can be obtained by:
```{r get.hist.quote daily series, eval=FALSE}
msft <- get.hist.quote(instrument = "MSFT", start = "2004-01-01", end = "2004-12-31")
```

A monthly series can be obtained and transformed by
```{r get.hist.quote monthly series, eval=FALSE}
msft2 <- get.hist.quote(instrument = "MSFT", start = "2004-01-01", end = "2004-12-31",
  compression = "m", quote = "Close")
```

Here, `"yearmon"` dates might be even more useful:
```{r change index to yearmon}
time(msft2) <- as.yearmon(time(msft2))
```

The same series can equivalently be computed from the daily series via
```{r compute same series via aggregate}
msft3 <- aggregate(msft[, "Close"], as.yearmon, tail, 1)
```

The corresponding returns can be computed via
```{r compute returns}
r <- prices2returns(msft3)
```
where `r` is still a `"zoo"` series.


## Summaries

Here we create a daily series and then find the series of
quarterly means and standard deviations and also for weekly
means and standard deviations where we define weeks to end
on Tuesay.

We do the above separately for mean and standard deviation, binding
the two results together and then show a different approach in
which we define a custom `ag` function that can accept multiple
function names as a vector argument.

```{r summaries}
date1 <- seq(as.Date("2001-01-01"), as.Date("2002-12-1"), by = "day")
len1 <- length(date1)
set.seed(1) # to make it reproducible
data1 <- zoo(rnorm(len1), date1)

# quarterly summary

data1q.mean <- aggregate(data1, as.yearqtr, mean)
data1q.sd <- aggregate(data1, as.yearqtr, sd)
head(cbind(mean = data1q.mean, sd = data1q.sd), main = "Quarterly")

# weekly summary - week ends on tuesday

# Given a date find the next Tuesday.
# Based on formula in Prices and Returns section.
nexttue <- function(x) 7 * ceiling(as.numeric(x - 2 + 4)/7) + as.Date(2 - 4)

data1w <- cbind(
       mean = aggregate(data1, nexttue, mean),
       sd = aggregate(data1, nexttue, sd)
)
head(data1w)

### ALTERNATIVE ###

# Create function ag like aggregate but takes vector of
# function names.

FUNs <- c(mean, sd)
ag <- function(z, by, FUNs) {
       f <- function(f) aggregate(z, by, f)
       do.call(cbind, sapply(FUNs, f, simplify = FALSE))
}

data1q <- ag(data1, as.yearqtr, c("mean", "sd"))
data1w <- ag(data1, nexttue, c("mean", "sd"))

head(data1q)
head(data1w)
```

A convenience function which can determine for a vector of `"Date"`
observations whether it is a weekend or not

```{r is.weekend convenience function}
is.weekend <- function(x) ((as.numeric(x)-2) %% 7) < 2
```

The function `is.weekend` introduced above exploits the fact that a `"Date"`
is essentially the number of days since 1970-01-01, a Thursday. A more intelligible
function which yields identical results could be based on the `"POSIXlt"` class

```{r is.weekend based on POSIXlt}
is.weekend <- function(x) {
  x <- as.POSIXlt(x)
  x$wday > 5 | x$wday < 1
}
```