Title:	High Precise Measurement of R Expressions Execution Time
Version:	0.2.5
Description:	Provides infrastructure to accurately measure and compare the execution time of R expressions.
URL:	https://gitlab.com/artemklevtsov/benchr
BugReports:	https://gitlab.com/artemklevtsov/benchr/issues
License:	GPL-2 | GPL-3 [expanded from: GPL (≥ 2)]
Encoding:	UTF-8
ByteCompile:	yes
LazyData:	yes
Imports:	Rcpp (≥ 0.12.11), RcppProgress, stats, graphics
Suggests:	tinytest, ggplot2 (≥ 3.3.0)
LinkingTo:	Rcpp, RcppProgress
SystemRequirements:	C++11
RoxygenNote:	7.0.2
Collate:	'RcppExports.R' 'order.R' 'utils.R' 'benchmark.R' 'benchr-package.R' 'boxplot.R' 'mean.R' 'plot.R' 'units.R' 'summary.R' 'print.R' 'zzz.R'
NeedsCompilation:	yes
Packaged:	2020-03-07 06:11:26 UTC; unikum
Author:	Artem Klevtsov [aut, cre], Anton Antonov [ctb], Philipp Upravitelev [ctb]
Maintainer:	Artem Klevtsov <a.a.klevtsov@gmail.com>
Repository:	CRAN
Date/Publication:	2020-03-07 06:30:03 UTC

High Precise Measurement of R Expressions Execution Time

Description

Package benchr provides an infrastructure (framework) for precise measurement of R expressions execution time.

Details

To measure execution time, benchr provides function benchmark(), as well as a number of additional methods for analysis and representation of results.

For precise time measurement we use a cross-platform monotone clock, provided by C++11 standard in header file chrono. The timer accuracy depends on the implementation by the compiler in use, the OS and the hardware. Most commonly, the precision is one micro- or nanosecond. We provide the opportunity to get the timer accuracy (time interval between two consecutive timer ticks) via function timer_precision(). This accuracy is also listed in the output of implicit or explicit print call.

We estimate the timer overhead before the actual measurement by running multiple (2*10^5 by default) calls to an empty function. By doing that, we not only estimate the overhead, but also produce a warm-up effect on the processor, taking it out from idle state. After the actual measurement results are adjusted by the timer overhead.

Time intervals are measured in seconds and stored as ⁠long double⁠, which lets us capture a wide range of possible values from .Machine$double.xmin to .Machine$double.xmax. This is quite enough to operate both within very small (nanoseconds) and very big time frames (e.g. the maximum time interval possible is .Machine$double.xmax / 3600 hours).

It should be noted that the R session is not an isolated container with strictly bounded resources, therefore the execution time can be influenced by various factors, which may lead to outliers. In order to increase measurement reliability, we repeat executions multiple times (100 repetitions for each expression by default). This approach allows to collect enough data for statistical analysis in time difference.

We have also implemented several execution regimes in order to minimize the probability of systematic errors in measurements. By default, a random order of execution is being used. There is also a block order of execution, when the first expression is repeated a fixed number of times, then the second and so on. In such regime one can decrease the influence of allocators and garbage collection, since the memory is allocated only at the beginning of each block. The third option is to execute expressions in the order, provided by the user.

Note that we do not make any checks regarding return objects, i.e. one can compare not only algorithms with the same result, but also the same algorithm with different input parameters (e.g. input data sets of different size).

We also do not check whether the expressions are language objects (see is.language()) and do not coerce to that type.

Package options

accessible through function arguments. We allow to modify these parameters via package options. We have tried to set optimal default values, which you may consider changing in some cases. Here's a complete list of package options:

benchr.warmup

Number of iterations for timer overhead estimation (2*10^5 by default).

benchr.print_details

Whether additional information on the measurement parameters should be displayed (FALSE by default).

benchr.use_ggplot

Whether ggplot2 package should be used to produce plots, if the package is installed (TRUE by default).

Author(s)

Maintainer: Artem Klevtsov a.a.klevtsov@gmail.com (ORCID)

Other contributors:

• Anton Antonov tonytonov@gmail.com [contributor]

• Philipp Upravitelev upravitelev@gmail.com [contributor]

See Also

Useful links:

• https://gitlab.com/artemklevtsov/benchr

• Report bugs at https://gitlab.com/artemklevtsov/benchr/issues

Examples

# Benchmark expressions
res <- benchmark(
  rep(1:10, each = 10),
  rep.int(1:10, rep.int(10, 10))
)
# Aggregated statistics
mean(res)
summary(res)
# Plot results
boxplot(res)

High Precise Measurement of R Expressions Execution Time

Description

benchmark serves as a more accurate replacement of the often seen system.time(replicate(1000, expr)) expression. It tries hard to accurately measure only the time it takes to evaluate expr. To achieve this, the sub-millisecond (supposedly nanosecond) accurate timing functions most modern operating systems provide are used. Additionally all evaluations of the expressions are done in C++ code to minimize any measurment error.

Usage

benchmark(
  ...,
  times = 100L,
  order = c("random", "inorder", "block"),
  envir = parent.frame(),
  progress = TRUE,
  gcFirst = TRUE,
  gcDuring = FALSE
)

Arguments

Details

Before evaluating each expression times times, the overhead of calling the timing functions and the C++ function call overhead are estimated. This estimated overhead is subtracted from each measured evaluation time. Should the resulting timing be negative, a warning is thrown and the respective value is replaced by 0. If the timing is zero, a warning is also raised. Should all evaluations result in one of the two error conditions described above, an error is raised.

Value

Object of class benchmark, which is a data.frame with a number of additional attributes and contains the following columns:

An object of class benchmark also contains the following attributes:

The order in which the expressions are evaluated

“random”

(the default) randomizes the execution order

“inorder”

executes each expression in order

“block”

executes all repetitions of each expression as one block.

Author(s)

Artem Klevtsov a.a.klevtsov@gmail.com

See Also

summary.benchmark(), mean.benchmark(), print.benchmark(), plot.benchmark(), boxplot.benchmark()

Examples

## Measure the time it takes to dispatch a simple function call
## compared to simply evaluating the constant NULL
f <- function() NULL
res <- benchmark(NULL, f(), times = 1000L)

## Print results:
print(res)

## Plot results
boxplot(res)

Boxplot method for the benchmark timings

Description

Displays measurement results as box plots, with R expressions on X axis and execution time on Y axis.

Usage

## S3 method for class 'benchmark'
boxplot(
  x,
  units = "auto",
  log = TRUE,
  xlab,
  ylab,
  horizontal = FALSE,
  violin = FALSE,
  ...
)

Arguments

Details

If ggplot2 package is available, it will be used. In order to switch to default boxplot from the graphics package set option benchr.use_ggplot to FALSE: options(benchr.use_ggplot = FALSE).

Author(s)

Artem Klevtsov a.a.klevtsov@gmail.com

See Also

plot.benchmark()

Examples

timings <- benchmark(
  rchisq(100, 0), rchisq(100, 1), rchisq(100, 2),
  rchisq(100, 3), rchisq(100, 5),
  times = 1000L
)
boxplot(timings)

Mean method for the benchmark timings.

Description

This method computes aggregated statistics (sample mean and confidence intervals) for each expression.

Usage

## S3 method for class 'benchmark'
mean(x, trim = 0.05, conf.level = 0.95, relative = "mean", ...)

Arguments

Value

The method returns a data.frame with additional attributes, which contains these columns:

Additional attributes:

Author(s)

Artem Klevtsov a.a.klevtsov@gmail.com

See Also

summary.benchmark()

Examples

timings <- benchmark(
  rchisq(100, 0), rchisq(100, 1), rchisq(100, 2),
  rchisq(100, 3), rchisq(100, 5),
  times = 1000L
)
mean(timings)

Plot method for the benchmark timings.

Description

Displays measurement results as a scatter plot, with R expressions on X axis and execution time on Y axis. Each expression is highlighted by its own colour.

Usage

## S3 method for class 'benchmark'
plot(x, units = "auto", log = TRUE, xlab, ylab, ...)

Arguments

Details

If ggplot2 package is available, it will be used. In order to switch to default boxplot from the graphics package set option benchr.use_ggplot to FALSE: options(benchr.use_ggplot = FALSE).

Author(s)

Artem Klevtsov a.a.klevtsov@gmail.com

See Also

boxplot.benchmark()

Examples

timings <- benchmark(
  rchisq(100, 0), rchisq(100, 1), rchisq(100, 2),
  rchisq(100, 3), rchisq(100, 5),
  times = 1000L
)
plot(timings)

Print method for the benchmark timings.

Description

This is universal method of measurement results representation, which can be called either implicitly or explicitly. The method uses summary method to compute aggregated statistics for benchmarking results. print also provides additional information about the timer precision and overhead, the execution regime and the number of repeats.

Usage

## S3 method for class 'benchmark'
print(
  x,
  units = "auto",
  order = "none",
  relative = "median",
  details = FALSE,
  ...
)

## S3 method for class 'summaryBenchmark'
print(x, units = "auto", order = "none", ...)

Arguments

Value

Apart from the table output produced by summary, the method also prints additional information about the benchmarking process (with details = TRUE):

Author(s)

Artem Klevtsov a.a.klevtsov@gmail.com

See Also

summary.benchmark(), mean.benchmark()

Examples

a1 <- a2 <- a3 <- a4 <- numeric(0)
res <- benchmark(
  a1 <- c(a1, 1),
  a2 <- append(a2, 1),
  a3[length(a3) + 1] <- 1,
  a4[[length(a4) + 1]] <- 1
)
print(res)

Summary method for the benchmark timings.

Description

This method computes aggregated statistics (quantiles, means and sums) for each expression.

Usage

## S3 method for class 'benchmark'
summary(object, relative = "median", ...)

Arguments

Value

The method returns a data.frame with additional attributes, which contains these columns:

Additional attributes:

Author(s)

Artem Klevtsov a.a.klevtsov@gmail.com

See Also

mean.benchmark()

Examples

timings <- benchmark(
  rchisq(100, 0), rchisq(100, 1), rchisq(100, 2),
  rchisq(100, 3), rchisq(100, 5),
  times = 1000L
)
summary(timings)

Get timer precision.

Description

Get timer precision.

Usage

timer_precision()

Value

Number of seconds from one clock tick to the next.