hellmer: Batch Processing for Chat Models

Description

Batch processing framework for 'ellmer' chat models. Provides both sequential and parallel processing of chat interactions with features including tool calling and structured data extraction. Enables workflow management through progress tracking and recovery and automatic retry with backoff. Additional quality-of-life features include verbosity (or echo) control and sound notifications. Parallel processing is implemented via the 'future' framework. Includes methods for retrieving progress status, chat texts, and chat objects.

Author(s)

Maintainer: Dylan Pieper dylanpieper@gmail.com

See Also

Useful links:

• https://dylanpieper.github.io/hellmer/

Batch result class for managing chat processing results

Description

Batch result class for managing chat processing results

Usage

batch(
  prompts = list(),
  responses = list(),
  completed = integer(0),
  state_path = character(0),
  type_spec = NULL,
  judgements = integer(0),
  progress = logical(0),
  input_type = character(0),
  max_retries = integer(0),
  initial_delay = integer(0),
  max_delay = integer(0),
  backoff_factor = integer(0),
  chunk_size = integer(0),
  workers = integer(0),
  plan = character(0),
  beep = logical(0),
  echo = logical(0),
  state = list()
)

Arguments

Value

Returns an S7 class object of class "batch" that represents a collection of prompts and their responses from chat models. The object contains all input parameters as properties and provides methods for:

• Extracting text responses via texts() (includes structured data when a type specification is provided)

• Accessing full chat objects via chats()

• Tracking processing progress via progress()

The batch object manages prompt processing, tracks completion status, and handles retries for failed requests.

Examples

# Create a chat processor
chat <- chat_sequential(chat_openai())

# Process a batch of prompts
batch <- chat$batch(list(
  "What is R?",
  "Explain base R versus tidyverse",
  "Explain vectors, lists, and data frames"
))

# Check the progress if interrupted
batch$progress()

# Return the responses as a vector or list
batch$texts()

# Return the chat objects
batch$chats()

Process a batch of prompts with a parallel chat

Description

Process a batch of prompts with a parallel chat

Usage

batch.future_chat(
  chat_env,
  prompts,
  type_spec = NULL,
  judgements = 0,
  state_path = tempfile("chat_", fileext = ".rds"),
  workers = NULL,
  chunk_size = parallel::detectCores() * 5,
  plan = "multisession",
  max_chunk_attempts = 3L,
  max_retries = 3L,
  initial_delay = 20,
  max_delay = 80,
  backoff_factor = 2,
  beep = TRUE,
  progress = TRUE,
  echo = FALSE,
  ...
)

Arguments

Value

A batch object with the processed results

Process a batch of prompts with a sequential chat

Description

Process a batch of prompts with a sequential chat

Usage

batch.sequential_chat(
  chat_env,
  prompts,
  type_spec = NULL,
  judgements = 0,
  state_path = tempfile("chat_", fileext = ".rds"),
  progress = TRUE,
  max_retries = 3L,
  initial_delay = 20,
  max_delay = 80,
  backoff_factor = 2,
  beep = TRUE,
  echo = FALSE,
  ...
)

Arguments

Value

A batch object with the processed results

Capture chat model response with proper handling

Description

Capture chat model response with proper handling

Usage

capture(original_chat, prompt, type_spec, judgements, echo, ...)

Arguments

Value

List containing response information

Capture chat model response with proper handling and retries

Description

Capture chat model response with proper handling and retries

Usage

capture_with_retry(
  original_chat,
  prompt,
  type_spec,
  judgements,
  max_retries,
  initial_delay,
  max_delay,
  backoff_factor,
  echo,
  ...
)

Arguments

Value

List containing response information

Process a batch of prompts in parallel

Description

Processes a batch of chat prompts using parallel workers. Splits prompts into chunks for processing while maintaining state. For sequential processing, use chat_sequential().

Usage

chat_future(chat_model = NULL, ...)

Arguments

Value

A batch object (S7 class) containing:

• prompts: Original input prompts

• responses: Raw response data for completed prompts

• completed: Number of successfully processed prompts

• state_path: Path where batch state is saved

• type_spec: Type specification used for structured data

• texts: Function to extract text responses or structured data

• chats: Function to extract chat objects

• progress: Function to get processing status

• batch: Function to process a batch of prompts

Batch Method

This function provides access to the batch() method for parallel processing of prompts. See ?batch.future_chat for full details of the method and its parameters.

Examples

# Create a parallel chat processor with an object
chat <- chat_future(chat_openai(system_prompt = "Reply concisely"))

# Or a function
chat <- chat_future(chat_openai, system_prompt = "Reply concisely, one sentence")

# Process a batch of prompts in parallel
batch <- chat$batch(
  list(
    "What is R?",
    "Explain base R versus tidyverse",
    "Explain vectors, lists, and data frames"
  ),
  chunk_size = 3
)

# Process batch with echo enabled (when progress is disabled)
batch <- chat$batch(
  list(
    "What is R?",
    "Explain base R versus tidyverse"
  ),
  progress = FALSE,
  echo = TRUE
)

# Check the progress if interrupted
batch$progress()

# Return the responses
batch$texts()

# Return the chat objects
batch$chats()

Process a batch of prompts in sequence

Description

Processes a batch of chat prompts one at a time in sequential order. Maintains state between runs and can resume interrupted processing. For parallel processing, use chat_future().

Usage

chat_sequential(chat_model = NULL, ...)

Arguments

Value

A batch object (S7 class) containing

• prompts: Original input prompts

• responses: Raw response data for completed prompts

• completed: Number of successfully processed prompts

• state_path: Path where batch state is saved

• type_spec: Type specification used for structured data

• texts: Function to extract text responses or structured data

• chats: Function to extract chat objects

• progress: Function to get processing status

• batch: Function to process a batch of prompts

Batch Method

This function provides access to the batch() method for sequential processing of prompts. See ?batch.sequential_chat for full details of the method and its parameters.

Examples

# Create a sequential chat processor with an object
chat <- chat_sequential(chat_openai(system_prompt = "Reply concisely"))

# Or a function
chat <- chat_sequential(chat_openai, system_prompt = "Reply concisely, one sentence")

# Process a batch of prompts in sequence
batch <- chat$batch(
  list(
    "What is R?",
    "Explain base R versus tidyverse",
    "Explain vectors, lists, and data frames"
  ),
  max_retries = 3L,
  initial_delay = 20,
  beep = TRUE
)

# Process batch with echo enabled (when progress is disabled)
batch <- chat$batch(
  list(
    "What is R?",
    "Explain base R versus tidyverse"
  ),
  progress = FALSE,
  echo = TRUE
)

# Check the progress if interrupted
batch$progress()

# Return the responses
batch$texts()

# Return the chat objects
batch$chats()

Extract chat objects from a batch result

Description

Extract chat objects from a batch result

Usage

chats(x, ...)

Arguments

Value

A list of chat objects

Examples

# Create a chat processor
chat <- chat_sequential(chat_openai())

# Process a batch of prompts
batch <- chat$batch(list(
  "What is R?",
  "Explain base R versus tidyverse",
  "Explain vectors, lists, and data frames"
))

# Return the chat objects
batch$chats()

Create results object from batch

Description

Create results object from batch

Usage

create_results(result)

Arguments

Value

Results object with class "batch"

Finish successful batch processing

Description

Called after successful completion of batch processing to update progress indicators and provide feedback

Usage

finish_successful_batch(pb, beep, progress)

Arguments

Value

NULL (invisibly)

Handle batch interruption

Description

Handle batch interruption

Usage

handle_batch_interrupt(result, beep)

Arguments

Value

NULL (called for side effects)

Check if an error is eligible for retry

Description

Check if an error is eligible for retry

Usage

is_retry_error(error)

Arguments

Value

TRUE if eligible for retry, FALSE otherwise

Process chunks of prompts in parallel

Description

Process chunks of prompts in parallel

Usage

process_chunks(
  chunks,
  result,
  chat_obj,
  type_spec,
  judgements,
  pb,
  state_path,
  progress,
  beep,
  max_retries,
  initial_delay,
  max_delay,
  backoff_factor,
  echo,
  ...
)

Arguments

Value

Updated batch object with processed results

Process prompts in parallel chunks with error handling and state management

Description

Process prompts in parallel chunks with error handling and state management

Usage

process_future(
  chat_obj,
  prompts,
  type_spec,
  judgements,
  state_path,
  workers,
  chunk_size,
  plan,
  max_chunk_attempts,
  max_retries,
  initial_delay,
  max_delay,
  backoff_factor,
  beep,
  progress,
  echo,
  ...
)

Arguments

Value

Batch results object

Process structured data extraction with judgement

Description

Process structured data extraction with judgement

Usage

process_judgements(
  chat_obj,
  prompt,
  type_spec,
  judgements = 0,
  echo = FALSE,
  ...
)

Arguments

Value

List containing extraction process

Process batch of prompts with progress tracking and retries

Description

Process batch of prompts with progress tracking and retries

Usage

process_sequential(
  chat_obj,
  prompts,
  type_spec,
  judgements,
  state_path,
  progress,
  max_retries,
  initial_delay,
  max_delay,
  backoff_factor,
  beep,
  echo,
  ...
)

Arguments

Value

Batch results object

Get progress information from a batch result

Description

Get progress information from a batch result

Usage

progress(x, ...)

Arguments

Value

A list containing progress details

Examples

# Create a chat processor
chat <- chat_sequential(chat_openai())

# Process a batch of prompts
batch <- chat$batch(list(
  "What is R?",
  "Explain base R versus tidyverse",
  "Explain vectors, lists, and data frames"
))

# Check the progress
batch$progress()

Extract texts or structured data from a batch result

Description

Extract texts or structured data from a batch result

Usage

texts(x, ...)

Arguments

Value

A character vector or list of text responses. If a type specification was provided to the batch, structured data objects will be returned instead.

Examples

# Create a chat processor
chat <- chat_sequential(chat_openai())

# Process a batch of prompts
batch <- chat$batch(list(
  "What is R?",
  "Explain base R versus tidyverse",
  "Explain vectors, lists, and data frames"
))

# Extract text responses
batch$texts()