object

english_tokenizer

English tokenization and sentence splitting provider using portable rule-based heuristics and a curated abbreviation inventory.

Availability:

logtalk_load(tokenization(loader))

Author: Paulo Moura

Version: 1:0:0

Date: 2026-09-05

Compilation flags:

static, context_switching_calls

Implements:

public tokenizer_language_protocol

Imports:

public tokenizer_rules

Remarks:

(none)

Inherited public predicates:

 check_option/1  check_options/1  default_option/1  default_options/1  option/2  option/3  split_sentence_codes/3  tokenize_codes/3  valid_option/1  valid_options/1

Public predicates

(no local declarations; see entity ancestors if any)

Protected predicates

(no local declarations; see entity ancestors if any)

Private predicates

(no local declarations; see entity ancestors if any)

Operators

(none)

See also

tokenizer(Representation,Language), tokenizer_rules