protocol

tokenizer_language_protocol

Language provider protocol for tokenization and sentence splitting using canonical character-code lists.

Availability:

logtalk_load(tokenization(loader))

Author: Paulo Moura

Version: 1:0:0

Date: 2026-09-05

Compilation flags:

static

Dependencies:

(none)

Remarks:

(none)

Inherited public predicates:

(none)

Public predicates

tokenize_codes/3

Tokenizes a character-code list using validated, merged facade options.

Compilation flags:

static

Template:

tokenize_codes(Codes,Tokens,Options)

Mode and number of proofs:

tokenize_codes(+list(character_code),-list(list(character_code)),+list(compound)) - one

---

split_sentence_codes/3

Splits a character-code list into sentence character-code lists using validated, merged facade options.

Compilation flags:

static

Template:

split_sentence_codes(Codes,Sentences,Options)

Mode and number of proofs:

split_sentence_codes(+list(character_code),-list(list(character_code)),+list(compound)) - one

---

Protected predicates

(none)

Private predicates

(none)

Operators

(none)

See also

tokenizer_protocol, sentence_splitter_protocol, tokenizer(Representation,Language)