protocol

tokenizer_protocol

Tokenization protocol for text represented as atoms, lists of characters, or lists of character codes.

Availability:
logtalk_load(tokenization(loader))
Author: Paulo Moura
Version: 1:0:0
Date: 2026-09-08
Compilation flags:
static
Dependencies:
(none)
Remarks:
(none)
Inherited public predicates:
(none)

Public predicates

tokenize/2

Tokenizes text using the default options.

Compilation flags:
static
Template:
tokenize(Text,Tokens)
Mode and number of proofs:
tokenize(+text,-list(text)) - one_or_error
Exceptions:
The Representation parameter is a variable:
instantiation_error
The Representation parameter is neither a variable nor atom, chars, or codes:
domain_error(text_representation,Representation)
The Language parameter is a variable:
instantiation_error
The Language parameter neither a variable nor an object identifier:
type_error(object_identifier,Language)
The Language parameter is an object identifier but not an object conforming to tokenizer_language_protocol:
domain_error(tokenizer_language_protocol,Language)
Text is not ground:
instantiation_error
The Representation parameter is atom but Text is not an atom:
type_error(atom,Text)
The Representation parameter is chars but Text is not a list of characters:
type_error(chars,Text)
The Representation parameter is codes but Text is not a list of character codes:
type_error(codes,Text)

tokenize/3

Tokenizes text using the given options. Recognized options are keep_punctuation(Boolean), lowercase(Boolean), normalize_quotes(Boolean), and normalize_dashes(Boolean).

Compilation flags:
static
Template:
tokenize(Text,Tokens,Options)
Mode and number of proofs:
tokenize(+text,-list(text),+list(compound)) - one_or_error
Exceptions:
The Representation parameter is a variable:
instantiation_error
The Representation parameter is neither a variable nor atom, chars, or codes:
domain_error(text_representation,Representation)
The Language parameter is a variable:
instantiation_error
The Language parameter neither a variable nor an object identifier:
type_error(object_identifier,Language)
The Language parameter is an object identifier but not an object conforming to tokenizer_language_protocol:
domain_error(tokenizer_language_protocol,Language)
Text is not ground:
instantiation_error
The Representation parameter is atom but Text is not an atom:
type_error(atom,Text)
The Representation parameter is chars but Text is not a list of characters:
type_error(chars,Text)
The Representation parameter is codes but Text is not a list of character codes:
type_error(codes,Text)
Options is a variable:
instantiation_error
Options is neither a variable nor a list:
type_error(list,Options)
An element Option of the list Options is a variable:
instantiation_error
An element Option of the list Options is neither a variable nor a compound term:
type_error(compound,Option)
An element Option of the list Options is a compound term but not a valid option:
domain_error(option,Option)

tokenize_sentences/2

Splits text into sentences and tokenizes each sentence using the default options.

Compilation flags:
static
Template:
tokenize_sentences(Text,TokenizedSentences)
Mode and number of proofs:
tokenize_sentences(+text,-list(list(text))) - one_or_error
Exceptions:
The Representation parameter is a variable:
instantiation_error
The Representation parameter is neither a variable nor atom, chars, or codes:
domain_error(text_representation,Representation)
The Language parameter is a variable:
instantiation_error
The Language parameter neither a variable nor an object identifier:
type_error(object_identifier,Language)
The Language parameter is an object identifier but not an object conforming to tokenizer_language_protocol:
domain_error(tokenizer_language_protocol,Language)
Text is not ground:
instantiation_error
The Representation parameter is atom but Text is not an atom:
type_error(atom,Text)
The Representation parameter is chars but Text is not a list of characters:
type_error(chars,Text)
The Representation parameter is codes but Text is not a list of character codes:
type_error(codes,Text)

tokenize_sentences/3

Splits text into sentences and tokenizes each sentence using the given options.

Compilation flags:
static
Template:
tokenize_sentences(Text,TokenizedSentences,Options)
Mode and number of proofs:
tokenize_sentences(+text,-list(list(text)),+list(compound)) - one_or_error
Exceptions:
The Representation parameter is a variable:
instantiation_error
The Representation parameter is neither a variable nor atom, chars, or codes:
domain_error(text_representation,Representation)
The Language parameter is a variable:
instantiation_error
The Language parameter neither a variable nor an object identifier:
type_error(object_identifier,Language)
The Language parameter is an object identifier but not an object conforming to tokenizer_language_protocol:
domain_error(tokenizer_language_protocol,Language)
Text is not ground:
instantiation_error
The Representation parameter is atom but Text is not an atom:
type_error(atom,Text)
The Representation parameter is chars but Text is not a list of characters:
type_error(chars,Text)
The Representation parameter is codes but Text is not a list of character codes:
type_error(codes,Text)
Options is a variable:
instantiation_error
Options is neither a variable nor a list:
type_error(list,Options)
An element Option of the list Options is a variable:
instantiation_error
An element Option of the list Options is neither a variable nor a compound term:
type_error(compound,Option)
An element Option of the list Options is a compound term but not a valid option:
domain_error(option,Option)

Protected predicates

(none)

Private predicates

(none)

Operators

(none)

See also

sentence_splitter_protocol, tokenizer_language_protocol, tokenizer(Representation,Language)