1z0-931-23 Exam - Question 44

The stage of the indexing pipeline that divides text into tokens is the tokenizer. Tokenization is the process of breaking down text into smaller units, typically words or terms, which can then be processed for indexing. The lexer, on the other hand, is typically used in programming language interpreters and compilers, where it divides a stream of text into tokens based on syntax rules. In the context of text indexing, the correct terminology for dividing text into tokens is tokenization, hence the correct answer is the tokenizer.