token — A unit of information


Content Model

token ::=


A Token identifies a unit of information. Usually, “tokens” are the result of some processing pass that has performed lexical analysis and divided a data set into the smallest units of information used for subsequent processing.

Exactly what constitutes a token varies by context.

Processing expectations

Formatted inline.


The following elements occur in token: text, alt, anchor, annotation, biblioref, indexterm (db.indexterm.endofrange), indexterm (db.indexterm.singular), indexterm (db.indexterm.startofrange), inlinemediaobject, link, olink, phrase (db._phrase), remark, replaceable, subscript, superscript, xref.

See Also

classname, property, symbol, type


<article xmlns=''>
<title>Example token</title>

<para>In parsing, line ends are turned into the <token>CRLF</token>, all other
whitespace becomes <token>WHITESP</token>.


In parsing, line ends are turned into the CRLF, all other whitespace becomes WHITESP.


This alpha reference page is $Revision: 1.4 $ published $Date: 2005/10/31 12:31:57 $.