The lexical analysis is independent of the syntax parsing and the semantic analysis. The lexical analyzer splits the source text up into tokens. The lexical grammar describes the syntax of those tokens. The grammar is designed to be suitable for high speed scanning and to make it easy to write a correct scanner for it. It has a minimum of special case rules and there is only one phase of translation.
For now, only one format of the source text is allowed UTF8 without BOM.
Character: any Unicode character
The source text is terminated by whichever comes first.
EndOfFile: physical end of the file \u0000 \u001A
EndOfLine: \u000D \u000A \u000D \u000A \u2028 \u2029 EndOfFile
WhiteSpace: Space Space WhiteSpace Space: \u0020 \u0009 \u000B \u000C
Keyword: any class const func struct