WAAVScript – Part2, Scanning for Tokens

A journy of a thousand programs begins with the first scanner.

WAAVScript is an interpreted language. That means there is a runtime that looks at input text, and tries to figure out the bits and pieces you’re trying to execute right then and there. Scanning text is the cornerstone to how performant this process will be, so it has to be super blazing fast, not use up too much memory, and deal with all the little corner cases that might arise.

WAAVScript has a few core data types and constructs to deal with, which can roughly be summarized by the following, which is actual WAAVScript code

The WAAVScript implementation is structured to be modular, and expandable. In addition, it follows the design philosophy of ‘zero allocation’, which really means, save allocations for as late as possible, when you really need them. The first step along this path is the code that turns this input into labeled chunks that a higher level thing can deal with.

The first file to look at is ps_lexer.h

The first thing to look at is the data type that represents a ‘lexeme’, which is simply a span of the stream that have labeled as being a specific type.

Here is a brief look at OctetCursor from ocspan.h