EncodingScheme

Various portions of the DAT file use a unique encoding mechanism. This page aims to describe this encoding scheme.

The encoder takes a stream of bytes and classifies the bytes into 2 categories, either the byte is a printable ASCII character, or the byte is non-ascii.

In order to increase the efficiency of the encoding scheme subsequent bytes of the same "type" are encoded together. This will make more sense in the following subsections.

There are additional rules as to the line lengths allowed for encoded data, and these rules sometimes cause blocks to be split into separate blocks, or empty blocks to be injected. This is documented below in the Formatting section.

ASCII Characters

The encoder determines a byte is an ASCII character if it's ASCII code is >= 32 and < 127.

ASCII characters are encoding in a very simple way. The characters are wrapped inside A() so for example the letter C would be encoded as A(C)

When multiple ASCII characters are together they get encoded into a single block. That is the word cat would be encoded to A(cat)

Escape Characters

Due to the nature of the ASCII block encoding, some printable ASCII characters present a problem. Specifically ( and ), because they are used to denote the start/end of the encoding blocks. To account for this \ is used as an escape character. This also leads to '' needing to be escaped.

The example string of color(blue) would be encoded as A(color\(blue\))

Non-ASCII Characters

If the byte in question is non-printable ASCII is it converted into a different form and stored in a B() block. The same rule applies for subsequent bytes that are of this type, they get appended just like the ASCII Characters do.

The byte in question is split into its upper 4 bits and lower 4 bits. These values are then mapped to the printable ASCII range by adding 65.

    var lowerHalf = (char)((b & 0xF) + 65);
    var upperHalf = (char)(((b & 0xF0) >> 4) + 65);

The two ASCII characters are appended upperHalf + lowerHalf.

The value 0 is encoded as B(AA) the value 2 is B(AC)

Formatting

TBD