d96a314a68
Previously, only the string contents were stored in lexerState::token_str (i.e. not including the delimeters). Now, the delimeters are stored as well, thus preserving them. This also simplifies the code a bit. A new function is added to handle the character storage, which is also now applied for normal identifiers. To that end, the MAX_STRING_LENGTH was boosted to 256 so that all reasonably sized identifiers may fit.