[svlug] Lex question about variable length fields

Robert Hajime Lanning lanning at lanning.cc
Thu May 22 16:32:23 PDT 2014

On 05/22/14 15:33, Bob Smith wrote:
> I would like to get three single word tokens followed by a fourth
> token that is whatever is left on the input line.  So, the input
> line "My name is Jabba the Hutt" would return tokens
> "My",
> "name",
> "is", and
> "Jabba the Hutt".
> The last token looks like several words and when I write the lex
> rules to pick up more than one word, it picks up the whole line.

I think most parsers tokenize the whole line, then piece back together.
This is why, without quoting, multiple spaces are condensed into one.

$ echo this      has    a  lot   of  spaces
this has a lot of spaces

Mr. Flibble
King of the Potato People

More information about the svlug mailing list