[svlug] Lex question about variable length fields

[Robert Hajime Lanning]

On 05/22/14 15:33, Bob Smith wrote:
> I would like to get three single word tokens followed by a fourth
> token that is whatever is left on the input line.  So, the input
> line "My name is Jabba the Hutt" would return tokens
> "My",
> "name",
> "is", and
> "Jabba the Hutt".
>
> The last token looks like several words and when I write the lex
> rules to pick up more than one word, it picks up the whole line.

I think most parsers tokenize the whole line, then piece back together.
This is why, without quoting, multiple spaces are condensed into one.

$ echo this      has    a  lot   of  spaces
this has a lot of spaces

-- 
Mr. Flibble
King of the Potato People
http://www.linkedin.com/in/RobertLanning