Antlr Lexer Quoted String Problem

Posted by Loki on Stack Overflow See other posts from Stack Overflow or by Loki
Published on 2010-05-23T05:27:58Z Indexed on 2010/05/23 5:30 UTC
Read the original article Hit count: 545

Filed under:
|
|
|
|

I'm trying to build a lexer to tokenize lone words and quoted strings. I got the following:

STRING:    QUOTE (options {greedy=false;} : . )* QUOTE ;
WS    :    SPACE+ { $channel = HIDDEN; } ;
WORD  :    ~(QUOTE|SPACE)+ ;

For the corner cases, it needs to parse:

"string" word1" word2

As three tokens: "string" as STRING and word1" and word2 as WORD. Basically, if there is a last quote, it needs to be part of the WORD were it is. If the quote is surrounded by white spaces, it should be a WORD.

I tried this rule for WORD, without success:

WORD:    ~(QUOTE|SPACE)+
    |    (~(QUOTE|SPACE)* QUOTE ~QUOTE*)=> ~(QUOTE|SPACE)* QUOTE ~(QUOTE|SPACE)* ; 

© Stack Overflow or respective owner

Related posts about string

Related posts about word