'Using TokenStreamRewriter to insert tokens after lexing but before parsing

Using ANTLR 4.9.2 for C++.

Depending on the first tokens I might need to insert some tokens before parsing. My approach (simplified)

    antlr4::ANTLRInputStream antlrIs(properlyEscaped);
    Lexer lexer(&antlrIs);
    antlr4::CommonTokenStream tokens(&lexer);
    antlr4::TokenStreamRewriter tokenStreamRewriter(&tokens);

    if (!(tokens.LA(1) == Lexer::MY_SPECIAL_TOKEN))
    {
        tokenStreamRewriter.insertBefore(tokens.LT(1), string("begin"));
    }

    Parser parser(&tokens);
    Parser::FileContext* fileContext = parser.file();

Stepping with the debugger I see that the token is actually inserted. But the new token I insert seems be be ignored by parser.file().

How can I insert tokens so that parser.file() uses them?

antlr4


Solution 1:[1]

TokenStreamRewriter just builds up a set of instructions for how the input stream should be changed. It doesn’t actually change the token stream itself.

Once you have executed all of your modification calls, you’ll need to call .getText() (or .getText(String programName)) to get get a String that has all of your changes incorporated. Then you can use that as the input to your Lexer to get a token stream containing your modifications.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Mike Cargal

Related Questions