ValueError: Ran into a $end where it wasn't expected - RPLY Parsing

Question

I'm trying to create a parser for a basic interpreted language.

When I run my program I get the following error:

ValueError: Ran into a $end where it wasn't expected

Here is my main.py:

from lexer import Lexer #imported from the lexer file
from parser_class_file import ParserClass
import time

#the actual code
text_input = "print(4 - 2);"


#creating the lexer object from my lexer file
lexer = Lexer().get_lexer()

#'lexs' the text_input
tokens = lexer.lex(text_input)

#prints all the tokens in the object
for token in tokens:
  print(token)

time.sleep(1)
pg = ParserClass()
pg.parse_()
parser_object = pg.get_parser()
parser_object.parse(tokens)

The last line of the above code is the problem statement.

Here is the parse_class_file.py:

"""Generating the parser"""
from rply import ParserGenerator
from ast import *

class ParserClass():

    def __init__(self):
        self.pg = ParserGenerator(
        #TOKENS ACCEPTED BY THE PARSER
        tokens=['NUMBER', 'PRINT', 'OPEN_PAREN', 'CLOSE_PAREN','SEMI_COLON', 'SUM', 'SUB', '$end']
    )

def parse_(self):
    @self.pg.production('program : PRINT OPEN_PAREN expression CLOSE_PAREN SEMI_COLON $end')
    def program(p):
        return Print(p[2]) #why the p[2]

    @self.pg.production('expression : expression SUM expression')
    @self.pg.production('expression : expression SUB expression')
    def expression(p):
        left = p[0]
        right = p[2]
        operator = p[1]

        if operator.gettokentype() == 'SUM':
            return Sum(left, right)
        elif operator.gettokentype() == 'SUB':
            return Sub(left, right)

    @self.pg.production('expression : NUMBER')
    def number(p):
        return Number(p[0].value)

    @self.pg.production('expression : expression $end')
    def end(p):
        print(p[3])

    @self.pg.error
    def error_handle(token):
        raise ValueError("Ran into a %s where it wasn't expected" % token.gettokentype())

def get_parser(self):
    return self.pg.build()

The line that says raise ValueError("Ran into a %s where it wasn't expected" % token.gettokentype()) is the problem statement in this file.

I left out my lexer and ast files as they don't seem to be involved in the error.

How do I fixed the error and address the presence of a $end token?

Answer 1

tokens = lexer.lex(text_input)

The lexer produced by Rply is a generator, not a list. Generators can only be iterated once. So once you do this:

#prints all the tokens in the object
for token in tokens:
  print(token)

you have completely consumed the generator. It ran to the end and it has nothing more to give.

You then call your parser with an exhausted token stream, which will raise StopIteration as soon as you ask it for the next token.

Since your grammar does not accept empty inputs, the parser will then produce an "unexpected end of input" error.

If you just don't attempt to print the token stream before parsing, you won't run into this particular error.