5.18. Returning Values from yylex()

The C code executed when a pattern matches a token can contain a return statement that returns a value from yylex() to its caller, typically a parser generated by yacc. The next time yylex() is called, the scanner picks up where it left off. When a scanner matches a token of interest to the parser (e.g., a keyword, variable name, or operator), it returns to pass the token back to the parser. When it matches a token not of interest to the parser (e.g., whitespace or a comment), it does not return, and the scanner immediately proceeds to match another token.

This means that you cannot restart a lexer just by calling yylex(). You have to reset it into the default state using BEGIN INITIAL and reset the input state so that the next call to input() will start reading the new input. A call to yyrestart(file), where file is a standard I/O file pointer, arranges to start reading from that file.