go/scanner: emit implicit semicolon tokens in correct order
Prior to this change, the scanner, in ScanComments mode, would emit
the implicit SEMICOLON token generated by a newline ahead of any
immediately preceding comment "tokens". For example:
foo /*a*/ /*b*/ /*c*/ \n => [IDENT SEMICOLON COMMENT COMMENT COMMENT]
Now, tokens and comments are emitted by the scanner in lexical order
of their start position. SEMICOLON tokens corresponding to newlines
always have the position of the newline (even in the special case
in which the newline is in the middle of a general comment /*\n*/).
The scanner no longer needs to look ahead, possibly across multiple
comments, for a newline, when it encounters a comment.
The scanner semicolon tests have been rewritten to be less magical.
The parser must now expect line comments before an implicit semicolon.
Line comments for an explicit semicolon still appear after.
The new assertions in the parser TestLeadAndLineComments are
not changes to behavior.
Fixes golang/go#54941
Change-Id: Iffe97fd10e9e0b52882da8659307698ccb31c093
Reviewed-on: https://go-review.googlesource.com/c/go/+/429635
Auto-Submit: Alan Donovan <adonovan@google.com>
Reviewed-by: Robert Griesemer <gri@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Alan Donovan <adonovan@google.com>