cmd/compile: adjust starting token value

The actual values assigned to tokens was inherited from the yacc-based
grammar. With the most recent cleanups, all single-char tokens such as
commas, semis, parens, etc., that get returned from lexer.next simply
as their Unicode values are below utf8.RuneSelf (i.e., 7bit ASCII).
Lower the initial starting value for named token constants accordingly.

Change-Id: I7eb8e584dbb3bc7f9dab849d1b68a91320cffebd
Reviewed-on: https://go-review.googlesource.com/19913
Reviewed-by: Matthew Dempsky <mdempsky@google.com>

diff --git a/src/cmd/compile/internal/gc/lex.go b/src/cmd/compile/internal/gc/lex.go
index 71ed8d68413e6665199d1ee2fd3bf9f34c5ba47d..8809ac40162b05872049f3e84d48a942a752219f 100644 (file)
--- a/src/cmd/compile/internal/gc/lex.go
+++ b/src/cmd/compile/internal/gc/lex.go
@@ -873,7 +873,9 @@ type lexer struct {
 }
 
 const (
-       LLITERAL = 57346 + iota
+       // The value of single-char tokens is just their character's Unicode value.
+       // They are all below utf8.RuneSelf. Shift other tokens up to avoid conflicts.
+       LLITERAL = utf8.RuneSelf + iota
        LASOP
        LCOLAS
        LBREAK
@@ -955,6 +957,7 @@ l0:
                }
                return
        }
+       // c < utf8.RuneSelf
 
        var c1 rune
        var op Op