This is a re-roll of CL 39710,
which broke deterministic builds.
typenamesym is called from three places:
typename, ngotype, and Type.Symbol.
Only in typename do we actually need a Node.
ngotype and Type.Symbol require only a Sym.
And writing the newly created Node to
Sym.Def is unsafe in a concurrent backend.
Rather than use a mutex protect to Sym.Def,
make typenamesym not touch Sym.Def.
The assignment to Sym.Def was serving a second purpose,
namely to prevent duplicate entries on signatlist.
Preserve that functionality by switching signatlist to a map.
This in turn requires that we sort signatlist
when exporting it, to preserve reproducibility.
We sort using exactly the same mechanism
that the export code (dtypesym) uses.
Failure to do that led to non-deterministic builds (#19872).
Since we've already calculated the Type's export name,
we could pass it to dtypesym, sparing it a bit of work.
That can be done as a future optimization.
cmd/compile: allow composite literal structs with _ fields
Given code such as
type T struct {
_ string
}
func f() {
var x = T{"space"}
// ...
}
the compiler rewrote the 'var x' line as
var x T
x._ = "space"
The compiler then rejected the assignment to
a blank field, thus rejecting valid code.
It also failed to catch a number of invalid assignments.
And there were insufficient checks for validity
when emitting static data, leading to ICEs.
To fix, check earlier for explicit blanks field names,
explicitly handle legit blanks in sinit,
and don't try to emit static data for nodes
for which typechecking has failed.
This dowidth currently happens during AST to SSA conversion.
As such, it is a concurrency pinch point.
It's a bit silly, but do it here in walk instead.
This appears (fingers crossed) to be the last
unresolved dowidth concurrency problem.
Dave Cheney [Fri, 7 Apr 2017 10:07:54 +0000 (20:07 +1000)]
cmd/compile/internal/gc: remove queuemethod
queuemethod was unused. As queuemethod is unused, nothing appends to the
methodqueue global. As methodqueue is always nil or empty, there are no
live callers of domethod, so it can be removed.
Change-Id: Ic7427ac4621bbf403947815e3988c3a1113487f2
Reviewed-on: https://go-review.googlesource.com/39931
Run-TryBot: Dave Cheney <dave@cheney.net>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Matthew Dempsky <mdempsky@google.com>
Matthew Dempsky [Fri, 7 Apr 2017 20:47:10 +0000 (13:47 -0700)]
cmd/compile/internal/gc: cleanup mkinlcall
I had too many failed attempts trying to remove iterFields that I
decided to overhaul this function. Much simpler and easier to
understand now (at least IMO).
Passes toolstash-check -all.
Change-Id: I41d00642a969698df3f4689e41a386346b966638
Reviewed-on: https://go-review.googlesource.com/39856
Run-TryBot: Matthew Dempsky <mdempsky@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Robert Griesemer <gri@golang.org>
Hiroshi Ioka [Thu, 19 Jan 2017 23:34:18 +0000 (08:34 +0900)]
cmd/cgo: support floating point #define macros
Current code doesn't support floating point #define macros.
This CL compiles floats to a object file and retrive values from it.
That approach is the same work as we've already done for integers.
Updates #18720
Change-Id: I88b7ab174d0f73bda975cf90c5aeb797961fe034
Reviewed-on: https://go-review.googlesource.com/35511 Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
There are some LSyms that are lazily initialized,
and which cannot be made eagerly initialized,
such as elements of a constant pool.
To avoid needing a mutex to protect the internals of
those LSyms, this CL introduces LookupInit,
which allows an LSym to be initialized only once.
By itself this is not fully concurrency-safe,
but Ctxt.Hash will need mutex protection anyway,
and that will be enough to support one-time LSym initialization.
Alessandro Arzilli [Tue, 7 Mar 2017 17:59:14 +0000 (18:59 +0100)]
cmd/compile: output DWARF lexical blocks for local variables
Change compiler and linker to emit DWARF lexical blocks in debug_info.
Version of debug_info is updated from DWARF v.2 to DWARF v.3 since version 2
does not allow lexical blocks with discontinuous ranges.
Second attempt at https://go-review.googlesource.com/#/c/29591/
Remaining open problems:
- scope information is removed from inlined functions
- variables in debug_info do not have DW_AT_start_scope attributes so a
variable will shadow other variables with the same name as soon as its
containing scope begins, before its declaration.
Updates golang/go#12899, golang/go#6913
Change-Id: I0e260a45b564d14a87b88974eb16c5387cb410a5
Reviewed-on: https://go-review.googlesource.com/36879
Run-TryBot: Matthew Dempsky <mdempsky@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Matthew Dempsky <mdempsky@google.com>
Ilya Tocar [Tue, 28 Mar 2017 20:30:31 +0000 (15:30 -0500)]
cmd/compile/internal/gc: improve comparison with constant strings
Currently we expand comparison with small constant strings into len check
and a sequence of byte comparisons. Generate 16/32/64-bit comparisons,
instead of bytewise on 386 and amd64. Also increase limits on what is
considered small constant string.
Shaves ~30kb (0.5%) from go executable.
This also updates test/prove.go to keep test case valid.
Robert Griesemer [Wed, 5 Apr 2017 00:54:02 +0000 (17:54 -0700)]
cmd/compile: factor out Pkg, Sym, and Type into package types
- created new package cmd/compile/internal/types
- moved Pkg, Sym, Type to new package
- to break cycles, for now we need the (ugly) types/utils.go
file which contains a handful of functions that must be installed
early by the gc frontend
- to break cycles, for now we need two functions to convert between
*gc.Node and *types.Node (the latter is a dummy type)
- adjusted the gc's code to use the new package and the conversion
functions as needed
- made several Pkg, Sym, and Type methods functions as needed
- renamed constructors typ, typPtr, typArray, etc. to types.New,
types.NewPtr, types.NewArray, etc.
Passes toolstash-check -all.
Change-Id: I8adfa5e85c731645d0a7fd2030375ed6ebf54b72
Reviewed-on: https://go-review.googlesource.com/39855 Reviewed-by: Matthew Dempsky <mdempsky@google.com>
typenamesym is called from three places:
typename, ngotype, and Type.Symbol.
Only in typename do we actually need a Node.
ngotype and Type.Symbol require only a Sym.
And writing the newly created Node to
Sym.Def is unsafe in a concurrent backend.
Rather than use a mutex protect to Sym.Def,
make typenamesym not touch Sym.Def.
The assignment to Sym.Def was serving a second purpose,
namely to prevent duplicate entries on signatlist.
Preserve that functionality by switching signatlist to a map.
This in turn requires that we sort signatlist
when exporting it, to preserve reproducibility.
We'd like to use Type.cmp for sorting,
but that causes infinite recursion at the moment;
see #19869.
For now, use Type.LongString as the sort key,
which is a complete description of the type.
Type.LongString is relatively expensive,
but we calculate it only once per type,
and signatlist is generally fairly small,
so the performance impact is minimal.
When a constant is both MOVCON (can fit into a MOV instruction)
and BITCON (can fit into a logical instruction), the assembler
chooses to use the MOVCON encoding, which is actually longer for
logical instructions. We add MBCON rules explicitly to make sure
it uses the BITCON encoding.
cmd/compile: use ANDconst to mask out leading/trailing bits on ARM64
For an AND that masks out leading or trailing bits, generic rules
rewrite it to a pair of shifts. On ARM64, the mask actually can
fit into an AND instruction. So we rewrite it back to AND.
Jeff Wendling [Thu, 6 Apr 2017 15:45:30 +0000 (15:45 +0000)]
cmd/dist: disable plugin test on linux-arm with GOARM=5
Plugin support is patchy at the moment, so disable the test for
now until the test can be fixed. This way, we can get builders
for ARMv5 running for the rest of the code.
The existing bulk/cached Prog allocator, Ctxt.NewProg, is not concurrency-safe.
This CL moves Prog allocation to its clients, the compiler and the assembler.
The assembler is so fast and generates so few Progs that it does not need
optimization of Prog allocation. I could not generate measureable changes.
And even if I could, the assembly is a miniscule portion of build times.
The compiler already has a natural place to manage Prog allocation;
this CL migrates the Prog cache there.
It will be made concurrency-safe in a later CL by
partitioning the Prog cache into chunks and assigning each chunk
to a different goroutine to manage.
This CL does cause a performance degradation when the compiler
is invoked with the -S flag (to dump assembly).
However, such usage is rare and almost always done manually.
The one instance I know of in a test is TestAssembly
in cmd/compile/internal/gc, and I did not detect
a measurable performance impact there.
Passes toolstash-check -all.
Minor compiler performance impact.
cmd/compile: teach assemblers to accept a Prog allocator
The existing bulk Prog allocator is not concurrency-safe.
To allow for concurrency-safe bulk allocation of Progs,
I want to move Prog allocation and caching upstream,
to the clients of cmd/internal/obj.
This is a preliminary enabling refactoring.
After this CL, instead of calling Ctxt.NewProg
throughout the assemblers, we thread through
a newprog function that returns a new Prog.
That function is set up to be Ctxt.NewProg,
so there are no real changes in this CL;
this CL only establishes the plumbing.
Austin Clements [Thu, 16 Mar 2017 18:55:10 +0000 (14:55 -0400)]
runtime: initialize more fields of stack spans
Stack spans don't internally use many of the fields of the mspan,
which means things like the size class and element size get left over
from whatever last used the mspan. This can lead to confusing crashes
and debugging.
Zero these fields or initialize them to something reasonable. This
also lets us simplify some code that currently has to distinguish
between heap and stack spans.
Commit 44ed88a5a7 moved printing of the "sweep done" gcpacertrace
message so that it is printed when the final sweeper finishes.
However, by this point some other thread has often already observed
that there are no more spans to sweep and zeroed sweepPagesPerByte.
Avoid printing a 0 sweep ratio in the trace when this race happens by
getting the value of the sweep ratio upon entry to sweepone and
printing that.
The comments in this package state that users should be
migrating code that uses the syscall package to its
corresponding package in x/sys. However, the syscall.Signal
and syscall.Errno types and the syscall.SysProcAttr struct is
not defined in the x/sys package and still need to be referenced
from within syscall. This adds a change to the comments to
clarify that the migration will need to continue to use some
references to syscall for now.
Fixes #19560
Change-Id: I8abb96b93bea90070ce461da16dc7bcf7b4b29c1
Reviewed-on: https://go-review.googlesource.com/39450 Reviewed-by: Rob Pike <r@golang.org>
Alex Brainman [Mon, 13 Feb 2017 03:52:19 +0000 (14:52 +1100)]
path/filepath: add test to walk symlink
For #17540.
Change-Id: Ie01f39797526934fa553f4279cbde6c7cbf14154
Reviewed-on: https://go-review.googlesource.com/36854 Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
This is a holdover from the days when we did not
have full SSA coverage and compiled things optimistically,
and catching the panic obscures useful information.
Filip Gruszczyński [Sat, 18 Mar 2017 03:10:38 +0000 (20:10 -0700)]
reflect: add MakeMapWithSize for creating maps with size hint
Providing size hint when creating a map allows avoiding re-allocating
underlying data structure if we know how many elements are going to
be inserted. This can be used for example during decoding maps in
gob.
Fixes #19599
Change-Id: I108035fec29391215d2261a73eaed1310b46bab1
Reviewed-on: https://go-review.googlesource.com/38335 Reviewed-by: Rob Pike <r@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Martin Möhrmann [Mon, 6 Mar 2017 08:34:39 +0000 (09:34 +0100)]
strings: speed up Fields
- use a string lookup to detect if a single byte is a space character
- determine the exact number of fields for ASCII and
a possibly underestimated number of fields for non ASCII strings
by doing a separate byte for byte scan of the input string
before collecting the fields in an extra pass
- provide a fast path for ASCII only strings when collecting the fields
- avoid utf8.DecodeRuneInString and unicode.IsSpace for ASCII characters
Used golang.org/cl/33108 from Joe Tsai as starting point.
Keith Randall [Fri, 17 Mar 2017 04:33:03 +0000 (21:33 -0700)]
cmd/compile: intrinsics for math/bits.OnesCount
Popcount instructions on amd64 are not guaranteed to be
present, so we must guard their call. Rewrite rules can't
generate control flow at the moment, so the intrinsifier
needs to generate that code.
When a map is small, it's not worth putting
the contents in an array and then looping over the array.
Just generate code instead.
This makes smaller binaries.
It might also be better for cache lines.
It also can avoids adding control flow in the middle
of the init function, which can be very large.
Eliminating this source of extra blocks
makes phi insertion easier for temp-heavy init functions.
This reduces the time required for compiler to
panic while compiling the code in #19751
from 15 minutes to 45 seconds.
The cutoff of 25 was chosen fairly unscientifically
by looking at the size of cmd/go.
There are probably more sophisticated mechanisms available.
For example, the smaller the key/value sizes, the better
generated code will be vs a table.
Nevertheless this is simple and seems like a good start.
Subsequent rules can then assume if there is a constant arg to Eq64,
it will be the first one. This fix kinda works, but it is fragile and
only works when we remember to include the required extra rules.
The fundamental problem is that the rule matcher doesn't
know anything about commuting ops. This CL fixes that fact.
We already have information about which ops commute. (The register
allocator takes advantage of commutivity.) The rule generator now
automatically generates multiple rules for a single source rule when
there are commutative ops in the rule. We can now drop all of our
almost-duplicate source-level rules and the canonicalization rules.
I have some CLs in progress that will be a lot less verbose when
the rule generator handles commutivity for me.
I had to reorganize the load-combining rules a bit. The 8-way OR rules
generated 128 different reorderings, which was causing the generator
to put too much code in the rewrite*.go files (the big ones were going
from 25K lines to 132K lines). Instead I reorganized the rules to
combine pairs of loads at a time. The generated rule files are now
actually a bit (5%) smaller.
Make.bash times are ~unchanged.
Compiler benchmarks are not observably different. Probably because
we don't spend much compiler time in rule matching anyway.
I've also done a pass over all of our ops adding commutative markings
for ops which hadn't had them previously.
Fixes #18292
Change-Id: Ic1c0e43fbf579539f459971625f69690c9ab8805
Reviewed-on: https://go-review.googlesource.com/38801
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: David Chase <drchase@google.com>
Instead of walking the list of nodes twice,
once to find static entries to add to an array
and once to find dynamic entries to generate code for,
do the split once up front, into two slices.
Then process each slice individually.
This makes the code easier to read
and more importantly, easier to modify.
While we're here, add a TODO to avoid
using temporaries for mapassign_fast calls.
It's not an important TODO;
the generated code would be basically identical.
It would just avoid a minor amount of
pointless SSA optimization work.
Passes toolstash-check.
No measureable compiler performance impact.
David Chase [Mon, 3 Apr 2017 15:50:54 +0000 (11:50 -0400)]
cmd/compile: rewrite upper-bit-clear idiom to use shift-rotate
Old buggy hardware incorrectly executes the shift-left-K
then shift-right-K idiom for clearing K leftmost bits.
Use a right rotate instead of shift to avoid triggering the
bug.
Daniel Martí [Mon, 3 Apr 2017 14:18:30 +0000 (15:18 +0100)]
go/parser: fix example to run on the playground
The example shouldn't rely on the existance of example_test.go. That
breaks in the playground, which is what the "run" button in
https://golang.org/pkg/go/parser/#example_ParseFile does.
Make the example self-sufficient by using a small piece of source via a
string literal instead.
Fixes #19823.
Change-Id: Ie8a3c6c5d00724e38ff727862b62e6a3621adc88
Reviewed-on: https://go-review.googlesource.com/39236
Run-TryBot: Daniel Martí <mvdan@mvdan.cc> Reviewed-by: Robert Griesemer <gri@golang.org>
cmd/internal/obj: use string instead of LSym in Pcln
In a concurrent backend, Ctxt.Lookup will need some
form of concurrency protection, which will make it
more expensive.
This CL changes the pcln table builder to track
filenames as strings rather than LSyms.
Those strings are then converted into LSyms
at the last moment, for writing the object file.
This CL removes over 85% of the calls to Ctxt.Lookup
in a run of make.bash.
Russ Cox [Fri, 31 Mar 2017 16:46:35 +0000 (12:46 -0400)]
testing/quick: generate all possible int64, uint64 values
When generating a random int8, uint8, int16, uint16, int32, uint32,
quick.Value chooses among all possible values.
But when generating a random int64 or uint64, it only chooses
values in the range [-2⁶², 2⁶²) (even for uint64).
It should, like for all the other integers, use the full range.
If it had, this would have caught #19807 earlier.
Instead it let us discover the presence of #19809.
While we are here, also make the default source of
randomness not completely deterministic.
nodfp is a global, so modifying it is unsafe in a concurrent backend.
It is also not necessary, since the Used marks
are only relevant for nodes in fn.Dcl.
For good measure, mark nodfp as always used.
Cherry Zhang [Fri, 31 Mar 2017 11:14:16 +0000 (07:14 -0400)]
cmd/link: canonicalize the "package" of dupok text symbols
Dupok symbols may be defined in multiple packages. Its associated
package is chosen sort of arbitrarily (the first containing package
that the linker loads). Canonicalize its package to the package
with which it will be laid down in text, which is the first package
in dependency order that defines the symbol. So later passes (for
example, trampoline insertion pass) know that the dupok symbol
is laid down along with the package.
cmd/compile: don't mutate shared nodes in orderinit
A few gc.Node ops may be shared across functions.
The compiler is (mostly) already careful to avoid mutating them.
However, from a concurrency perspective, replacing (say)
an empty list with an empty list still counts as a mutation.
One place this occurs is orderinit. Avoid it.
This requires fixing one spot where shared nodes were mutated.
It doesn't result in any functional or performance changes.
Russ Cox [Fri, 31 Mar 2017 16:34:25 +0000 (12:34 -0400)]
time: test and fix Time.Round, Duration.Round for d > 2⁶²
Round uses r+r < d to decide whether the remainder is
above or below half of d (to decide whether to round up or down).
This is wrong when r+r wraps negative, because it looks < d
but is really > d.
No one will ever care about rounding to a multiple of
d > 2⁶² (about 146 years), but might as well get it right.
Fixes #19807.
Change-Id: I1b55a742dc36e02a7465bc778bf5dd74fe71f7c0
Reviewed-on: https://go-review.googlesource.com/39151
Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-by: Rob Pike <r@golang.org> Reviewed-by: David Chase <drchase@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
newnamel is newname but with no dependency on lineno or Curfn.
This makes it suitable for use in a concurrent back end.
Use it now to make tempAt global-free.
The decision to push the assignment to n.Name.Curfn
to the caller of newnamel is based on mdempsky's
comments in #19683 that he'd like to do that
for callers of newname as well.
Passes toolstash-check. No compiler performance impact.
Convert yyerrors into Fatals.
Remove the goto.
Move variable declaration closer to use.
Unify printing strings a bit.
Convert an int param into a bool.
Passes toolstash-check. No compiler performance impact.
Robert Griesemer [Fri, 31 Mar 2017 17:26:30 +0000 (10:26 -0700)]
cmd/compile: use std "DO NOT EDIT" comment for generated files
Also: Fix (testdata/gen/) copyGen.go, zeroGen.go, and arithConstGen.go
to actually match (testdata/) copy.go, zero.go, and arithConst.go, all
of which were manually edited in https://go-review.googlesource.com/20823
and https://go-review.googlesource.com/22748 despite the 'do not edit'
(or perhaps because it was missing in the case of arithConst.go).
For #13560.
Change-Id: I366e1b521e51885e0d318ae848760e5e14ccd488
Reviewed-on: https://go-review.googlesource.com/39172 Reviewed-by: Rob Pike <r@golang.org>
cmd/compile: catch and report nowritebarrier violations later
Prior to this CL, the SSA backend reported violations
of the //go:nowritebarrier annotation immediately.
This necessitated emitting errors during SSA compilation,
which is not compatible with a concurrent backend.
Instead, check for such violations later.
We already save the data required to do a late check
for violations of the //go:nowritebarrierrec annotation.
Use the same data, and check //go:nowritebarrier at the same time.
One downside to doing this is that now only a single
violation will be reported per function.
Given that this is for the runtime only,
and violations are rare, this seems an acceptable cost.
While we are here, remove several 'nerrors != 0' checks
that are rendered pointless.
Updates #15756
Fixes #19250 (as much as it ever can be)
cmd/compile: rework reporting of oversized stack frames
We don't support stack frames over 2GB.
Rather than detect this during backend compilation,
check for it at the end of compilation.
This is arguably a more accurate check anyway,
since it takes into account the full frame,
including local stack, arguments, and arch-specific
rounding, although it's unlikely anyone would ever notice.
Also, rather than reporting the error right away,
take note of it and report it later, at the top level.
This is not relevant now, but it will help with making
the backend concurrent, as the append to the list of
oversized functions can be cheaply protected by a plain mutex.