Bryan C. Mills [Thu, 21 Nov 2019 21:26:09 +0000 (16:26 -0500)]
cmd/go: do not panic when computing Shlib for a package with no Target
In module mode, a non-main package lacks an install target.
The location of the .shlib corresponding to a given target is stored
in a .shlibname file alongside its install target, so in module mode
a non-main package also lacks a .shlibname file.
This also implies that such a package cannot be installed with
'go install -buildmode=linkshared', but that is a problem
for another day.
Fixes #35759
Updates #34347
Change-Id: Id3e0e068266d5fb9b061a59e70f9a65985d4973b
Reviewed-on: https://go-review.googlesource.com/c/go/+/208233
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Jay Conrod <jayconrod@google.com>
Bryan C. Mills [Fri, 22 Nov 2019 18:26:58 +0000 (13:26 -0500)]
cmd/go: add a 'buildmode' condition for script tests
In CL 208233 I am fixing a panic that occurs only with a specific
build mode. I want that test to run on all platforms that support that
build mode, but the logic for determining support is somewhat
involved.
For now, I am duplicating that logic into the cmd/internal/sys
package, which already reports platform support for other build flags.
We can refactor cmd/go/internal/work to use the extracted function in
a followup CL.
Updates #35759
Change-Id: Ibbaedde4d1e8f683c650beedd10849bc27e7a6e7
Reviewed-on: https://go-review.googlesource.com/c/go/+/208457
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Jay Conrod <jayconrod@google.com>
Bryan C. Mills [Thu, 21 Nov 2019 20:03:24 +0000 (15:03 -0500)]
misc/cgo/testshared: make -v output less verbose
Previously, 'go test -v' in this directory would result in a massive
dump of go command output, because the test plumbed -v to 'build -x'.
This change separates them into distinct flags, so that '-v' only
implies the display of default 'go' command output.
Updates #30316
Change-Id: Ifb125f35ec6a0bebe7e8286e7c546d132fb213df
Reviewed-on: https://go-review.googlesource.com/c/go/+/208232
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Michael Anthony Knyszek [Fri, 22 Nov 2019 16:34:16 +0000 (16:34 +0000)]
runtime: release worldsema before Gosched in STW GC mode
After CL 182657 we no longer hold worldsema across the GC, we hold
gcsema instead.
However in STW GC mode we don't release worldsema before calling Gosched
on the user goroutine (note that user goroutines are disabled during STW
GC) so that user goroutine holds onto it. When the GC is done and the
runtime inevitably wants to "stop the world" again (though there isn't
much to stop) it'll sit there waiting for worldsema which won't be
released until the aforementioned goroutine is scheduled, which it won't
be until the GC is done!
So, we have a deadlock.
The fix is easy: just release worldsema before calling Gosched.
TestPhysicalMemoryUtilization occasionally fails on some platforms by
only a small margin. The reason for this is that it assumes the
scavenger will always be able to scavenge all the memory that's released
by sweeping, but because of the page cache, there could be free and
unscavenged memory held onto by a P which the scavenger simply cannot
get to.
As a result, if the page cache gets filled completely (512 KiB of free
and unscavenged memory) this could skew a test which expects to
scavenge roughly 8 MiB of memory. More specifically, this is 512 KiB of
memory per P, and if a system is more inclined to bounce around
between Ps (even if there's only one goroutine), this memory can get
"stuck".
Through some experimentation, I found that failures correlated highly
with relatively large amounts of memory ending up in some page cache
(like 60 or 64 pages) on at least one P.
This change changes the test's threshold such that it accounts for the
page cache, and scales up with GOMAXPROCS. Because the test constants
themselves don't change, however, the test must now also bound
GOMAXPROCS such that the threshold doesn't get too high (at which point
the test becomes meaningless).
Fixes #35580.
Change-Id: I6bdb70706de991966a9d28347da830be4a19d3a1
Reviewed-on: https://go-review.googlesource.com/c/go/+/208377
Run-TryBot: Michael Knyszek <mknyszek@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Keith Randall <khr@golang.org>
Jay Conrod [Thu, 21 Nov 2019 23:54:38 +0000 (18:54 -0500)]
cmd/go: add 'go generate' commands to modfile_flag test
Verify that 'go generate' works with -modfile. Also check that
go commands starts with 'go generate' do not inherit -modfile, but
they should still work if -modfile is set in GOFLAGS.
Updates #34506
Change-Id: I5e1f897b4e38e4fdaccc0fbb7a71b8d0e9fc0660
Reviewed-on: https://go-review.googlesource.com/c/go/+/208236
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
Bryan C. Mills [Wed, 20 Nov 2019 19:39:19 +0000 (14:39 -0500)]
misc/cgo/testcshared: avoid writing to GOROOT in tests
The tests in this package invoked 'go install -i -buildmode=c-shared'
in order to generate an archive as well as multiple C header files.
Unfortunately, the behavior of the '-i' flag is inappropriately broad
for this use-case: it not only generates the library and header files
(as desired), but also attempts to install a number of (unnecessary)
archive files for transitive dependencies to
GOROOT/pkg/$GOOS_$GOARCH_testcshared_shared, which may not be writable
— for example, if GOROOT is owned by the root user but the test is
being run by a non-root user.
Instead, for now we generate the header files for transitive dependencies
separately by running 'go tool cgo -exportheader'.
In the future, we should consider how to improve the ergonomics for
generating transitive header files without coupling that to
unnecessary library installation.
Updates #28387
Updates #30316
Updates #35715
Change-Id: I622426a860828020d98f7040636f374e5c766d28
Reviewed-on: https://go-review.googlesource.com/c/go/+/208119
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Bryan C. Mills [Wed, 20 Nov 2019 15:19:43 +0000 (10:19 -0500)]
misc/cgo/testcarchive: avoid writing to GOROOT in tests
Also add a -testwork flag to facilitate debugging the test itself.
Three of the tests of this package invoked 'go install -i
-buildmode=c-archive' in order to generate an archive as well as
multiple C header files.
Unfortunately, the behavior of the '-i' flag is inappropriately broad
for this use-case: it not only generates the library and header files
(as desired), but also attempts to install a number of (unnecessary)
archive files for transitive dependencies to
GOROOT/pkg/$GOOS_$GOARCH_shared, which may not be writable — for
example, if GOROOT is owned by the root user but the test is being run
by a non-root user.
Instead, for now we generate the header files for transitive dependencies
separately by running 'go tool cgo -exportheader'.
In the future, we should consider how to improve the ergonomics for
generating transitive header files without coupling that to
unnecessary library installation.
Updates #28387
Updates #30316
Updates #35715
Change-Id: I3d483f84e22058561efe740aa4885fc3f26137b5
Reviewed-on: https://go-review.googlesource.com/c/go/+/208117
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Hana Kim [Mon, 18 Nov 2019 17:44:33 +0000 (12:44 -0500)]
runtime/pprof: avoid crash due to truncated stack traces
The profile proto message builder maintains a location entry cache
that maps a location (possibly involving multiple user frames
that represent inlined function calls) to the location id. We have
been using the first pc of the inlined call sequence as the key of
the cached location entry assuming that, for a given pc, the sequence
of frames representing the inlined call stack is deterministic and
stable. Then, when analyzing the new stack trace, we expected the
exact number of pcs to be present in the captured stack trace upon
the cache hit.
This assumption does not hold, however, in the presence of the stack
trace truncation in the runtime during profiling, and also with the
potential bugs in runtime.
A better fix is to use all the pcs of the inlined call sequece as
the key instead of the first pc. But that is a bigger code change.
This CL avoids the crash assuming the trace was truncated.
Fixes #35538
Change-Id: I8c6bae98bc8b178ee51523c7316f56b1cce6df16
Reviewed-on: https://go-review.googlesource.com/c/go/+/207609
Run-TryBot: Hyang-Ah Hana Kim <hyangah@gmail.com> Reviewed-by: Keith Randall <khr@golang.org>
David Chase [Tue, 19 Nov 2019 18:23:35 +0000 (13:23 -0500)]
cmd/compile: try harder to not use an empty src.XPos for a bogus line
The fix for #35652 did not guarantee that it was using a non-empty
src position to replace an empty one. The new code checks again
and falls back to a more certain position. (The input in question
compiles to a single empty infinite loop, and none of the actual instructions
had any source position at all. That is a bug, but given the pathology
of this input, not one worth dealing with this late in the release cycle,
if ever.)
TODO: Add runtime.InfiniteLoop(), replace infinite loops with a call to
that, and use an eco-friendly runtime.gopark instead. (This was Cherry's
excellent idea.)
Roberto Clapis [Mon, 18 Nov 2019 09:05:07 +0000 (10:05 +0100)]
text/template: harden JSEscape to also escape ampersand and equal
Ampersand and equal are not dangerous in a JS/JSString context
but they might cause issues if interpolated in HTML attributes.
This change makes it harder to introduce XSS by misusing
escaping.
Thanks to t1ddl3r <t1ddl3r@gmail.com> for reporting this common
misuse scenario.
Fixes #35665
Change-Id: Ice6416477bba4cb2ba2fe2cfdc20e027957255c0
Reviewed-on: https://go-review.googlesource.com/c/go/+/207637 Reviewed-by: Filippo Valsorda <filippo@golang.org> Reviewed-by: Mike Samuel <mikesamuel@gmail.com> Reviewed-by: Andrew Bonventre <andybons@golang.org> Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Jay Conrod [Thu, 21 Nov 2019 17:50:14 +0000 (12:50 -0500)]
cmd/go: report an error for 'go list -m ...' outside a module
Previously, we just reported an error for "all". Now we report an
error for any pattern that matches modules in the build list. The
build list can only contain the module "command-line-arguments", so
these patterns are not meaningful.
Fixes #35728
Change-Id: Ibc736491ec9164588f9657c09d1b9683b33cf1de
Reviewed-on: https://go-review.googlesource.com/c/go/+/208222
Run-TryBot: Jay Conrod <jayconrod@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Cherry Zhang [Tue, 19 Nov 2019 21:04:32 +0000 (16:04 -0500)]
runtime: relax TestAsyncPreempt
In TestAsyncPreempt, the function being tested for preemption,
although still asynchronously preemptible, may have only samll
ranges of PCs that are preemtible. In an unlucky run, it may
take quite a while to have a signal that lands on a preemptible
instruction. The test case is kind of an extreme. Relax it to
make it more preemptible.
In the original version, the first closure has more work to do,
and it is not a leaf function, and the second test case is a
frameless leaf function. In the current version, the first one
is also a frameless leaf function (the atomic is intrinsified).
Add some calls to it. It is still not preemptible without async
preemption.
Andrew [Wed, 20 Nov 2019 17:06:51 +0000 (12:06 -0500)]
all: base64-encode binaries that will cause Apple notarization to fail
Starting with macOS 10.15 (Catalina), Apple now requires all software
distributed outside of the App Store to be notarized. Any binaries we
distribute must abide by a strict set of requirements like code-signing
and having a minimum target SDK of 10.9 (amongst others).
Apple’s notarization service will recursively inspect archives looking to
find notarization candidate binaries. If it finds a binary that does not
meet the requirements or is unable to decompress an archive, it will
reject the entire distribution. From cursory testing, it seems that the
service uses content sniffing to determine file types, so changing
the file extension will not work.
There are some binaries and archives included in our distribution that
are being detected by Apple’s service as potential candidates for
notarization or decompression. As these are files used by tests and some
are intentionally invalid, we don’t intend to ever make them compliant.
As a workaround for this, we base64-encode any binaries or archives that
Apple’s notarization service issues a warning for, as these warnings will
become errors in January 2020.
Bryan C. Mills [Wed, 20 Nov 2019 19:43:04 +0000 (14:43 -0500)]
misc/cgo/testplugin: avoid writing to GOROOT
One of the 'go build' commands executed by this test passed the '-i'
flag, which caused the 'go' command to attempt to install transitive
standard-library dependencies to GOROOT/pkg/$GOOS_$GOARCH_dynlink.
That failed if GOROOT/pkg was not writable (for example, if GOROOT was
owned by the root user, but the user running the test was not root).
As far as I can tell the '-i' flag is not necessary in this test.
Prior to the introduction of the build cache it may have been an
optimization, but now that the build cache is required the '-i' flag
only adds extra work.
Bryan C. Mills [Wed, 20 Nov 2019 22:04:35 +0000 (17:04 -0500)]
misc/cgo/fortran: avoid writing to $PWD
The bash script that drives this test needs to know whether the
fortran compiler works, but it doesn't actually care about the
generated binary. Write that binary to /dev/null.
Eric Rutherford [Tue, 19 Nov 2019 02:35:33 +0000 (20:35 -0600)]
path: minor changes to improve documentation for Join
Reworking the comments in path to call out how leading
empty elements are treated. Also updating filepath.Join
since it shared much of the wording from path.Join.
Updates #35655
Change-Id: I5b15c5d36e9d19831ed39e6bcc7f2fd6c1330033
Reviewed-on: https://go-review.googlesource.com/c/go/+/207797 Reviewed-by: Rob Pike <r@golang.org>
Austin Clements [Fri, 25 Oct 2019 20:17:41 +0000 (16:17 -0400)]
runtime: support preemption on windows/{386,amd64}
This implements preemptM on Windows using SuspendThead and
ResumeThread.
Unlike on POSIX platforms, preemptM on Windows happens synchronously.
This means we need a make a few other tweaks to suspendG:
1. We need to CAS the G back to _Grunning before doing the preemptM,
or there's a good chance we'll just catch the G spinning on its
status in the runtime, which won't be preemptible.
2. We need to rate-limit preemptM attempts. Otherwise, if the first
attempt catches the G at a non-preemptible point, the busy loop in
suspendG may hammer it so hard that it never makes it past that
non-preemptible point.
Austin Clements [Tue, 19 Nov 2019 01:21:38 +0000 (20:21 -0500)]
runtime: ensure thread handle is valid in profileloop1
On Windows, there is currently a race between unminit closing the
thread's handle and profileloop1 suspending the thread using its
handle. If another handle reuses the same handle value, this can lead
to unpredictable results.
To fix this, we protect the thread handle with a lock and duplicate it
under this lock in profileloop1 before using it.
This is going to become a much bigger problem with non-cooperative
preemption (#10958, #24543), which uses the same basic mechanism as
profileloop1.
Clément Chigot [Wed, 20 Nov 2019 12:57:24 +0000 (13:57 +0100)]
runtime: disable GDB tests on AIX with -short
Since the new page allocator, AIX's GDB has trouble running Go programs.
It does work but it can be really slow. Therefore, they are disable when
tests are run with -short.
Updates: #35710
Change-Id: Ibfc4bd2cd9714268f1fe172aaf32a73612e262d6
Reviewed-on: https://go-review.googlesource.com/c/go/+/207919
Run-TryBot: Ian Lance Taylor <iant@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Bryan C. Mills [Tue, 19 Nov 2019 19:29:10 +0000 (14:29 -0500)]
cmd/dist: remove redundant 'go test -race' call
In CL 207962, I removed a seemingly-redundant -i flag. As it turns
out, the -i flag has *two* meanings: “install dependencies”, and “do
not actually run the test”. Without the flag, we omit the former
behavior, but add the latter.
We're about to run specific tests from these binaries on the very next
line, so don't preemptively run all of the tests.
Bryan C. Mills [Tue, 19 Nov 2019 18:48:35 +0000 (13:48 -0500)]
cmd/go: consolidate TestInstalls into gopath_install script test
TestInstalls was already mostly redundant with
TestInstallInto{GOPATH,GOBIN}, except for one additional check for the
install location of cmd/fix.
We can't assume that GOROOT is writable in general, so we also can't
assume that the test will be able to reinstall cmd/fix at run time.
Moreover, other processes running in parallel may expect to invoke
cmd/fix themselves, so this test temporarily removing it could induce
systemwide flakes.
We could carefully construct a parallel GOROOT and install cmd/fix
into it, but we can get *almost* as much coverage — at a much lower
cost — by checking the output of 'go list' instead of actually
rebuilding and reinstalling the binary.
Updates #28387
Updates #30316
Change-Id: Id49f44a68b0c52dfabb84c665f63c4e7db58dd49
Reviewed-on: https://go-review.googlesource.com/c/go/+/207965
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Jay Conrod <jayconrod@google.com>
Than McIntosh [Tue, 19 Nov 2019 15:09:28 +0000 (10:09 -0500)]
cmd/cgo: better handling for '.' in pkgpath for gccgo
Update gccgoPkgpathToSymbolNew() to bring it into conformance
with the way that gccgo now handles packagepaths with embedded
dots (see CL 200838). See also https://gcc.gnu.org/PR61880, a
related bug.
Updates #35623.
Change-Id: I32f064320b9af387fc17771530c745a9e3003c20
Reviewed-on: https://go-review.googlesource.com/c/go/+/207957
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Jay Conrod [Mon, 11 Nov 2019 23:19:04 +0000 (18:19 -0500)]
doc: add glossary to module reference documentation
These terms will be defined throughout the document, and more terms
will be added. After drafting a few sections, it's clear that a
glossary will be useful. There are enough terms that it would be
overwhelming at the beginning.
Also, add anchors for each heading and add a couple more headings.
Bryan C. Mills [Tue, 19 Nov 2019 18:25:03 +0000 (13:25 -0500)]
cmd/dist: remove extraneous '-i' from 'go test -race' command
At one point (before GOCACHE), the '-i' flag meant, effectively,
“save the intermediate results of this command to make
future commands faster”.
However, now that we require GOCACHE to be enabled everywhere, '-i' no
longer has that meaning: the intermediate results are already saved in
GOCACHE, so the -i flag merely adds extra work (copying or linking
things from GOCACHE into pkg), and also adds additional failure modes
resulting from that extra work (particularly when 'pkg' is read-only).
Since the flag now causes more harm than good, omit it.
Bryan C. Mills [Tue, 19 Nov 2019 17:05:30 +0000 (12:05 -0500)]
cmd/go: ignore irrelevant 'go test' failure in TestGoTestRaceInstallCgo
This test runs 'go test -race -i runtime/race' and checks that it did
not overwrite cmd/cgo.
If GOROOT/pkg is read-only and GOROOT/pkg/$GOOS_$GOARCH_race is not
already populated, as are the conditions if the Go toolchain was
installed from source as root using 'make.bash', then 'go test -race
-i' itself will fail because it cannot install packages to GOROOT/pkg.
However, such a failure is not relevant to the test: even if 'go test
-race -i' fails, we can still verify that it did not incidentally
overwrite cmd/cgo.
Updates #28387
Updates #30316
Change-Id: Iff2f75a0aeb4c926290ac3062c83695604522078
Reviewed-on: https://go-review.googlesource.com/c/go/+/207959
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Jay Conrod <jayconrod@google.com>
Ian Lance Taylor [Fri, 15 Nov 2019 18:05:13 +0000 (10:05 -0800)]
runtime: release timersLock while running timer
Dan Scales pointed out a theoretical deadlock in the runtime.
The timer code runs timer functions while holding the timers lock for a P.
The scavenger queues up a timer function that calls wakeScavenger,
which acquires the scavenger lock.
The scavengeSleep function acquires the scavenger lock,
then calls resetTimer which can call addInitializedTimer
which acquires the timers lock for the current P.
So there is a potential deadlock, in that the scavenger lock and
the timers lock for some P may both be acquired in different order.
It's not clear to me whether this deadlock can ever actually occur.
Issue 35532 describes another possible deadlock.
The pollSetDeadline function acquires pd.lock for some poll descriptor,
and in some cases calls resettimer which can in some cases acquire
the timers lock for the current P.
The timer code runs timer functions while holding the timers lock for a P.
The timer function for poll descriptors winds up in netpolldeadlineimpl
which acquires pd.lock.
So again there is a potential deadlock, in that the pd lock for some
poll descriptor and the timers lock for some P may both be acquired in
different order. I think this can happen if we change the deadline
for a network connection exactly as the former deadline expires.
Looking at the code, I don't see any reason why we have to hold
the timers lock while running a timer function.
This CL implements that change.
Updates #6239
Updates #27707
Fixes #35532
Change-Id: I17792f5a0120e01ea07cf1b2de8434d5c10704dd
Reviewed-on: https://go-review.googlesource.com/c/go/+/207348
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Michael Knyszek <mknyszek@google.com>
David Chase [Mon, 18 Nov 2019 19:14:22 +0000 (14:14 -0500)]
cmd/compile: make a better bogus line for empty infinite loops
The old recipe for making an infinite loop not be infinite
in the debugger could create an instruction (Prog) with a
line number not tied to any file (index == 0). This caused
downstream failures in DWARF processing.
So don't do that. Also adds a test, also adds a check+panic
to ensure that the next time this happens the error is less
mystifying.
Bryan C. Mills [Mon, 18 Nov 2019 21:09:37 +0000 (16:09 -0500)]
cmd/go: convert TestBuildDashIInstallsDependencies to a script test
Updates #28387
Updates #30316
Change-Id: I06e50c8d148cb4d7e08cdc2ba90de5e91d35781d
Reviewed-on: https://go-review.googlesource.com/c/go/+/207699
Run-TryBot: Bryan C. Mills <bcmills@google.com> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Cherry Zhang [Sat, 16 Nov 2019 22:08:11 +0000 (17:08 -0500)]
runtime: always use Go signal stack in non-cgo program
When initializing an M, we set up its signal stack to the gsignal
stack if an alternate signal stack is not already set. On Android,
an alternate signal stack is always set, even cgo is not used.
This breaks the logic of saving/fetching G on the signal stack
during VDSO, which assumes the signal stack is allocated by Go if
cgo is not used (if cgo is used, we use TLS for saving G).
When cgo is not used, we can always use the Go signal stack, even
if an alternate signal stack is already set. Since cgo is not
used, no one other than the Go runtime will care.
Fixes #35554.
Change-Id: Ia9d84cd55cb35097f3df46f37996589c86f10e0f
Reviewed-on: https://go-review.googlesource.com/c/go/+/207445
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Bryan C. Mills [Mon, 18 Nov 2019 13:58:58 +0000 (13:58 +0000)]
test: avoid writing temporary files to GOROOT
This reverts CL 207477, restoring CL 207352 with a fix for the
regression observed in the Windows builders.
cmd/compile evidently does not fully support NUL as an output on
Windows, so this time we write ignored 'compile' outputs
to temporary files (instead of os.DevNull as in CL 207352).
Clément Chigot [Wed, 13 Nov 2019 12:46:42 +0000 (13:46 +0100)]
runtime: add arenaBaseOffset on aix/ppc64
On AIX, addresses returned by mmap are between 0x0a00000000000000
and 0x0afffffffffffff. The previous solution to handle these large
addresses was to increase the arena size up to 60 bits addresses,
cf CL 138736.
However, with the new page allocator, the 60bit heap addresses are
causing huge memory allocations, especially by (s *pageAlloc).init. mmap
and munmap syscalls dealing with these allocations are reducing
performances of every Go programs.
In order to avoid these allocations, arenaBaseOffset is set to
0x0a00000000000000 and heap addresses are on 48bit, as others operating
systems.
Updates: #35451
Change-Id: Ice916b8578f76703428ec12a82024147a7592bc0
Reviewed-on: https://go-review.googlesource.com/c/go/+/206841
Run-TryBot: Michael Knyszek <mknyszek@google.com> Reviewed-by: Michael Knyszek <mknyszek@google.com>
Bryan C. Mills [Fri, 15 Nov 2019 21:19:11 +0000 (16:19 -0500)]
cmd/dist: skip GOCACHE and .git when making GOROOT read-only
When we run tests, we may need to write the test binary (and/or test
variants of its dependencies) to GOCACHE. (This also fixes several
test cases in cmd/go, which preserves the GOCACHE variable for
efficiency.)
It is highly unlikely that tests will try to modify .git, and that
directory contains many files, so don't bother with it.
Elias Naur [Sun, 10 Nov 2019 07:42:54 +0000 (08:42 +0100)]
cmd/link/internal/ld: set MachO platform to PLATFORM_IOS on iOS
CL 205340 changed the linker to skip certain flags when linking
for iOS. However, for host linking on iOS (such as on the Corellium
darwin/arm64 builders) the MachO platform defaults to PLATFORM_MACOS,
and the flags are not skipped.
Avoids warnings such as
ld: warning: -no_pie ignored for arm64
Updates #32963
Change-Id: Ib6b4c2375fd14cf89410bf5ff1537b692b7a1c15
Reviewed-on: https://go-review.googlesource.com/c/go/+/206337
Run-TryBot: Elias Naur <mail@eliasnaur.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Bryan C. Mills [Fri, 15 Nov 2019 17:26:09 +0000 (12:26 -0500)]
cmd/dist: fix GOROOT permissions on failure
While running various tests for #28387, I keep ending up with an
unwritable GOROOT after a failure.
While the unwritable GOROOT is a fairly exotic condition (normally
only happens on builders), it's somewhat annoying when debugging, so
I'm switching all of the log.Fatal* call sites to use the existing
fatalf function, which supports general atexit-like cleanup.
Michael Anthony Knyszek [Wed, 13 Nov 2019 16:56:50 +0000 (16:56 +0000)]
runtime: check summary before scavenging in fast path
In scavengeOne's fast path, we currently don't check the summary for the
chunk that scavAddr points to, which means that we might accidentally
scavenge unused address space if the previous scavenge moves the
scavAddr into that space. The result of this today is a crash.
This change makes it so that scavengeOne's fast path only happens after
the check, following the comment in mpagealloc.go. It also adds a test
for this case.
go101 [Fri, 15 Nov 2019 00:38:04 +0000 (00:38 +0000)]
reflect: factor out special channel assignability rule from haveIdenticalUnderlyingType
Go specification says: A value x is assignable to a variable of type T if x
is a bidirectional channel value, T is a channel type, x's type V and T have
identical element types, and at least one of V or T is not a defined type.
However, the current reflection implementation is incorrect which makes
"x is assignable to T" even if type V and T are both defined type.
The current reflection implementation also mistakes the base types of two
non-defined pointer types share the same underlying type when the two
base types satisfy the above mentioned special channel assignability rule.
Fixes #29469
Change-Id: Ia4b9c4ac47dc8e76a11faef422b2e5c5726b78b3
GitHub-Last-Rev: 487c20a564091a1d2ba5feb95ab5196331c699c2
GitHub-Pull-Request: golang/go#29739
Reviewed-on: https://go-review.googlesource.com/c/go/+/157822
Run-TryBot: Ian Lance Taylor <iant@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Cherry Zhang [Wed, 13 Nov 2019 20:05:17 +0000 (15:05 -0500)]
cmd/internal/obj/x86: mark 2-instruction TLS access nonpreemptible
The 2-instruction TLS access sequence
MOVQ TLS, BX
MOVQ 0(BX)(TLS*1), BX
is not async preemptible, as if it is preempted and resumed on a
different thread, the TLS address may become invalid.
May fix #35349. (This is a rare failure and I haven't been able
to reproduce it.)
Cherry Zhang [Thu, 14 Nov 2019 02:01:01 +0000 (21:01 -0500)]
runtime: crash if a signal is received with bad G and no extra M
When we receive a signal, if G is nil we call badsignal, which
calls needm. When cgo is not used, there is no extra M, so needm
will just hang. In this situation, even GOTRACEBACK=crash cannot
get a stack trace, as we're in the signal handler and cannot
receive another signal (SIGQUIT).
Instead, just crash.
For #35554.
Updates #34391.
Change-Id: I061ac43fc0ac480435c050083096d126b149d21f
Reviewed-on: https://go-review.googlesource.com/c/go/+/206959
Run-TryBot: Cherry Zhang <cherryyz@google.com> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Sergey Zagursky [Thu, 14 Nov 2019 20:07:01 +0000 (23:07 +0300)]
net: fix improper Context.Deadline usage in DialContext
The existing implementation is erroneously assume that having no
deadline in context.Context means that time returned from Deadline
method will have IsZero() == true. But technically speaking this is an
invalid assumption. The context.Context interface specification doesn't
specify what time should be returned from Deadline method when there is
no deadline set. It only specifies that second result of Deadline should
be false.
Fixes #35594
Change-Id: Ife00aad77ab3585e469f15017550ac6c0431b140
Reviewed-on: https://go-review.googlesource.com/c/go/+/207297
Run-TryBot: Ian Lance Taylor <iant@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Jay Conrod [Thu, 14 Nov 2019 21:17:47 +0000 (16:17 -0500)]
cmd/go: include module path and version in cache key with -trimpath
When -trimpath is used, packages built from the module cache still
have debug into that contains the module path and version. Only the
module cache directory is stripped.
With this CL, we now include the module path and version in the cache
key for build actions.
Fixes #35412
Change-Id: I1956592d0d86fcea2cca7c5fc8957e83543d6aa2
Reviewed-on: https://go-review.googlesource.com/c/go/+/207317
Run-TryBot: Jay Conrod <jayconrod@google.com> Reviewed-by: Bryan C. Mills <bcmills@google.com>
PC:300656 and PC:300664 belong to a single location entry,
but the bug in the current tryAdd logic placed the entire stack into one
location entry.
Also adds tests - this crash is a tricky case to test because I think it
should happen with normal go code. The new TestTryAdd simulates it by
using fake call sequences. The test crashed without the fix.
Update #35538
Change-Id: I6d3483f757abf4c429ab91616e4def90832fc04a
Reviewed-on: https://go-review.googlesource.com/c/go/+/206958 Reviewed-by: Keith Randall <khr@golang.org>
Ian Lance Taylor [Thu, 14 Nov 2019 01:33:03 +0000 (17:33 -0800)]
os/signal: don't ignore SIGINT in TestAtomicStop child process
Fixes #35085
Change-Id: Ice611e1223392f687061a43fd4c2298ea22774fb
Reviewed-on: https://go-review.googlesource.com/c/go/+/207081
Run-TryBot: Ian Lance Taylor <iant@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
Yasuhiro Matsumoto [Wed, 13 Nov 2019 01:02:17 +0000 (10:02 +0900)]
os: handle backslash and slash both in the path on Windows
Fixes #35492
Change-Id: I00dce8fd1228f809e0c61013ac4de7a5953cbbf9
Reviewed-on: https://go-review.googlesource.com/c/go/+/206997
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Jay Conrod [Wed, 13 Nov 2019 18:26:13 +0000 (13:26 -0500)]
cmd/go: fix error for empty packages referenced with relative paths
'go build' now reports a more useful error when a relative path on the
command line points to a directory that doesn't exist or a directory
without .go files. Errors are generated by go/build.Context.ImportDir
instead of a vague call to base.Fatalf in modload.
Fixes #35414
Change-Id: I2642230c5e409107b98bb6d6c3a484d8d25b4147
Reviewed-on: https://go-review.googlesource.com/c/go/+/206902
Run-TryBot: Jay Conrod <jayconrod@google.com> Reviewed-by: Bryan C. Mills <bcmills@google.com>
Ian Lance Taylor [Wed, 13 Nov 2019 19:09:27 +0000 (11:09 -0800)]
os/exec: skip poll descriptors when checking for open descriptors
It turns out that there is a path that initializes netpoll and opens
file descriptors before running the os/exec init function: on some
systems, the uses of NewFile when setting os.Stdin and friends can
initialize netpoll which can open file descriptors. This in itself
is not a problem, but when we check whether the new files are open
using os.NewFile, a side-effect is to put them into non-blocking mode.
This can then break future uses of netpoll.
Updates #35469
Fixes #35566
Change-Id: I1b2e2c943695d1c2d29496b050abbce9ee710a00
Reviewed-on: https://go-review.googlesource.com/c/go/+/207078 Reviewed-by: Bryan C. Mills <bcmills@google.com>
Rémy Oudompheng [Wed, 13 Nov 2019 05:53:22 +0000 (06:53 +0100)]
math/big: fix out-of-bounds panic in divRecursive
The bounds in the last carry branch were wrong as there
is no reason for len(u) >= n+n/2 to always hold true.
We also adjust test to avoid using a remainder of 1
(in which case, the last step of the algorithm computes
(qhatv+1) - qhatv which rarely produces a carry).
Ian Lance Taylor [Wed, 13 Nov 2019 01:22:28 +0000 (17:22 -0800)]
runtime: acquire timersLocks around moveTimers
In the discussion of CL 171828 we decided that it was not necessary to
acquire timersLock around the call to moveTimers, because the world is
stopped. However, that is not correct, as sysmon runs even when the world
is stopped, and it calls timeSleepUntil which looks through the timers.
timeSleepUntil acquires timersLock, but that doesn't help if moveTimers
is running at the same time.
Updates #6239
Updates #27707
Updates #35462
Change-Id: I346c5bde594c4aff9955ae430b37c2b6fc71567f
Reviewed-on: https://go-review.googlesource.com/c/go/+/206938
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Michael Knyszek <mknyszek@google.com>
Than McIntosh [Wed, 13 Nov 2019 15:03:19 +0000 (10:03 -0500)]
cmd/link: don't run TestDWARF in c-archive mode on Windows
Test fix: in dwarf_test.go don't try to run the TestDWARF testpoint on
windows with c-archive build mode (linker + debug/pe support for that
build mode on Windows is not fully baked it seems).
Fixes #35512.
Change-Id: I1c87ff3d62e5b98e75062b184d762fb5ed937745
Reviewed-on: https://go-review.googlesource.com/c/go/+/206899
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
Ian Lance Taylor [Wed, 13 Nov 2019 02:17:37 +0000 (18:17 -0800)]
os/exec: don't run TestExtraFiles if extra files were open for the test
Our attempts to close existing open files are flaky. They will fail if,
for example, file descriptor 3 is open when the test binary starts.
Instead, report any such cases, and skip TestExtraFiles.
Bryan C. Mills [Wed, 13 Nov 2019 14:06:25 +0000 (09:06 -0500)]
cmd/dist: remove chatty log.Print
In CL 206757 I added a log.Printf to identify when GOROOT is not read-only.
However, it interacts badly with test sharding in the builders:
the log is repeated for every shard.
Since the log statement isn't particularly high-value, just remove it.