Austin Clements [Thu, 5 Dec 2019 19:41:24 +0000 (14:41 -0500)]
runtime: give useful failure message on mlock failure
Currently, we're ignoring failures to mlock signal stacks in the
workaround for #35777. This means if your mlock limit is low, you'll
instead get random memory corruption, which seems like the wrong
trade-off.
This CL checks for mlock failures and panics with useful guidance.
Ian Lance Taylor [Thu, 5 Dec 2019 22:49:25 +0000 (14:49 -0800)]
sync: deflake TestWaitGroupMisuse3
If one of the helper goroutine panics, the main goroutine call to Wait
may hang forever waiting for something to call Done. Put that call in
a goroutine like the others.
Fixes #35774
Change-Id: I8d2b58d8f473644a49a95338f70111d4e6ed4e12
Reviewed-on: https://go-review.googlesource.com/c/go/+/210218
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
Alex Brainman [Sat, 9 Nov 2019 08:06:24 +0000 (19:06 +1100)]
all: fix most of the remaining windows -d=checkptr violations
This change replaces
buf := [HUGE_CONST]*T)(unsafe.Pointer(p))[:]
with
buf := [HUGE_CONST]*T)(unsafe.Pointer(p))[:n:n]
Pointer p points to n of T elements. New unsafe pointer conversion
logic verifies that both first and last elements point into the same
Go variable.
This change replaces [:] with [:n:n] to please pointer checker.
According to @mdempsky, compiler specially recognizes when you
combine a pointer conversion with a full slice operation in a single
expression and makes an exception.
After this, only one failure in net remains when running:
go test -a -short -gcflags=all=-d=checkptr std cmd
Updates #34972
Change-Id: I2c8731650c856264bc788e4e07fa0530f7c250fa
Reviewed-on: https://go-review.googlesource.com/c/go/+/208617
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-by: Matthew Dempsky <mdempsky@google.com>
Robert Griesemer [Mon, 2 Dec 2019 17:38:03 +0000 (09:38 -0800)]
go/types: print package path in error messages if package name is not unique
Change package qualification to print the full package path for packages
that have non-unique names (that is, where multiple different packages
have the same name). Use the package name as qualifier in all other cases
(but don't print any qualification if we're talking about the package
being type-checked).
This matches the behavior of the compiler.
Fixes #35895.
Change-Id: I33ab8e7adfae1378907c01e33cabda114f65887f
Reviewed-on: https://go-review.googlesource.com/c/go/+/209578
Run-TryBot: Robert Griesemer <gri@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Cherry Zhang [Tue, 3 Dec 2019 03:56:40 +0000 (22:56 -0500)]
cmd/compile: mark empty block preemptible
Currently, a block's control instruction gets the liveness info
of the last Value in the block. However, for an empty block, the
control instruction gets the invalid liveness info and therefore
not preemptible. One example is empty infinite loop, which has
only a control instruction. The control instruction being non-
preemptible makes the whole loop non-preemptible.
Fix this by using a different, preemptible liveness info for
empty block's control. We can choose an arbitrary preemptible
liveness info, as at run time we don't really use the liveness
map at that instruction.
As before, if the last Value in the block is non-preemptible, so
is the block control. For example, the conditional branch in the
write barrier test block is still non-preemptible.
Also, only update liveness info if we are actually emitting
instructions. So zero-width Values' liveness info (which are
always invalid) won't affect the block control's liveness info.
For example, if the last Values in a block is a tuple-generating
operation and a Select, the block control instruction is still
preemptible.
Cherry Zhang [Thu, 5 Dec 2019 23:56:54 +0000 (18:56 -0500)]
cmd/compile: don't fuse branches with side effects
Count Values with side effects but no use as live, and don't fuse
branches that contain such Values. (This can happen e.g. when it
is followed by an infinite loop.) Otherwise this may lead to
miscompilation (side effect fired at wrong condition) or ICE (two
stores live simultaneously).
Fixes #36005.
Change-Id: If202eae4b37cb7f0311d6ca120ffa46609925157
Reviewed-on: https://go-review.googlesource.com/c/go/+/210179 Reviewed-by: Keith Randall <khr@golang.org>
Rhys Hiltner [Thu, 5 Dec 2019 03:37:00 +0000 (19:37 -0800)]
cmd/go: print newline after GOOS/GOARCH error
The newline was dropped during the refactor in CL 194617.
Fixes #35984
Change-Id: I7e0d7aa2d7a4d1f44898921f8bb40401620d78b2
Reviewed-on: https://go-review.googlesource.com/c/go/+/209965
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Jay Conrod [Thu, 5 Dec 2019 18:28:57 +0000 (13:28 -0500)]
cmd/go: include imports in 'go list -e' output even after parse errors
If we aren't able to load imports from one file in a package due to a
parse error (scanner.ErrorList), 'go list -e' should still list
imports in other files.
Fixes #35973
Change-Id: I59f171877949bb7afaf252b6c8a970de22e60c7a
Reviewed-on: https://go-review.googlesource.com/c/go/+/210097
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
Keith Randall [Wed, 4 Dec 2019 22:58:04 +0000 (14:58 -0800)]
os: reset dirinfo when seeking on Darwin
The first Readdirnames calls opendir and caches the result.
The behavior of that cached opendir result isn't specified on a seek
of the underlying fd. Free the opendir result on a seek so that
we'll allocate a new one the next time around.
Also fix wasm behavior in this regard, so that a seek to the
file start resets the Readdirnames position, regardless of platform.
Dmitri Shuralyov [Thu, 5 Dec 2019 14:13:21 +0000 (17:13 +0300)]
doc: add CherryPickApproved filter to Release History links
Not all closed issues in a given minor milestone are included in that
release, only the ones that have been labeled as CherryPickApproved are.
Update the links to the GitHub issue tracker to include a filter on the
CherryPickApproved label, so that the default view shows only the
backports that were included in a given release. This should more useful
to most people than seeing all backports (considered and approved).
Do this only for Go 1.9.1 and newer releases, as that is when we started
using the CherryPickCandidate and CherryPickApproved labels.
Fixes #35988
Change-Id: I51e07c1bc3ab9c4a5744e8f668c5470adf78bffe
Reviewed-on: https://go-review.googlesource.com/c/go/+/209918 Reviewed-by: Alexander Rakoczy <alex@golang.org>
taisa [Wed, 4 Dec 2019 08:31:59 +0000 (17:31 +0900)]
testing: fix testing docs
The Perm function return 0 or 1 or 2 or 3. 4 is not returned,
so that changed the argument to 5.
Change-Id: Ic980c71a9f29f522bdeef4fce70a6c2dd136d791
Reviewed-on: https://go-review.googlesource.com/c/go/+/209777 Reviewed-by: Ian Lance Taylor <iant@golang.org>
Ian Lance Taylor [Tue, 26 Nov 2019 06:17:29 +0000 (22:17 -0800)]
cmd/link: when changing to Segrelrodata, reset datsize
Otherwise we leave a gap at the start of Segrelrodata equal to the
size of the read-only non-relro data, which causes -buildmode=pie
executables to be noticeably larger than -buildmode=exe executables.
Change-Id: I98956ef29d5b7a57ad8e633c823ac09d9ca36a45
Reviewed-on: https://go-review.googlesource.com/c/go/+/208897
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com>
Austin Clements [Mon, 2 Dec 2019 22:36:25 +0000 (17:36 -0500)]
runtime: mlock top of signal stack on Linux 5.2–5.4.1
Linux 5.2 introduced a bug that can corrupt vector registers on return
from a signal if the signal stack isn't faulted in:
https://bugzilla.kernel.org/show_bug.cgi?id=205663
This CL works around this by mlocking the top page of all Go signal
stacks on the affected kernels.
Fixes #35326, #35777
Change-Id: I77c80a2baa4780827633f92f464486caa222295d
Reviewed-on: https://go-review.googlesource.com/c/go/+/209899
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com> Reviewed-by: Ian Lance Taylor <iant@golang.org> Reviewed-by: David Chase <drchase@google.com>
Tobias Klauser [Wed, 4 Dec 2019 10:44:06 +0000 (11:44 +0100)]
cmd/objdump: reference tracking bug in TestDisasmCode skip message
Issue #12559 was closed and split into #19158 for mips{,le} and #19156
for mips64{,le}. Instead of referencing the individual GOARCH-specific
issues in the skip test messages of TestDisasmCode use the tracking bug
Change-Id: I6929d25f4ec5aef4f069b7692c4e29106088ce65
Reviewed-on: https://go-review.googlesource.com/c/go/+/209817
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Michael Anthony Knyszek [Thu, 14 Nov 2019 23:58:50 +0000 (23:58 +0000)]
runtime: convert page allocator bitmap to sparse array
Currently the page allocator bitmap is implemented as a single giant
memory mapping which is reserved at init time and committed as needed.
This causes problems on systems that don't handle large uncommitted
mappings well, or institute low virtual address space defaults as a
memory limiting mechanism.
This change modifies the implementation of the page allocator bitmap
away from a directly-mapped set of bytes to a sparse array in same vein
as mheap.arenas. This will hurt performance a little but the biggest
gains are from the lockless allocation possible with the page allocator,
so the impact of this extra layer of indirection should be minimal.
In fact, this is exactly what we see:
https://perf.golang.org/search?q=upload:20191125.5
This reduces the amount of mapped (PROT_NONE) memory needed on systems
with 48-bit address spaces to ~600 MiB down from almost 9 GiB. The bulk
of this remaining memory is used by the summaries.
Go processes with 32-bit address spaces now always commit to 128 KiB of
memory for the bitmap. Previously it would only commit the pages in the
bitmap which represented the range of addresses (lowest address to
highest address, even if there are unused regions in that range) used by
the heap.
Xiangdong Ji [Fri, 22 Nov 2019 17:02:06 +0000 (17:02 +0000)]
cmd/vet: honor analyzer flags when running vet outside $GOROOT/src
Additional vet flags specified by user are discarded if 'go vet'
is invoked outside $GOROOT/src to check a package under $GOROOT
(including those under "vendor" of $GOROOT), fix it by avoiding the
overwriting, the logic of detemining if the package under vetting
comes from $GOROOT remains untouched.
Also checked 'go tool vet <options> <cfg>' and 'go vet <options>
<user pkg>', both worked w./w.o this fix.
Fixes #35837.
Change-Id: I549af7964e40440afd35f2d1971f77eee6f8de34
Reviewed-on: https://go-review.googlesource.com/c/go/+/209498
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Jay Conrod <jayconrod@google.com>
Ian Lance Taylor [Mon, 2 Dec 2019 20:07:22 +0000 (12:07 -0800)]
runtime: use current P's race context in timer code
We were using the race context of the P that held the timer,
but since we unlock the P's timers while executing a timer
that could lead to a race on the race context itself.
Updates #6239
Updates #27707
Fixes #35906
Change-Id: I5f9d5f52d8e28dffb88c3327301071b16ed1a913
Reviewed-on: https://go-review.googlesource.com/c/go/+/209580
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Keith Randall <khr@golang.org> Reviewed-by: Michael Knyszek <mknyszek@google.com>
Than McIntosh [Mon, 25 Nov 2019 19:07:59 +0000 (14:07 -0500)]
cmd/link: additional fixes for -newobj and "ld -r" ELF host objects
The previous fix for this issue (CL 208479) was not general enough;
this patch revises it to handle more cases.
The problem with the original fix was that once a sym.Symbol is
created for a given static symbol and given a bogus anonymous version
of -1, we hit problems if some other non-anonymous symbol (created by
host object loading) had relocations targeting the static symbol.
In this patch instead of assigning a fixed anonymous version of -1 to
such symbols, each time loader.Create is invoked we create a new
(unique) anonymous version for the sym.Symbol, then enter the result
into the loader's extStaticSyms map, permitting it to be found in
lookups when processing relocation targets.
NB: this code will hopefully get a lot simpler once we can move host
object loading away from early sym.Symbol creation.
Richard Miller [Tue, 19 Nov 2019 13:15:28 +0000 (13:15 +0000)]
runtime: on plan9 don't return substitute address for sysReserve
Plan 9 doesn't have a way to reserve virtual memory, so the
implementation of sysReserve allocates memory space (which won't
be backed with real pages until the virtual pages are referenced).
If the space is then freed with sysFree, it's not returned to
the OS (because Plan 9 doesn't allow shrinking a shared address
space), but it must be cleared to zeroes in case it's reallocated
subsequently.
This interacts badly with the way mallocinit on 64-bit machines
sets up the heap, calling sysReserve repeatedly for a very large
(64MB?) arena with a non-nil address hint, and then freeing the space
again because it doesn't have the expected alignment. The
repeated clearing of multiple megabytes adds significant startup
time to every go program.
We correct this by restricting sysReserve to allocate memory only
when the caller doesn't provide an address hint. If a hint is
provided, sysReserve will now return nil instead of allocating memory
at a different address.
Fixes #27744
Change-Id: Iae5a950adefe4274c4bc64dd9c740d19afe4ed1c
Reviewed-on: https://go-review.googlesource.com/c/go/+/207917
Run-TryBot: David du Colombier <0intro@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: David du Colombier <0intro@gmail.com>
Michael Anthony Knyszek [Fri, 22 Nov 2019 21:06:15 +0000 (21:06 +0000)]
runtime: ready scavenger without next
This change makes it so that waking up the scavenger readies its
goroutine without "next" set, so that it doesn't interfere with the
application's use of the runnext feature in the scheduler which helps
fairness.
As of CL 201763 the scavenger began waking up much more often, and in
TestPingPongHog this meant that it would sometimes supercede either a
hog or light goroutine in runnext, leading to a skew in the results and
ultimately a test flake.
This change thus re-enables the TestPingPongHog test on the builders.
Michael Anthony Knyszek [Tue, 26 Nov 2019 21:16:43 +0000 (21:16 +0000)]
runtime: reset scavenge address in scavengeAll
Currently scavengeAll (which is called by debug.FreeOSMemory) doesn't
reset the scavenge address before scavenging, meaning it could miss
large portions of the heap. Fix this by reseting the address before
scavenging, which will ensure it is able to walk over the entire heap.
Brad Fitzpatrick [Tue, 26 Nov 2019 23:57:35 +0000 (23:57 +0000)]
net/http: update bundled x/net/http2
Updates bundled http2 to x/net git rev ef20fe5d7 for:
http2: make Transport.IdleConnTimeout consider wall (not monotonic) time
https://golang.org/cl/208798 (#29308)
http2: make CipherSuites validation error more verbose
https://golang.org/cl/200317 (#34776)
http2: track unread bytes when the pipe is broken
https://golang.org/cl/187377 (#28634)
http2: split cookie pair into separate hpack header fields
https://golang.org/cl/155657 (#29386)
Fixes #29308
Fixes #28634
Change-Id: I71a03ca62ccb5ff35a5cfadd8dc705a4491ae7ea
Reviewed-on: https://go-review.googlesource.com/c/go/+/209077 Reviewed-by: Ian Lance Taylor <iant@golang.org>
Cherry Zhang [Fri, 15 Nov 2019 02:34:35 +0000 (21:34 -0500)]
runtime: print more information on stack overflow
Print the current SP and (old) stack bounds when the stack grows
too large. This helps to identify the problem: whether a large
stack is used, or something else goes wrong.
Cherry Zhang [Fri, 15 Nov 2019 01:23:17 +0000 (20:23 -0500)]
cmd/internal/obj: mark split-stack prologue nonpreemptible
When there are both a synchronous preemption request (by
clobbering the stack guard) and an asynchronous one (by signal),
the running goroutine may observe the synchronous request first
in stack bounds check, and go to the path of calling morestack.
If the preemption signal arrives at this point before the call to
morestack, the goroutine will be asynchronously preempted,
entering the scheduler. When it is resumed, the scheduler clears
the preemption request, unclobbers the stack guard. But the
resumed goroutine will still call morestack, as it is already on
its way. morestack will, as there is no preemption request,
double the stack unnecessarily. If this happens multiple times,
the stack may grow too big, although only a small amount is
actually used.
To fix this, we mark the stack bounds check and the call to
morestack async-nonpreemptible, starting after the memory
instruction (mostly a load, on x86 CMP with memory).
Not done for Wasm as it does not support async preemption.
Fixes #35470.
Change-Id: Ibd7f3d935a3649b80f47539116ec9b9556680cf2
Reviewed-on: https://go-review.googlesource.com/c/go/+/207350 Reviewed-by: David Chase <drchase@google.com>
Cherry Zhang [Thu, 14 Nov 2019 22:03:44 +0000 (17:03 -0500)]
cmd/internal/obj, runtime: use register map to mark unsafe points
Currently we use stack map index -2 to mark unsafe points, i.e.
PC ranges that is not safe for async preemption. This has a
problem: it cannot mark CALL instructions, because for stack scan
a valid stack map index is needed.
This CL switches to use register map index for marking unsafe
points instead, which does not conflict with stack scan and can
be applied on CALL instructions. This is necessary as next CL
will mark call to morestack nonpreemptible.
For #35470.
Change-Id: I357bf26c996e1fee1e7eebe4e6bb07d62930d3f7
Reviewed-on: https://go-review.googlesource.com/c/go/+/207349 Reviewed-by: David Chase <drchase@google.com>
Cherry Zhang [Tue, 26 Nov 2019 04:29:02 +0000 (23:29 -0500)]
runtime: disable async preemption on darwin/arm(64) if no cgo
On darwin, we use libc calls, and cgo is required on ARM and
ARM64 so we have TLS set up to save/restore G during C calls. If
cgo is absent, we cannot save/restore G in TLS, and if a signal
is received during C execution we cannot get the G. Therefore
don't send signals (and hope that we won't receive any signal
during C execution).
This can only happen in the go_bootstrap program (otherwise cgo
is required).
Fixes #35800.
Change-Id: I6c02a9378af02c19d32749a42db45165b578188d
Reviewed-on: https://go-review.googlesource.com/c/go/+/208818
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Bryan C. Mills [Fri, 22 Nov 2019 21:06:11 +0000 (16:06 -0500)]
misc: remove use of relative directories in overlayDir functions
It turns out that the relative-path support never worked in the first
place.
It had been masked by the fact that we ~never invoke overlayDir with
an absolute path, which caused filepath.Rel to always return an error,
and overlayDir to always fall back to absolute paths.
Since the absolute paths seem to be working fine (and are simpler),
let's stick with those. As far as I can recall, the relative paths
were only a space optimization anyway.
Updates #28387
Updates #30316
Change-Id: Ie8cd28f3c41ca6497ace2799f4193d7f5dde7a37
Reviewed-on: https://go-review.googlesource.com/c/go/+/208481
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Jay Conrod <jayconrod@google.com>
Dan Scales [Tue, 19 Nov 2019 20:41:51 +0000 (12:41 -0800)]
runtime: add go:nosplit to cgo_mmap.go:mmap() and sys_darwin.go:mmap()
cgo_mmap.go:mmap() is called by mem_linux.go:sysAlloc(), a low-level memory
allocation function. mmap() should be nosplit, since it is called in a lot of
low-level parts of the runtime and callers often assume it won't acquire any
locks.
As an example there is a potential deadlock involving two threads if mmap is not nosplit:
trace.bufLock acquired, then stackpool[order].item.mu, then mheap_.lock
- can happen for traceEvents that are not invoked on the system stack and cause
a traceFlush, which causes a sysAlloc, which calls mmap(), which may cause a
stack split. mheap_.lock
mheap_.lock acquired, then trace.bufLock
- can happen when doing a trace in reclaimChunk (which holds the mheap_ lock)
Also, sysAlloc() has a comment that it is nosplit because it may be invoked
without a valid G, in which case its callee mmap() should also be nosplit.
Similarly, sys_darwin.go:mmap() is called by mem_darwin.go:sysAlloc(), and should
be nosplit for the same reasons.
Extra gomote testing: linux/arm64, darwin/amd64
Change-Id: Ia4d10cec5cf1e186a0fe5aab2858c6e0e5b80fdc
Reviewed-on: https://go-review.googlesource.com/c/go/+/207844 Reviewed-by: Ian Lance Taylor <iant@golang.org>
Than McIntosh [Sat, 23 Nov 2019 14:59:30 +0000 (09:59 -0500)]
cmd/link: disable new testpoint on mips pending investigation
Skip TestMinusRSymsWithSameName testpoint on MIPS for the time being
since it triggers failures on that arch. Will re-enable once the
problems are fixed.
Because of concurrent goroutines it is possible for multiple event
handlers to return at the same time. This was not properly supported
and caused the wrong goroutine to continue, which in turn caused
memory corruption.
This change adds a stack of events so it is always clear which is the
innermost event that needs to return next.
Than McIntosh [Fri, 22 Nov 2019 19:31:13 +0000 (14:31 -0500)]
cmd/link: add new linker testpoint for "ld -r" host object
This adds a new test that builds a small Go program with linked
against a *.syso file that is the result of an "ld -r" link. The
sysobj in question has multiple static symbols in the same section
with the same name, which triggered a bug in the loader in -newobj
mode.
Than McIntosh [Fri, 22 Nov 2019 20:27:01 +0000 (15:27 -0500)]
cmd/link: fix bug with -newobj and "ld -r" ELF host objects
When the ELF host object loader encounters a static/hidden symbol, it
creates a sym.Symbol for it but does not enter it into the sym.Symbols
lookup table. Under -newobj mode, this was not happening correctly; we
were adding the sym via loader.LookupOrCreate, which resulted in
collisions when it encountered symbols with the same name + version +
section (this can happen for "ld -r" objects).
Dmitri Shuralyov [Sat, 16 Nov 2019 23:19:33 +0000 (23:19 +0000)]
net/http: rename tests for Redirect and StripPrefix
Before May 2018, I mistakenly thought the _suffix naming convention¹
used by examples also applied to tests. Thanks to a code review comment²
from Ian Lance Taylor, I have since learned that is not true.
This trivial change fixes some collateral damage from my earlier
misunderstanding, resulting in improved test naming consistency.
Brad Fitzpatrick [Fri, 1 Nov 2019 18:23:05 +0000 (11:23 -0700)]
net/http: make Transport.IdleConnTimeout consider wall (not monotonic) time
Both laptops closing their lids and cloud container runtimes
suspending VMs both faced the problem where an idle HTTP connection
used by the Transport could be cached for later reuse before the
machine is frozen, only to wake up many minutes later to think that
their HTTP connection was still good (because only a second or two of
monotonic time passed), only to find out that the peer hung up on them
when they went to write.
HTTP/1 connection reuse is inherently racy like this, but no need for
us to step into a trap if we can avoid it. Also, not everybody sets
Request.GetBody to enable re-tryable POSTs. And we can only safely
retry requests in some cases.
So with this CL, before reusing an old connection, double check the walltime.
Testing was done both with a laptop (closing the lid for a bit) and
with QEMU, running "stop" and "cont" commands in the monitor and
sending QMP guest agent commands to update its wall clock after the
"cont":
Jay Conrod [Thu, 21 Nov 2019 21:55:11 +0000 (16:55 -0500)]
cmd/go: fix and re-enable build_trimpath test
The test was comparing a binary built from a list of files to a test
build from a named package. That should not (and did not) work. The
test now compares two binaries built the same way in different
directories.
Also add a portion of the test for GOPATH and fix the gccgo portion of
the test (verified manually).
Fixes #35435
Change-Id: I2535a0011c9d97d2274e5550ae277302dbb91e6f
Reviewed-on: https://go-review.googlesource.com/c/go/+/208234
Run-TryBot: Jay Conrod <jayconrod@google.com> Reviewed-by: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Bryan C. Mills [Thu, 21 Nov 2019 21:26:09 +0000 (16:26 -0500)]
cmd/go: do not panic when computing Shlib for a package with no Target
In module mode, a non-main package lacks an install target.
The location of the .shlib corresponding to a given target is stored
in a .shlibname file alongside its install target, so in module mode
a non-main package also lacks a .shlibname file.
This also implies that such a package cannot be installed with
'go install -buildmode=linkshared', but that is a problem
for another day.
Fixes #35759
Updates #34347
Change-Id: Id3e0e068266d5fb9b061a59e70f9a65985d4973b
Reviewed-on: https://go-review.googlesource.com/c/go/+/208233
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Jay Conrod <jayconrod@google.com>
Bryan C. Mills [Fri, 22 Nov 2019 18:26:58 +0000 (13:26 -0500)]
cmd/go: add a 'buildmode' condition for script tests
In CL 208233 I am fixing a panic that occurs only with a specific
build mode. I want that test to run on all platforms that support that
build mode, but the logic for determining support is somewhat
involved.
For now, I am duplicating that logic into the cmd/internal/sys
package, which already reports platform support for other build flags.
We can refactor cmd/go/internal/work to use the extracted function in
a followup CL.
Updates #35759
Change-Id: Ibbaedde4d1e8f683c650beedd10849bc27e7a6e7
Reviewed-on: https://go-review.googlesource.com/c/go/+/208457
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Jay Conrod <jayconrod@google.com>
Bryan C. Mills [Thu, 21 Nov 2019 20:03:24 +0000 (15:03 -0500)]
misc/cgo/testshared: make -v output less verbose
Previously, 'go test -v' in this directory would result in a massive
dump of go command output, because the test plumbed -v to 'build -x'.
This change separates them into distinct flags, so that '-v' only
implies the display of default 'go' command output.
Updates #30316
Change-Id: Ifb125f35ec6a0bebe7e8286e7c546d132fb213df
Reviewed-on: https://go-review.googlesource.com/c/go/+/208232
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Michael Anthony Knyszek [Fri, 22 Nov 2019 16:34:16 +0000 (16:34 +0000)]
runtime: release worldsema before Gosched in STW GC mode
After CL 182657 we no longer hold worldsema across the GC, we hold
gcsema instead.
However in STW GC mode we don't release worldsema before calling Gosched
on the user goroutine (note that user goroutines are disabled during STW
GC) so that user goroutine holds onto it. When the GC is done and the
runtime inevitably wants to "stop the world" again (though there isn't
much to stop) it'll sit there waiting for worldsema which won't be
released until the aforementioned goroutine is scheduled, which it won't
be until the GC is done!
So, we have a deadlock.
The fix is easy: just release worldsema before calling Gosched.
TestPhysicalMemoryUtilization occasionally fails on some platforms by
only a small margin. The reason for this is that it assumes the
scavenger will always be able to scavenge all the memory that's released
by sweeping, but because of the page cache, there could be free and
unscavenged memory held onto by a P which the scavenger simply cannot
get to.
As a result, if the page cache gets filled completely (512 KiB of free
and unscavenged memory) this could skew a test which expects to
scavenge roughly 8 MiB of memory. More specifically, this is 512 KiB of
memory per P, and if a system is more inclined to bounce around
between Ps (even if there's only one goroutine), this memory can get
"stuck".
Through some experimentation, I found that failures correlated highly
with relatively large amounts of memory ending up in some page cache
(like 60 or 64 pages) on at least one P.
This change changes the test's threshold such that it accounts for the
page cache, and scales up with GOMAXPROCS. Because the test constants
themselves don't change, however, the test must now also bound
GOMAXPROCS such that the threshold doesn't get too high (at which point
the test becomes meaningless).
Fixes #35580.
Change-Id: I6bdb70706de991966a9d28347da830be4a19d3a1
Reviewed-on: https://go-review.googlesource.com/c/go/+/208377
Run-TryBot: Michael Knyszek <mknyszek@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Keith Randall <khr@golang.org>
Jay Conrod [Thu, 21 Nov 2019 23:54:38 +0000 (18:54 -0500)]
cmd/go: add 'go generate' commands to modfile_flag test
Verify that 'go generate' works with -modfile. Also check that
go commands starts with 'go generate' do not inherit -modfile, but
they should still work if -modfile is set in GOFLAGS.
Updates #34506
Change-Id: I5e1f897b4e38e4fdaccc0fbb7a71b8d0e9fc0660
Reviewed-on: https://go-review.googlesource.com/c/go/+/208236
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
Bryan C. Mills [Wed, 20 Nov 2019 19:39:19 +0000 (14:39 -0500)]
misc/cgo/testcshared: avoid writing to GOROOT in tests
The tests in this package invoked 'go install -i -buildmode=c-shared'
in order to generate an archive as well as multiple C header files.
Unfortunately, the behavior of the '-i' flag is inappropriately broad
for this use-case: it not only generates the library and header files
(as desired), but also attempts to install a number of (unnecessary)
archive files for transitive dependencies to
GOROOT/pkg/$GOOS_$GOARCH_testcshared_shared, which may not be writable
— for example, if GOROOT is owned by the root user but the test is
being run by a non-root user.
Instead, for now we generate the header files for transitive dependencies
separately by running 'go tool cgo -exportheader'.
In the future, we should consider how to improve the ergonomics for
generating transitive header files without coupling that to
unnecessary library installation.
Updates #28387
Updates #30316
Updates #35715
Change-Id: I622426a860828020d98f7040636f374e5c766d28
Reviewed-on: https://go-review.googlesource.com/c/go/+/208119
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Bryan C. Mills [Wed, 20 Nov 2019 15:19:43 +0000 (10:19 -0500)]
misc/cgo/testcarchive: avoid writing to GOROOT in tests
Also add a -testwork flag to facilitate debugging the test itself.
Three of the tests of this package invoked 'go install -i
-buildmode=c-archive' in order to generate an archive as well as
multiple C header files.
Unfortunately, the behavior of the '-i' flag is inappropriately broad
for this use-case: it not only generates the library and header files
(as desired), but also attempts to install a number of (unnecessary)
archive files for transitive dependencies to
GOROOT/pkg/$GOOS_$GOARCH_shared, which may not be writable — for
example, if GOROOT is owned by the root user but the test is being run
by a non-root user.
Instead, for now we generate the header files for transitive dependencies
separately by running 'go tool cgo -exportheader'.
In the future, we should consider how to improve the ergonomics for
generating transitive header files without coupling that to
unnecessary library installation.
Updates #28387
Updates #30316
Updates #35715
Change-Id: I3d483f84e22058561efe740aa4885fc3f26137b5
Reviewed-on: https://go-review.googlesource.com/c/go/+/208117
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Hana Kim [Mon, 18 Nov 2019 17:44:33 +0000 (12:44 -0500)]
runtime/pprof: avoid crash due to truncated stack traces
The profile proto message builder maintains a location entry cache
that maps a location (possibly involving multiple user frames
that represent inlined function calls) to the location id. We have
been using the first pc of the inlined call sequence as the key of
the cached location entry assuming that, for a given pc, the sequence
of frames representing the inlined call stack is deterministic and
stable. Then, when analyzing the new stack trace, we expected the
exact number of pcs to be present in the captured stack trace upon
the cache hit.
This assumption does not hold, however, in the presence of the stack
trace truncation in the runtime during profiling, and also with the
potential bugs in runtime.
A better fix is to use all the pcs of the inlined call sequece as
the key instead of the first pc. But that is a bigger code change.
This CL avoids the crash assuming the trace was truncated.
Fixes #35538
Change-Id: I8c6bae98bc8b178ee51523c7316f56b1cce6df16
Reviewed-on: https://go-review.googlesource.com/c/go/+/207609
Run-TryBot: Hyang-Ah Hana Kim <hyangah@gmail.com> Reviewed-by: Keith Randall <khr@golang.org>
David Chase [Tue, 19 Nov 2019 18:23:35 +0000 (13:23 -0500)]
cmd/compile: try harder to not use an empty src.XPos for a bogus line
The fix for #35652 did not guarantee that it was using a non-empty
src position to replace an empty one. The new code checks again
and falls back to a more certain position. (The input in question
compiles to a single empty infinite loop, and none of the actual instructions
had any source position at all. That is a bug, but given the pathology
of this input, not one worth dealing with this late in the release cycle,
if ever.)
TODO: Add runtime.InfiniteLoop(), replace infinite loops with a call to
that, and use an eco-friendly runtime.gopark instead. (This was Cherry's
excellent idea.)
Roberto Clapis [Mon, 18 Nov 2019 09:05:07 +0000 (10:05 +0100)]
text/template: harden JSEscape to also escape ampersand and equal
Ampersand and equal are not dangerous in a JS/JSString context
but they might cause issues if interpolated in HTML attributes.
This change makes it harder to introduce XSS by misusing
escaping.
Thanks to t1ddl3r <t1ddl3r@gmail.com> for reporting this common
misuse scenario.
Fixes #35665
Change-Id: Ice6416477bba4cb2ba2fe2cfdc20e027957255c0
Reviewed-on: https://go-review.googlesource.com/c/go/+/207637 Reviewed-by: Filippo Valsorda <filippo@golang.org> Reviewed-by: Mike Samuel <mikesamuel@gmail.com> Reviewed-by: Andrew Bonventre <andybons@golang.org> Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Jay Conrod [Thu, 21 Nov 2019 17:50:14 +0000 (12:50 -0500)]
cmd/go: report an error for 'go list -m ...' outside a module
Previously, we just reported an error for "all". Now we report an
error for any pattern that matches modules in the build list. The
build list can only contain the module "command-line-arguments", so
these patterns are not meaningful.
Fixes #35728
Change-Id: Ibc736491ec9164588f9657c09d1b9683b33cf1de
Reviewed-on: https://go-review.googlesource.com/c/go/+/208222
Run-TryBot: Jay Conrod <jayconrod@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Cherry Zhang [Tue, 19 Nov 2019 21:04:32 +0000 (16:04 -0500)]
runtime: relax TestAsyncPreempt
In TestAsyncPreempt, the function being tested for preemption,
although still asynchronously preemptible, may have only samll
ranges of PCs that are preemtible. In an unlucky run, it may
take quite a while to have a signal that lands on a preemptible
instruction. The test case is kind of an extreme. Relax it to
make it more preemptible.
In the original version, the first closure has more work to do,
and it is not a leaf function, and the second test case is a
frameless leaf function. In the current version, the first one
is also a frameless leaf function (the atomic is intrinsified).
Add some calls to it. It is still not preemptible without async
preemption.
Andrew [Wed, 20 Nov 2019 17:06:51 +0000 (12:06 -0500)]
all: base64-encode binaries that will cause Apple notarization to fail
Starting with macOS 10.15 (Catalina), Apple now requires all software
distributed outside of the App Store to be notarized. Any binaries we
distribute must abide by a strict set of requirements like code-signing
and having a minimum target SDK of 10.9 (amongst others).
Apple’s notarization service will recursively inspect archives looking to
find notarization candidate binaries. If it finds a binary that does not
meet the requirements or is unable to decompress an archive, it will
reject the entire distribution. From cursory testing, it seems that the
service uses content sniffing to determine file types, so changing
the file extension will not work.
There are some binaries and archives included in our distribution that
are being detected by Apple’s service as potential candidates for
notarization or decompression. As these are files used by tests and some
are intentionally invalid, we don’t intend to ever make them compliant.
As a workaround for this, we base64-encode any binaries or archives that
Apple’s notarization service issues a warning for, as these warnings will
become errors in January 2020.
Bryan C. Mills [Wed, 20 Nov 2019 19:43:04 +0000 (14:43 -0500)]
misc/cgo/testplugin: avoid writing to GOROOT
One of the 'go build' commands executed by this test passed the '-i'
flag, which caused the 'go' command to attempt to install transitive
standard-library dependencies to GOROOT/pkg/$GOOS_$GOARCH_dynlink.
That failed if GOROOT/pkg was not writable (for example, if GOROOT was
owned by the root user, but the user running the test was not root).
As far as I can tell the '-i' flag is not necessary in this test.
Prior to the introduction of the build cache it may have been an
optimization, but now that the build cache is required the '-i' flag
only adds extra work.
Bryan C. Mills [Wed, 20 Nov 2019 22:04:35 +0000 (17:04 -0500)]
misc/cgo/fortran: avoid writing to $PWD
The bash script that drives this test needs to know whether the
fortran compiler works, but it doesn't actually care about the
generated binary. Write that binary to /dev/null.
Eric Rutherford [Tue, 19 Nov 2019 02:35:33 +0000 (20:35 -0600)]
path: minor changes to improve documentation for Join
Reworking the comments in path to call out how leading
empty elements are treated. Also updating filepath.Join
since it shared much of the wording from path.Join.
Updates #35655
Change-Id: I5b15c5d36e9d19831ed39e6bcc7f2fd6c1330033
Reviewed-on: https://go-review.googlesource.com/c/go/+/207797 Reviewed-by: Rob Pike <r@golang.org>
Austin Clements [Fri, 25 Oct 2019 20:17:41 +0000 (16:17 -0400)]
runtime: support preemption on windows/{386,amd64}
This implements preemptM on Windows using SuspendThead and
ResumeThread.
Unlike on POSIX platforms, preemptM on Windows happens synchronously.
This means we need a make a few other tweaks to suspendG:
1. We need to CAS the G back to _Grunning before doing the preemptM,
or there's a good chance we'll just catch the G spinning on its
status in the runtime, which won't be preemptible.
2. We need to rate-limit preemptM attempts. Otherwise, if the first
attempt catches the G at a non-preemptible point, the busy loop in
suspendG may hammer it so hard that it never makes it past that
non-preemptible point.
Austin Clements [Tue, 19 Nov 2019 01:21:38 +0000 (20:21 -0500)]
runtime: ensure thread handle is valid in profileloop1
On Windows, there is currently a race between unminit closing the
thread's handle and profileloop1 suspending the thread using its
handle. If another handle reuses the same handle value, this can lead
to unpredictable results.
To fix this, we protect the thread handle with a lock and duplicate it
under this lock in profileloop1 before using it.
This is going to become a much bigger problem with non-cooperative
preemption (#10958, #24543), which uses the same basic mechanism as
profileloop1.
Clément Chigot [Wed, 20 Nov 2019 12:57:24 +0000 (13:57 +0100)]
runtime: disable GDB tests on AIX with -short
Since the new page allocator, AIX's GDB has trouble running Go programs.
It does work but it can be really slow. Therefore, they are disable when
tests are run with -short.
Updates: #35710
Change-Id: Ibfc4bd2cd9714268f1fe172aaf32a73612e262d6
Reviewed-on: https://go-review.googlesource.com/c/go/+/207919
Run-TryBot: Ian Lance Taylor <iant@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Bryan C. Mills [Tue, 19 Nov 2019 19:29:10 +0000 (14:29 -0500)]
cmd/dist: remove redundant 'go test -race' call
In CL 207962, I removed a seemingly-redundant -i flag. As it turns
out, the -i flag has *two* meanings: “install dependencies”, and “do
not actually run the test”. Without the flag, we omit the former
behavior, but add the latter.
We're about to run specific tests from these binaries on the very next
line, so don't preemptively run all of the tests.