[release-branch.go1.5] AUTHORS: add Oracle as corporate copyright holder
Some commits made by Aram from his personal email address are
actually copyright Oracle:
a77fcb3 net: fix comment in sendFile b0e71f4 net: link with networking libraries when net package is in use 92e959a syscall, net: use sendfile on Solaris db8d5b7 net: try to fix setKeepAlivePeriod on Solaris fe5ef5c runtime, syscall: link Solaris binaries directly instead of using dlopen/dlsym 2b90c3e go/build: enable cgo by default on solaris/amd64 2d18ab7 doc/progs: disable cgo tests that use C.Stdout on Solaris 2230e9d misc/cgo: add various solaris build lines 649c7b6 net: add cgo support for Solaris 24396da os/user: small fixes for Solaris 121489c runtime/cgo: add cgo support for solaris/amd64 83b25d9 cmd/ld: make .rela and .rela.plt sections contiguous c94f1f7 runtime: always load address of libcFunc on Solaris e481aac cmd/6l: use .plt instead of .got on Solaris
See bug for clarification.
Fixes #12452
Change-Id: I0aeb1b46c0c7d09c5c736e383ecf40240d2cf85f
Reviewed-on: https://go-review.googlesource.com/14380 Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-on: https://go-review.googlesource.com/14393 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Austin Clements [Wed, 26 Aug 2015 19:06:43 +0000 (15:06 -0400)]
[release-branch.go1.5] runtime: check that stack barrier unwind is in sync
Currently the stack barrier stub blindly unwinds the next stack
barrier from the G's stack barrier array without checking that it's
the right stack barrier. If through some bug the stack barrier array
position gets out of sync with where we actually are on the stack,
this could return to the wrong PC, which would lead to difficult to
debug crashes. To address this, this commit adds a check to the amd64
stack barrier stub that it's unwinding the correct stack barrier.
Brad Fitzpatrick [Wed, 26 Aug 2015 17:53:59 +0000 (10:53 -0700)]
[release-branch.go1.5] net/http/httputil: permit nil request body in ReverseProxy
Accepting a request with a nil body was never explicitly supported but
happened to work in the past.
This doesn't happen in most cases because usually people pass
a Server's incoming Request to the ReverseProxy's ServeHTTP method,
and incoming server requests are guaranteed to have non-nil bodies.
Still, it's a regression, so fix.
Fixes #12344
Change-Id: Id9a5a47aea3f2875d195b66c9a5f8581c4ca2aed
Reviewed-on: https://go-review.googlesource.com/13935 Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-on: https://go-review.googlesource.com/14245
Ulrich Kunitz [Thu, 20 Aug 2015 16:56:18 +0000 (18:56 +0200)]
[release-branch.go1.5] cmd/compile: fix register allocation for == operator
The issue 12226 has been caused by the allocation of the same register
for the equality check of two byte values. The code in cgen.go freed the
register for the second operand before the allocation of the register
for the first operand.
Fixes #12226
Change-Id: Ie4dc33a488bd48a17f8ae9b497fd63c1ae390555
Reviewed-on: https://go-review.googlesource.com/13771 Reviewed-by: Russ Cox <rsc@golang.org>
Reviewed-on: https://go-review.googlesource.com/14227 Reviewed-by: Ian Lance Taylor <iant@golang.org>
Austin Clements [Wed, 26 Aug 2015 16:16:51 +0000 (12:16 -0400)]
[release-branch.go1.5] runtime: don't install a stack barrier in cgocallback_gofunc's frame
Currently the runtime can install stack barriers in any frame.
However, the frame of cgocallback_gofunc is special: it's the one
function that switches from a regular G stack to the system stack on
return. Hence, the return PC slot in its frame on the G stack is
actually used to save getg().sched.pc (so tracebacks appear to unwind
to the last Go function running on that G), and not as an actual
return PC for cgocallback_gofunc.
Because of this, if we install a stack barrier in cgocallback_gofunc's
return PC slot, when cgocallback_gofunc does return, it will move the
stack barrier stub PC in to getg().sched.pc and switch back to the
system stack. The rest of the runtime doesn't know how to deal with a
stack barrier stub in sched.pc: nothing knows how to match it up with
the G's stack barrier array and, when the runtime removes stack
barriers, it doesn't know to undo the one in sched.pc. Hence, if the C
code later returns back in to Go code, it will attempt to return
through the stack barrier saved in sched.pc, which may no longer have
correct unwinding information.
Fix this by blacklisting cgocallback_gofunc's frame so the runtime
won't install a stack barrier in it's return PC slot.
Austin Clements [Wed, 26 Aug 2015 17:54:26 +0000 (13:54 -0400)]
[release-branch.go1.5] runtime: add GODEBUG for stack barriers at every frame
Currently enabling the debugging mode where stack barriers are
installed at every frame requires recompiling the runtime. However,
this is potentially useful for field debugging and for runtime tests,
so make this mode a GODEBUG.
Austin Clements [Mon, 24 Aug 2015 17:35:49 +0000 (13:35 -0400)]
[release-branch.go1.5] cmd/compile: fix uninitialized memory in compare of interface value
A comparison of the form l == r where l is an interface and r is
concrete performs a type assertion on l to convert it to r's type.
However, the compiler fails to zero the temporary where the result of
the type assertion is written, so if the type is a pointer type and a
stack scan occurs while in the type assertion, it may see an invalid
pointer on the stack.
Fix this by zeroing the temporary. This is equivalent to the fix for
type switches from c4092ac.
Didier Spezia [Tue, 25 Aug 2015 16:25:11 +0000 (16:25 +0000)]
[release-branch.go1.5] cmd/asm: fix potential infinite loop in parser
For ARM machines, the assembler supports list of registers
operands such as [R1,R2].
A list missing a ']' results in the parser issuing many errors
and consuming all the tokens. At EOF (i.e. end of the line),
it still loops.
Normally, a counter is maintained to make sure the parser
stops after 10 errors. However, multiple errors occuring on the
same line are simply ignored. Only the first one is reported.
At most one error per line is accounted.
Missing ']' in a register list therefore results in an
infinite loop.
Fixed the parser by explicitly checking for ']' to interrupt
this loops
In the operand tests, also fixed a wrong entry which I think was
not set on purpose (but still led to a successful result).
Fixes #11764
Change-Id: Ie87773388ee0d21b3a2a4cb941d4d911d0230ba4
Reviewed-on: https://go-review.googlesource.com/13920 Reviewed-by: Rob Pike <r@golang.org>
Reviewed-on: https://go-review.googlesource.com/14225
Shenghou Ma [Tue, 1 Sep 2015 23:58:31 +0000 (19:58 -0400)]
cmd/link/internal/ld: align PE .text section to 32-byte when external linking
Some symbols, for example, masks requires 16-byte alignment, and
they are placed in the text section. Before this change, the text
section is only aligned to 4-byte, and it's making masks unaligned.
Fixes #12415.
Change-Id: I7767778d1b4f7d3e74c2719a02848350782a4160
Reviewed-on: https://go-review.googlesource.com/14166
Run-TryBot: Minux Ma <minux@golang.org> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
(cherry picked from commit 821e124c24c2b2d753be22a04a3b20b7bf579627)
Reviewed-on: https://go-review.googlesource.com/14279
Dave Cheney [Thu, 3 Sep 2015 22:47:40 +0000 (08:47 +1000)]
[release-branch.go1.5] build: Fix bootstrap.bash for official source tarballs
At the moment, bootstrap.bash assumes it is called from a git working
copy. Hence, it fails to complete when running in an unpacked official
source tarball where .git and .gitignore do not exist. This fix adds a
test for existence for .git and a -f switch for the removal of
.gitignore.
Fixes #12223
Change-Id: I7f305b83b38d5115504932bd38dadb7bdeb5d487
Reviewed-on: https://go-review.googlesource.com/13770 Reviewed-by: Dave Cheney <dave@cheney.net> Reviewed-by: Andrew Gerrand <adg@golang.org>
Reviewed-on: https://go-review.googlesource.com/14281
Russ Cox [Wed, 19 Aug 2015 02:50:12 +0000 (22:50 -0400)]
net: respect go vs cgo resolver selection in all lookup routines
This is especially important for LookupAddr, which used to be pure Go
(lightweight, one goroutine per call) and without this CL is now
unconditionally cgo (heavy, one thread per call).
Russ Cox [Wed, 19 Aug 2015 02:19:58 +0000 (22:19 -0400)]
net: force LookupAddr results to be rooted DNS paths when using cgo
Go 1.4 and before have always returned DNS names with a trailing dot
for reverse lookups, as they do for basically all other routines returning
DNS names. Go 1.4 and before always implemented LookupAddr using
pure Go (not C library calls).
Go 1.5 added the ability to make a C library call to implement LookupAddr.
Unfortunately the C library call returns a DNS name without a trailing dot
(an unrooted name), meaning that if turn off cgo during make.bash then
you still get the rooted name but with cgo on you get an unrooted name.
The unrooted name is inconsistent with the pure Go implementation
and with all previous Go releases, so change it to a rooted name.
Fixes #12189.
Change-Id: I3d6b72277c121fe085ea6af30e5fe8019fc490ad
Reviewed-on: https://go-review.googlesource.com/13697 Reviewed-by: Rob Pike <r@golang.org>
Russ Cox [Tue, 18 Aug 2015 15:15:15 +0000 (11:15 -0400)]
doc: adjust binary install page supported system list
Make clear that this list is the list of supported systems
for binary distributions, and that other systems may be
able to build the distribution from source, in addition
to using gccgo.
Drop freebsd/arm from the list on this page.
We have never issued a binary distribution for freebsd/arm,
and we're not going to start in Go 1.5, since we don't even
have a working builder for it.
Drop freebsd/386 from the list on the page,
because we are unable to build binary distributions, per adg.
I think the wording here should probably be revised further,
but not now.
Change-Id: Ib43b6b64f5c438bfb9aa4d3daa43393f1e33b71f
Reviewed-on: https://go-review.googlesource.com/13690 Reviewed-by: Andrew Gerrand <adg@golang.org> Reviewed-by: Rob Pike <r@golang.org>
Russ Cox [Tue, 18 Aug 2015 13:58:24 +0000 (09:58 -0400)]
cmd/vet: power64 is now ppc64
This was missed when we did the rename months ago
because cmd/vet did not live in the main tree.
Now vet's asmdecl checks will apply to ppc64 assembly too.
Change-Id: I687cba89fef702f29dd118de76a7ca1041c414f6
Reviewed-on: https://go-review.googlesource.com/13677 Reviewed-by: Andrew Gerrand <adg@golang.org>
Russ Cox [Tue, 18 Aug 2015 01:38:46 +0000 (21:38 -0400)]
cmd/compile: fix interaction between GOEXPERIMENT=fieldtrack and race detector
Tested by hand.
Only lines of code changing are protected by Fieldtrack_enabled > 0,
which is never true in standard Go distributions.
Fixes #12171.
Change-Id: I963b9997dac10829db8ad4bfc97a7d6bf14b55c6
Reviewed-on: https://go-review.googlesource.com/13676 Reviewed-by: Ian Lance Taylor <iant@golang.org>
Russ Cox [Tue, 18 Aug 2015 01:26:45 +0000 (21:26 -0400)]
cmd/go: fix vendor-related index out of range panic on bad file tree
Fixes #12156.
Change-Id: I2d71163b98bcc770147eb9e78dc551a9d0b5b817
Reviewed-on: https://go-review.googlesource.com/13674 Reviewed-by: Ian Lance Taylor <iant@golang.org>
Also modified cmd/vet to report the copy-of-mutex bug statically
in CL 13646, and fixed two other instances in the code found by vet.
But vet could not have told us about cloneTLSConfig vs cloneTLSClientConfig.
Confirmed that original report is also fixed by this.
Dmitry Vyukov [Wed, 12 Aug 2015 19:26:25 +0000 (21:26 +0200)]
cmd/trace: fix static file reference
Use runtime.GOROOT instead of os.Getenv("GOROOT") to reference
trace-viewer html file. GOROOT env var is not necessary set,
runtime.GOROOT has a default value for such case.
Keith Randall [Thu, 13 Aug 2015 19:25:19 +0000 (12:25 -0700)]
cmd/compile: remove stale register use array
The reg[] array in .../gc is where truth lies. The copy in .../ARCH
is incorrect as it is mostly not updated to reflect regalloc decisions.
This bug was introduced in the rewrite
https://go-review.googlesource.com/#/c/7853/. The new reg[] array was
introduced in .../gc but not all of the uses were removed in the
.../ARCH directories.
Russ Cox [Tue, 11 Aug 2015 14:35:30 +0000 (10:35 -0400)]
cmd/go: run test binaries in original environment
Fixes #12096.
Followup to CL 12483, which fixed #11709 and #11449.
Change-Id: I9031ea36cc60685f4d6f65c39f770c89b3e3395a
Reviewed-on: https://go-review.googlesource.com/13449 Reviewed-by: Ian Lance Taylor <iant@golang.org>
Russ Cox [Fri, 7 Aug 2015 17:34:56 +0000 (13:34 -0400)]
runtime: make sure heapBitsBulkBarrier cannot be preempted
Changes the torture test in #12068 from failing about 1/10 times
to not failing in almost 2,000 runs.
This was only happening in -race mode because functions are
bigger in -race mode, so a few of the helpers for heapBitsBulkBarrier
were not being inlined, and they were not marked nosplit,
so (only in -race mode) the write barrier was being preempted by GC,
causing missed pointer updates.
Filed issue #12069 for diagnosis of any other similar errors.
Russ Cox [Fri, 7 Aug 2015 15:48:52 +0000 (11:48 -0400)]
runtime: run on GOARM=5 and GOARM=6 uniprocessor freebsd/arm systems
Also, crash early on non-Linux SMP ARM systems when GOARM < 7;
without the proper synchronization, SMP cannot work.
Linux is okay because we call kernel-provided routines for
synchronization and barriers, and the kernel takes care of
providing the right routines for the current system.
On non-Linux systems we are left to fend for ourselves.
It is possible to use different synchronization on GOARM=6,
but it's too late to do that in the Go 1.5 cycle.
We don't believe there are any non-Linux SMP GOARM=6 systems anyway.
Austin Clements [Thu, 6 Aug 2015 19:36:50 +0000 (15:36 -0400)]
runtime: call goexit1 instead of goexit
Currently, runtime.Goexit() calls goexit()—the goroutine exit stub—to
terminate the goroutine. This *mostly* works, but can cause a
"leftover stack barriers" panic if the following happens:
1. Goroutine A has a reasonably large stack.
2. The garbage collector scan phase runs and installs stack barriers
in A's stack. The top-most stack barrier happens to fall at address X.
3. Goroutine A unwinds the stack far enough to be a candidate for
stack shrinking, but not past X.
4. Goroutine A calls runtime.Goexit(), which calls goexit(), which
calls goexit1().
5. The garbage collector enters mark termination.
6. Goroutine A is preempted right at the prologue of goexit1() and
performs a stack shrink, which calls gentraceback.
gentraceback stops as soon as it sees goexit on the stack, which is
only two frames up at this point, even though there may really be many
frames above it. More to the point, the stack barrier at X is above
the goexit frame, so gentraceback never sees that stack barrier. At
the end of gentraceback, it checks that it saw all of the stack
barriers and panics because it didn't see the one at X.
The fix is simple: call goexit1, which actually implements the process
of exiting a goroutine, rather than goexit, the exit stub.
To make sure this doesn't happen again in the future, we also add an
argument to the stub prototype of goexit so you really, really have to
want to call it in order to call it. We were able to reliably
reproduce the above sequence with a fair amount of awful code inserted
at the right places in the runtime, but chose to change the goexit
prototype to ensure this wouldn't happen again rather than pollute the
runtime with ugly testing code.
Russ Cox [Thu, 6 Aug 2015 17:44:37 +0000 (13:44 -0400)]
runtime: fix race that dropped GoSysExit events from trace
This makes TestTraceStressStartStop much less flaky.
Running under stress, it changes the failure rate from
above 1/100 to under 1/50000. That very unlikely
failure happens when an unexpected GoSysExit is
written. Not sure how that happens yet, but it is much
less important.
Russ Cox [Thu, 6 Aug 2015 02:12:16 +0000 (22:12 -0400)]
net/url: allow all valid host chars in RawPath
The old code was only allowing the chars we choose not to escape.
We sometimes prefer to escape chars that do not strictly need it.
Allowing those to be used in RawPath lets people override that
preference, which is in fact the whole point of RawPath (new in Go 1.5).
While we are here, also allow [ ] in RawPath.
This is not strictly spec-compliant, but it is what modern browers
do and what at least some people expect, and the [ ] do not cause
any ambiguity (the usual reason they would be escaped, as they are
part of the RFC gen-delims class).
The argument for allowing them now instead of waiting until Go 1.6
is that this way RawPath has one fixed meaning at the time it is
introduced, that we should not need to change or expand.
Fixes #5684.
Change-Id: If9c82a18f522d7ee1d10310a22821ada9286ee5c
Reviewed-on: https://go-review.googlesource.com/13258 Reviewed-by: Andrew Gerrand <adg@golang.org>
Russ Cox [Thu, 6 Aug 2015 01:45:30 +0000 (21:45 -0400)]
net/url: do not percent-encode valid host characters
The code in question was added as part of allowing zone identifiers
in IPv6 literals like http://[ipv6%zone]:port/foo, in golang.org/cl/2431.
The old condition makes no sense. It refers to §3.2.1, which is the wrong section
of the RFC, it excludes all the sub-delims, which §3.2.2 (the right section)
makes clear are valid, and it allows ':', which is not actually valid,
without an explanation as to why (because we keep :port in the Host field
of the URL struct).
The new condition allows all the sub-delims, as specified in RFC 3986,
plus the additional characters [ ] : seen in IP address literals and :port suffixes,
which we also keep in the Host field.
This allows mysql://a,b,c/path to continue to parse, as it did in Go 1.4 and earlier.
This CL does not break any existing tests, suggesting the over-conservative
behavior was not intended and perhaps not realized.
It is especially important not to over-escape the host field, because
Go does not unescape the host field during parsing: it rejects any
host field containing % characters.
Russ Cox [Thu, 6 Aug 2015 01:22:10 +0000 (21:22 -0400)]
net/url: restrict :port checking to [ipv6]:port form
Go 1.4 and earlier accepted mysql://x@y(z:123)/foo
and I don't see any compelling reason to break that.
The CL during Go 1.5 that broke this syntax was
trying to fix #11208 and was probably too aggressive.
I added a test case for #11208 to make sure that stays
fixed.
Relaxing the check did not re-break #11208 nor did
it cause any existing test to fail. I added a test for the
mysql://x@y(z:123)/foo syntax being preserved.
Russ Cox [Wed, 5 Aug 2015 13:53:56 +0000 (09:53 -0400)]
crypto/tls: fix ConnectionState().VerifiedChains for resumed connection
Strengthening VerifyHostname exposed the fact that for resumed
connections, ConnectionState().VerifiedChains was not being saved
and restored during the ClientSessionCache operations.
Do that.
This change just saves the verified chains in the client's session
cache. It does not re-verify the certificates when resuming a
connection.
There are arguments both ways about this: we want fast, light-weight
resumption connections (thus suggesting that we shouldn't verify) but
it could also be a little surprising that, if the verification config
is changed, that would be ignored if the same session cache is used.
On the server side we do re-verify client-auth certificates, but the
situation is a little different there. The client session cache is an
object in memory that's reset each time the process restarts. But the
server's session cache is a conceptual object, held by the clients, so
can persist across server restarts. Thus the chance of a change in
verification config being surprisingly ignored is much higher in the
server case.
Jed Denlea [Tue, 4 Aug 2015 01:00:44 +0000 (18:00 -0700)]
net/http: close server conn after broken trailers
Prior to this change, broken trailers would be handled by body.Read, and
an error would be returned to its caller (likely a Handler), but that
error would go completely unnoticed by the rest of the server flow
allowing a broken connection to be reused. This is a possible request
smuggling vector.
Adam Langley [Wed, 5 Aug 2015 17:55:41 +0000 (10:55 -0700)]
crypto/tls: update testing certificates.
This change alters the certificate used in many tests so that it's no
longer self-signed. This allows some tests to exercise the standard
certificate verification paths in the future.
Change-Id: I9c3fcd6847eed8269ff3b86d9b6966406bf0642d
Reviewed-on: https://go-review.googlesource.com/13244 Reviewed-by: Russ Cox <rsc@golang.org>
Run-TryBot: Adam Langley <agl@golang.org> Reviewed-by: Adam Langley <agl@golang.org>
Austin Clements [Wed, 5 Aug 2015 15:07:47 +0000 (11:07 -0400)]
runtime: don't recheck heap trigger for periodic GC
88e945f introduced a non-speculative double check of the heap trigger
before actually starting a concurrent GC. This was necessary to fix a
race for heap-triggered GC, but broke sysmon-triggered periodic GC,
since the heap check will of course fail for periodically triggered
GC.
Fix this by telling startGC whether or not this GC was triggered by
heap size or a timer and only doing the heap size double check for GCs
triggered by heap size.
Russ Cox [Wed, 5 Aug 2015 14:20:02 +0000 (10:20 -0400)]
doc/go1.5: fix hyperlink for runtime/trace
Missed in CL 13074.
Change-Id: Ic0600341abbc423cd8d7b2201bf50e3b0bf398a7
Reviewed-on: https://go-review.googlesource.com/13167 Reviewed-by: Ian Lance Taylor <iant@golang.org>
Russ Cox [Wed, 5 Aug 2015 13:12:18 +0000 (09:12 -0400)]
go/build: enable cgo on freebsd/arm
Now that it works we need to turn it back on.
Fixes #10119.
Change-Id: I9c62d3026f7bb62c49a601ad73f33bf655372915
Reviewed-on: https://go-review.googlesource.com/13162 Reviewed-by: Ian Lance Taylor <iant@golang.org>
Russ Cox [Wed, 5 Aug 2015 14:19:12 +0000 (10:19 -0400)]
cmd/go: skip external tests on freebsd-arm builder
It is just far too slow.
I have a CL for Go 1.6 that makes many of these into internal tests.
That will improve the coverage.
It does not matter much, because basically none of the go command
tests are architecture dependent, so the other builders will catch
any problems.
Fixes freebsd-arm builder.
Change-Id: I8b2f6ac2cc1e7657019f7731c6662dc43e20bfb5
Reviewed-on: https://go-review.googlesource.com/13166 Reviewed-by: Ian Lance Taylor <iant@golang.org>
Russ Cox [Wed, 5 Aug 2015 14:11:59 +0000 (10:11 -0400)]
internal/testenv: add Builder, to report builder name
This works after golang.org/cl/13120 is running on the
coordinator (maybe it already is).
Change-Id: I4053d8e2f32fafd47b927203a6f66d5858e23376
Reviewed-on: https://go-review.googlesource.com/13165 Reviewed-by: Ian Lance Taylor <iant@golang.org>
Russ Cox [Wed, 5 Aug 2015 03:44:06 +0000 (23:44 -0400)]
runtime: align stack pointer during initcgo call on arm
This is what is causing freebsd/arm to crash mysteriously when using cgo.
The bug was introduced in golang.org/cl/4030, which moved this code out
of rt0_go and into its own function. The ARM ABI says that calls must
be made with the stack pointer at an 8-byte boundary, but only FreeBSD
seems to crash when this is violated.
Fixes #10119.
Change-Id: Ibdbe76b2c7b80943ab66b8abbb38b47acb70b1e5
Reviewed-on: https://go-review.googlesource.com/13161 Reviewed-by: Ian Lance Taylor <iant@golang.org> Reviewed-by: Dave Cheney <dave@cheney.net>
Andrew Gerrand [Wed, 5 Aug 2015 01:58:44 +0000 (11:58 +1000)]
doc: adjust installation instructions dynamically for a given download
This change allows the download page to redirect the user to
/doc/install?download=filename so the user can see installation
instructions specific to the file they are downloading.
This change also expands the "Test your Go installation" section
to instruct the user to create a workspace, hopefully leading
to less confusion down the line.
It also changes the front page download link to go directly
to the downloads page, which will in turn take them to the
installation instructions (the original destination).
This is related to this change to the tools repo:
https://golang.org/cl/13180
Change-Id: I658327bdb93ad228fb1846e389b281b15da91b1d
Reviewed-on: https://go-review.googlesource.com/13151 Reviewed-by: Chris Broadfoot <cbro@golang.org>
Austin Clements [Mon, 3 Aug 2015 22:06:05 +0000 (18:06 -0400)]
runtime: fix assist utilization computation
When commit 510fd13 enabled assists during the scan phase, it failed
to also update the code in the GC controller that computed the assist
CPU utilization and adjusted the trigger based on it. Fix that code so
it uses the start of the scan phase as the wall-clock time when
assists were enabled rather than the start of the mark phase.
Austin Clements [Mon, 3 Aug 2015 21:48:47 +0000 (17:48 -0400)]
runtime: revise assist ratio aggressively
At the start of a GC cycle, the garbage collector computes the assist
ratio based on the total scannable heap size. This was intended to be
conservative; after all, this assumes the entire heap may be reachable
and hence needs to be scanned. But it only assumes that the *current*
entire heap may be reachable. It fails to account for heap allocated
during the GC cycle. If the trigger ratio is very low (near zero), and
most of the heap is reachable when GC starts (which is likely if the
trigger ratio is near zero), then it's possible for the mutator to
create new, reachable heap fast enough that the assists won't keep up
based on the assist ratio computed at the beginning of the cycle. As a
result, the heap can grow beyond the heap goal (by hundreds of megs in
stress tests like in issue #11911).
We already have some vestigial logic for dealing with situations like
this; it just doesn't run often enough. Currently, every 10 ms during
the GC cycle, the GC revises the assist ratio. This was put in before
we switched to a conservative assist ratio (when we really were using
estimates of scannable heap), and it turns out to be exactly what we
need now. However, every 10 ms is far too infrequent for a rapidly
allocating mutator.
This commit reuses this logic, but replaces the 10 ms timer with
revising the assist ratio every time the heap is locked, which
coincides precisely with when the statistics used to compute the
assist ratio are updated.
Austin Clements [Mon, 3 Aug 2015 13:46:50 +0000 (09:46 -0400)]
runtime: make sweep proportional to spans bytes allocated
Proportional concurrent sweep is currently based on a ratio of spans
to be swept per bytes of object allocation. However, proportional
sweeping is performed during span allocation, not object allocation,
in order to minimize contention and overhead. Since objects are
allocated from spans after those spans are allocated, the system tends
to operate in debt, which means when the next GC cycle starts, there
is often sweep debt remaining, so GC has to finish the sweep, which
delays the start of the cycle and delays enabling mutator assists.
For example, it's quite likely that many Ps will simultaneously refill
their span caches immediately after a GC cycle (because GC flushes the
span caches), but at this point, there has been very little object
allocation since the end of GC, so very little sweeping is done. The
Ps then allocate objects from these cached spans, which drives up the
bytes of object allocation, but since these allocations are coming
from cached spans, nothing considers whether more sweeping has to
happen. If the sweep ratio is high enough (which can happen if the
next GC trigger is very close to the retained heap size), this can
easily represent a sweep debt of thousands of pages.
Fix this by making proportional sweep proportional to the number of
bytes of spans allocated, rather than the number of bytes of objects
allocated. Prior to allocating a span, both the small object path and
the large object path ensure credit for allocating that span, so the
system operates in the black, rather than in the red.
Combined with the previous commit, this should eliminate all sweeping
from GC start up. On the stress test in issue #11911, this reduces the
time spent sweeping during GC (and delaying start up) by several
orders of magnitude:
mean 99%ile max
pre fix 1 ms 11 ms 144 ms
post fix 270 ns 735 ns 916 ns
Austin Clements [Mon, 3 Aug 2015 13:25:23 +0000 (09:25 -0400)]
runtime: always give concurrent sweep some heap distance
Currently it's possible for the next_gc heap size trigger computed for
the next GC cycle to be less than the current allocated heap size.
This means the next cycle will start immediately, which means there's
no time to perform the concurrent sweep between GC cycles. This places
responsibility for finishing the sweep on GC itself, which delays GC
start-up and hence delays mutator assist.
Fix this by ensuring that next_gc is always at least a little higher
than the allocated heap size, so we won't trigger the next cycle
instantly.
runtime: assist the GC during GC startup and shutdown
Currently there are two sensitive periods during which a mutator can
allocate past the heap goal but mutator assists can't be enabled: 1)
at the beginning of GC between when the heap first passes the heap
trigger and sweep termination and 2) at the end of GC between mark
termination and when the background GC goroutine parks. During these
periods there's no back-pressure or safety net, so a rapidly
allocating mutator can allocate past the heap goal. This is
exacerbated if there are many goroutines because the GC coordinator is
scheduled as any other goroutine, so if it gets preempted during one
of these periods, it may stay preempted for a long period (10s or 100s
of milliseconds).
Normally the mutator does scan work to create back-pressure against
allocation, but there is no scan work during these periods. Hence, as
a fall back, if a mutator would assist but can't yet, simply yield the
CPU. This delays the mutator somewhat, but more importantly gives more
CPU time to the GC coordinator for it to complete the transition.
This is obviously a workaround. Issue #11970 suggests a far better but
far more invasive way to fix this.
Updates #11911. (This very nearly fixes the issue, but about once
every 15 minutes I get a GC cycle where the assists are enabled but
don't do enough work.)
runtime: recheck GC trigger before actually starting GC
Currently allocation checks the GC trigger speculatively during
allocation and then triggers the GC without rechecking. As a result,
it's possible for G 1 and G 2 to detect the trigger simultaneously,
both enter startGC, G 1 actually starts GC while G 2 gets preempted
until after the whole GC cycle, then G 2 immediately starts another GC
cycle even though the heap is now well under the trigger.
Fix this by re-checking the GC trigger non-speculatively just before
actually kicking off a new GC cycle.
This contributes to #11911 because when this happens, we definitely
don't finish the background sweep before starting the next GC cycle,
which can significantly delay the start of concurrent scan.
Vincent Batts [Mon, 3 Aug 2015 16:26:38 +0000 (12:26 -0400)]
archive/tar: don't treat multiple file system links as a tar hardlink
Do not assume that if stat shows multiple links that we should mark the
file as a hardlink in the tar format. If the hardlink link was not
referenced, this caused a link to "/". On an overlay file system, all
files have multiple links.
The caller must keep the inode references and set TypeLink, Size = 0,
and LinkName themselves.
Change-Id: I873b8a235bc8f8fbb271db74ee54232da36ca013
Reviewed-on: https://go-review.googlesource.com/13045 Reviewed-by: Ian Lance Taylor <iant@golang.org>