Keith Randall [Tue, 25 Mar 2014 21:11:34 +0000 (14:11 -0700)]
runtime: redo stack map entries to avoid false retention
Change two-bit stack map entries to encode:
0 = dead
1 = scalar
2 = pointer
3 = multiword
If multiword, the two-bit entry for the following word encodes:
0 = string
1 = slice
2 = iface
3 = eface
That way, during stack scanning we can check if a string
is zero length or a slice has zero capacity. We can avoid
following the contained pointer in those cases. It is safe
to do so because it can never be dereferenced, and it is
desirable to do so because it may cause false retention
of the following block in memory.
Slice feature turned off until issue 7564 is fixed.
Ian Lance Taylor [Tue, 25 Mar 2014 20:22:19 +0000 (13:22 -0700)]
runtime: accurately record whether heap memory is reserved
The existing code did not have a clear notion of whether
memory has been actually reserved. It checked based on
whether in 32-bit mode or 64-bit mode and (on GNU/Linux) the
requested address, but it confused the requested address and
the returned address.
Brad Fitzpatrick [Tue, 25 Mar 2014 17:59:09 +0000 (10:59 -0700)]
net/http: don't re-use Transport connections if we've seen an EOF
This the second part of making persistent HTTPS connections to
certain servers (notably Amazon) robust.
See the story in part 1: https://golang.org/cl/76400046/
This is the http Transport change that notes whether our
net.Conn.Read has ever seen an EOF. If it has, then we use
that as an additional signal to not re-use that connection (in
addition to the HTTP response headers)
Brad Fitzpatrick [Tue, 25 Mar 2014 17:58:35 +0000 (10:58 -0700)]
crypto/tls: make Conn.Read return (n, io.EOF) when EOF is next in buffer
Update #3514
An io.Reader is permitted to return either (n, nil)
or (n, io.EOF) on EOF or other error.
The tls package previously always returned (n, nil) for a read
of size n if n bytes were available, not surfacing errors at
the same time.
Amazon's HTTPS frontends like to hang up on clients without
sending the appropriate HTTP headers. (In their defense,
they're allowed to hang up any time, but generally a server
hangs up after a bit of inactivity, not immediately.) In any
case, the Go HTTP client tries to re-use connections by
looking at whether the response headers say to keep the
connection open, and because the connection looks okay, under
heavy load it's possible we'll reuse it immediately, writing
the next request, just as the Transport's always-reading
goroutine returns from tls.Conn.Read and sees (0, io.EOF).
But because Amazon does send an AlertCloseNotify record before
it hangs up on us, and the tls package does its own internal
buffering (up to 1024 bytes) of pending data, we have the
AlertCloseNotify in an unread buffer when our Conn.Read (to
the HTTP Transport code) reads its final bit of data in the
HTTP response body.
This change makes that final Read return (n, io.EOF) when
an AlertCloseNotify record is buffered right after, if we'd
otherwise return (n, nil).
A dependent change in the HTTP code then notes whether a
client connection has seen an io.EOF and uses that as an
additional signal to not reuse a HTTPS connection. With both
changes, the majority of Amazon request failures go
away. Without either one, 10-20 goroutines hitting the S3 API
leads to such an error rate that empirically up to 5 retries
are needed to complete an API call.
Nigel Tao [Tue, 25 Mar 2014 02:32:18 +0000 (13:32 +1100)]
database/sql: add "defer rows.Close()" to the example code.
Strictly speaking, it's not necessary in example_test.go, as the
Rows.Close docs say that "If Next returns false, the Rows are closed
automatically". However, if the for loop breaks or returns early, it's
not obvious that you'll leak unless you explicitly call Rows.Close.
Russ Cox [Tue, 25 Mar 2014 01:22:16 +0000 (21:22 -0400)]
runtime: use VEH, not SEH, for windows/386 exception handling
Structured Exception Handling (SEH) was the first way to handle
exceptions (memory faults, divides by zero) on Windows.
The S might as well stand for "stack-based": the implementation
interprets stack addresses in a few different ways, and it gets
subtly confused by Go's management of stacks. It's also something
that requires active maintenance during cgo switches, and we've
had bugs in that maintenance in the past.
We have recently come to believe that SEH cannot work with
Go's stack usage. See http://golang.org/issue/7325 for details.
Vectored Exception Handling (VEH) is more like a Unix signal
handler: you set it once for the whole process and forget about it.
This CL drops all the SEH code and replaces it with VEH code.
Many special cases and 7 #ifdefs disappear.
VEH was introduced in Windows XP, so Go on windows/386 will
now require Windows XP or later. The previous requirement was
Windows 2000 or later. Windows 2000 immediately preceded
Windows XP, so Windows 2000 is the only affected version.
Microsoft stopped supporting Windows 2000 in 2010.
See http://golang.org/s/win2000-golang-nuts for details.
Fixes #7325.
LGTM=alex.brainman, r
R=golang-codereviews, alex.brainman, stephen.gutekanst, dave
CC=golang-codereviews, iant, r
https://golang.org/cl/74790043
Rob Pike [Tue, 25 Mar 2014 00:25:20 +0000 (11:25 +1100)]
math/cmplx: define Pow(0, x) for problematic values of x.
Currently it's always zero, but that is inconsistent with math.Pow
and also plain wrong.
This is a proposal for how it should be defined.
Fixes #7583.
Russ Cox [Mon, 24 Mar 2014 23:11:21 +0000 (19:11 -0400)]
doc: allow buffered channel as semaphore without initialization
This rule not existing has been the source of many discussions
on golang-dev and on issues. We have stated publicly that it is
true, but we have never written it down. Write it down.
Rui Ueyama [Mon, 24 Mar 2014 18:48:34 +0000 (11:48 -0700)]
bufio: fix bug that ReadFrom stops before EOF or error
ReadFrom should not return until it receives a non-nil error
or too many contiguous (0, nil)s from a given reader.
Currently it immediately returns if it receives one (0, nil).
Fixes #7611.
David du Colombier [Fri, 21 Mar 2014 18:26:47 +0000 (19:26 +0100)]
cmd/ld: fix warnings on Plan 9
warning: src/cmd/ld/macho.c:595 sign-extended character constant
warning: src/cmd/ld/macho.c:595 sign-extended character constant
warning: src/cmd/ld/symtab.c:63 sign-extended character constant
warning: src/cmd/ld/symtab.c:63 sign-extended character constant
Rob Pike [Fri, 21 Mar 2014 02:59:30 +0000 (13:59 +1100)]
doc/go_faq.html: update description of stack management
They aren't segmented any more, at least with gc.
Also improve the comparison of goroutines and threads.
Fixes #7373.
Rémy Oudompheng [Thu, 20 Mar 2014 21:22:37 +0000 (22:22 +0100)]
cmd/6g, cmd/8g: skip CONVNOP nodes in bgen.
Revision 3ae4607a43ff introduced CONVNOP layers
to fix type checking issues arising from comparisons.
The added complexity made 8g run out of registers
when compiling an equality function in go.net/ipv6.
A similar issue occurred in test/sizeof.go on
amd64p32 with 6g.
Rui Ueyama [Thu, 20 Mar 2014 05:00:34 +0000 (16:00 +1100)]
base64: fix bug that decoder fails to detect corruption
Encoding.Decode() failed to detect trailing garbages if input contains "==" followed by garbage smaller than 3 bytes (for example, it failed to detect "x" in "AA==x"). This patch fixes the bug and adds a few tests.
Rob Pike [Wed, 19 Mar 2014 21:51:06 +0000 (08:51 +1100)]
fmt: make %F a synonym for %f
Rationale:
It already is for scanning.
It is accepted for complexes already, but doesn't work.
It's analogous to %G and %E.
C accepts it too, and we try to be roughly compatible.
Fixes #7518.
Dmitriy Vyukov [Wed, 19 Mar 2014 13:04:51 +0000 (17:04 +0400)]
runtime: fix stack split detection around fork
If runtime_BeforeFork splits stack, it will unsplit it
with spoiled g->stackguard. It leads to check failure in oldstack:
Rick Arnold [Tue, 18 Mar 2014 02:03:03 +0000 (13:03 +1100)]
build: fix race in doc/articles/wiki test
The original test would open a local port and then immediately close it
and use the port number in subsequent tests. Between the port being closed
and reused by the later process, it could be opened by some other program
on the machine.
Changed the test to run the server process directly and have it save the
assigned port to a text file to be used by client processes.
Rob Pike [Tue, 18 Mar 2014 00:25:04 +0000 (11:25 +1100)]
fmt: document GoStringer and explain application of formats to compound objects
%q quotes each element of a string slice; this was never explained in the docs.
Fixes #7015.
Catalin Patulea [Mon, 17 Mar 2014 22:47:16 +0000 (15:47 -0700)]
net/http/fcgi: fix handling of request ID reuse
Request ID reuse is allowed by the FastCGI spec [1]. In particular nginx uses
the same request ID, 1, for all requests on a given connection. Because
serveRequest does not remove the request from conn.requests, this causes it to
treat the second request as a duplicate and drops the connection immediately
after beginRequest. This manifests with nginx option 'fastcgi_keep_conn on' as
the following message in nginx error log:
2014/03/17 01:39:13 [error] 730#0: *109 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: x.x.x.x, server: example.org, request: "GET / HTTP/1.1", upstream: "fastcgi://127.0.0.1:9001", host: "example.org"
Because handleRecord and serveRequest run in different goroutines, access to
conn.requests must now be synchronized.
Nathan John Youngman [Sun, 16 Mar 2014 22:35:04 +0000 (09:35 +1100)]
doc: Revise Contribution Guidelines.
Smooth out the setup process for new contributors.
* Remove references $GOROOT (often not defined).
* Add a note for contributing to subrepositories.
* Emphasize that hg mail also uploads the latest copy.
Dmitriy Vyukov [Fri, 14 Mar 2014 19:32:12 +0000 (23:32 +0400)]
runtime: fix another race in bgsweep
It's possible that bgsweep constantly does not catch up for some reason,
in this case runfinq was not woken at all.
Dmitriy Vyukov [Fri, 14 Mar 2014 19:25:48 +0000 (23:25 +0400)]
runtime: fix spans corruption
The problem was that spans end up in wrong lists after split
(e.g. in h->busy instead of h->central->empty).
Also the span can be non-swept before split,
I don't know what it can cause, but it's safer to operate on swept spans.
Fixes #7544.
Shenghou Ma [Fri, 14 Mar 2014 14:07:51 +0000 (10:07 -0400)]
cmd/gc: replace '·' as '.' in ELF/Mach-O symbol tables
Old versions of DTrace (as those shipped in OS X and FreeBSD)
don't support unicode characters in symbol names. Replace '·'
to '.' to make DTrace happy.
Aram Hăvărneanu [Fri, 14 Mar 2014 13:53:05 +0000 (17:53 +0400)]
runtime: fix use after close race in Solaris network poller
The Solaris network poller uses event ports, which are
level-triggered. As such, it has to re-arm itself after each
wakeup. The arming mechanism (which runs in its own thread) raced
with the closing of a file descriptor happening in a different
thread. When a network file descriptor is about to be closed,
the network poller is awaken to give it a chance to remove its
association with the file descriptor. Because the poller always
re-armed itself, it raced with code that closed the descriptor.
This change makes the network poller check before re-arming if
the file descriptor is about to be closed, in which case it will
ignore the re-arming request. It uses the per-PollDesc lock in
order to serialize access to the PollDesc.
This change also adds extensive documentation describing the
Solaris implementation of the network poller.
Dmitriy Vyukov [Thu, 13 Mar 2014 09:25:59 +0000 (13:25 +0400)]
runtime: harden conditions when runtime panics on crash
This is especially important for SetPanicOnCrash,
but also useful for e.g. nil deref in mallocgc.
Panics on such crashes can't lead to anything useful,
only to deadlocks, hangs and obscure crashes.
This is a copy of broken but already LGTMed
https://golang.org/cl/68540043/
Dmitriy Vyukov [Thu, 13 Mar 2014 09:16:02 +0000 (13:16 +0400)]
runtime: fix stack size check
When we copy stack, we check only new size of the top segment.
This is incorrect, because we can have other segments below it.
Rémy Oudompheng [Thu, 13 Mar 2014 07:14:05 +0000 (08:14 +0100)]
cmd/gc: fix spurious type errors in walkselect.
The lowering to runtime calls introduces hidden pointers to the
arguments of select clauses. When implicit conversions were
involved it could end up with incompatible pointers. Since the
pointed-to types have the same representation, we can introduce a
forced conversion.
Anthony Martin [Thu, 13 Mar 2014 02:41:36 +0000 (19:41 -0700)]
cmd/gc: make the fpu handle all exceptions on Plan 9
The compilers expect to not be interrupted by floating
point exceptions. On Plan 9, every process starts with
interrupts enabled for invalid operation, stack overflow,
and divide by zero exceptions.
Anthony Martin [Thu, 13 Mar 2014 01:12:56 +0000 (18:12 -0700)]
os: relax the way we kill processes on Plan 9
Previously, we wrote "kill" to the process control file
to kill a program. This is problematic because it doesn't
let the program gracefully exit.
This matters especially if the process we're killing is a
Go program. On Unix, sending SIGKILL to a Go program will
automatically kill all runtime threads. On Plan 9, there
are no threads so when the program wants to exit it has to
somehow signal all of the runtime processes. It can't do
this if we mercilessly kill it by writing to it's control
file.
Instead, we now send it a note to invoke it's note handler
and let it perform any cleanup before exiting.
Anthony Martin [Thu, 13 Mar 2014 01:12:25 +0000 (18:12 -0700)]
runtime: use unoptimized memmove and memclr on Plan 9
On Plan 9, the kernel disallows the use of floating point
instructions while handling a note. Previously, we worked
around this by using a simple loop in place of memmove.
When I added that work-around, I verified that all paths
from the note handler didn't end up calling memmove. Now
that memclr is using SSE instructions, the same process
will have to be done again.
Instead of doing that, however, this CL just punts and
uses unoptimized functions everywhere on Plan 9.
Anthony Martin [Thu, 13 Mar 2014 01:10:31 +0000 (18:10 -0700)]
cmd/ld: give acid a fighting chance at unwinding the stack
Acid can't produce a stack trace without .frame symbols.
Of course, it can only unwind through linear stacks but
this is still better than nothing. (I wrote an acid func
to do the full unwind a long time ago but lost it and
haven't worked up the courage to write it again).
Note that these will only be present in the native symbol
table for Plan 9 binaries.