Ian Lance Taylor [Mon, 27 Jan 2014 18:18:22 +0000 (10:18 -0800)]
debug/dwarf, debug/elf: add support for reading DWARF 4 type info
In DWARF 4 the debug info for large types is put into
.debug_type sections, so that the linker can discard duplicate
info. This change adds support for reading type units.
Another small change included here is that DWARF 3 supports
storing the byte offset of a struct field as a formData rather
than a formDwarfBlock.
R=golang-codereviews, r
CC=golang-codereviews
https://golang.org/cl/56300043
Dmitriy Vyukov [Mon, 27 Jan 2014 16:29:21 +0000 (20:29 +0400)]
runtime: fix buffer overflow in stringtoslicerune
On 32-bits n*sizeof(r[0]) can overflow.
Or it can become 1<<32-eps, and mallocgc will "successfully"
allocate 0 pages for it, there are no checks downstream
and MHeap_Grow just does:
npage = (npage+15)&~15;
ask = npage<<PageShift;
Dmitriy Vyukov [Mon, 27 Jan 2014 11:11:12 +0000 (15:11 +0400)]
runtime: smarter slice grow
When growing slice take into account size of the allocated memory block.
Also apply the same optimization to string->[]byte conversion.
Fixes #6307.
benchmark old ns/op new ns/op delta
BenchmarkAppendGrowByte 45410364434108 -2.35%
BenchmarkAppendGrowString 5988567344813604 -25.17%
Dmitriy Vyukov [Fri, 24 Jan 2014 18:35:11 +0000 (22:35 +0400)]
runtime: combine small NoScan allocations
Combine NoScan allocations < 16 bytes into a single memory block.
Reduces number of allocations on json/garbage benchmarks by 10+%.
Dmitriy Vyukov [Fri, 24 Jan 2014 18:29:53 +0000 (22:29 +0400)]
sync: scalable Pool
Introduce fixed-size P-local caches.
When local caches overflow/underflow a batch of items
is transferred to/from global mutex-protected cache.
Dmitriy Vyukov [Fri, 24 Jan 2014 18:29:01 +0000 (22:29 +0400)]
runtime: do not zero terminate strings
On top of "tiny allocator" (cl/38750047), reduces number of allocs by 1% on json.
No code must rely on zero termination. So will also make debugging simpler,
by uncovering issues earlier.
json-1
allocated 79496867915766 -0.43%
allocs 93778 92790 -1.05%
time 10095779597250949 -3.67%
rest of the metrics are too noisy.
Russ Cox [Fri, 24 Jan 2014 04:11:04 +0000 (23:11 -0500)]
cmd/gc: add zeroing to enable precise stack accounting
There is more zeroing than I would like right now -
temporaries used for the new map and channel runtime
calls need to be eliminated - but it will do for now.
This CL only has an effect if you are building with
GOEXPERIMENT=precisestack ./all.bash
(or make.bash). It costs about 5% in the overall time
spent in all.bash. That number will come down before
we make it on by default, but this should be enough for
Keith to try using the precise maps for copying stacks.
amd64 only (and it's not really great generated code).
Russ Cox [Fri, 24 Jan 2014 03:51:39 +0000 (22:51 -0500)]
liblink, runtime: fix cgo on arm
The addition of TLS to ARM rewrote the MRC instruction
differently depending on whether we were using internal
or external linking mode. That's clearly not okay, since we
don't know that during compilation, which is when we now
generate the code. Also, because the change did not introduce
a real MRC instruction but instead just macro-expanded it
in the assembler, liblink is rewriting a WORD instruction that
may actually be looking for that specific constant, which would
lead to very unexpected results. It was also using one value
that happened to be 8 where a different value that also
happened to be 8 belonged. So the code was correct for those
values but not correct in general, and very confusing.
Throw it all away.
Replace with the following. There is a linker-provided symbol
runtime.tlsgm with a value (address) set to the offset from the
hardware-provided TLS base register to the g and m storage.
Any reference to that name emits an appropriate TLS relocation
to be resolved by either the internal linker or the external linker,
depending on the link mode. The relocation has exactly the
semantics of the R_ARM_TLS_LE32 relocation, which is what
the external linker provides.
This symbol is only used in two routines, runtime.load_gm and
runtime.save_gm. In both cases it is now used like this:
MRC 15, 0, R0, C13, C0, 3 // fetch TLS base pointer
MOVW $runtime·tlsgm(SB), R2
ADD R2, R0 // now R0 points at thread-local g+m storage
It is likely that this change breaks the generation of shared libraries
on ARM, because the MOVW needs to be rewritten to use the global
offset table and a different relocation type. But let's get the supported
functionality working again before we worry about unsupported
functionality.
LGTM=dave, iant
R=iant, dave
CC=golang-codereviews
https://golang.org/cl/56120043
Dmitriy Vyukov [Thu, 23 Jan 2014 20:13:21 +0000 (15:13 -0500)]
bufio: fix benchmarks behavior
Currently the benchmarks lie to testing package by doing O(N)
work under StopTimer. And that hidden O(N) actually consitutes
the bulk of benchmark work (e.g includes GC per iteration).
This behavior accounts for windows-amd64-race builder hangs.
««« original CL description
runtime: increase page size to 8K
Tcmalloc uses 8K, 32K and 64K pages, and in custom setups 256K pages.
Only Chromium uses 4K pages today (in "slow but small" configuration).
The general tendency is to increase page size, because it reduces
metadata size and DTLB pressure.
This change reduces GC pause by ~10% and slightly improves other metrics.
Dmitriy Vyukov [Thu, 23 Jan 2014 14:59:43 +0000 (18:59 +0400)]
runtime: increase page size to 8K
Tcmalloc uses 8K, 32K and 64K pages, and in custom setups 256K pages.
Only Chromium uses 4K pages today (in "slow but small" configuration).
The general tendency is to increase page size, because it reduces
metadata size and DTLB pressure.
This change reduces GC pause by ~10% and slightly improves other metrics.
Gautham Thambidorai [Wed, 22 Jan 2014 23:24:03 +0000 (18:24 -0500)]
crypto/tls: Client side support for TLS session resumption.
Adam (agl@) had already done an initial review of this CL in a branch.
Added ClientSessionState to Config which now allows clients to keep state
required to resume a TLS session with a server. A client handshake will try
and use the SessionTicket/MasterSecret in this cached state if the server
acknowledged resumption.
We also added support to cache ClientSessionState object in Config that will
be looked up by server remote address during the handshake.
Russ Cox [Wed, 22 Jan 2014 21:39:39 +0000 (16:39 -0500)]
runtime: fix typo in ARM code
The typo was introduced by one of Dmitriy's CLs this morning.
The fix makes the ARM build compile again; it still won't pass
its tests, but one thing at a time.
Jeff Sickel [Wed, 22 Jan 2014 21:21:53 +0000 (22:21 +0100)]
net: plan9 changes for default net directory
This change include updates to the probeIPv4Stack
and probeIPv6Stack to ensure that one or both
protocols are supported by ip(3).
The addition of fdMutex to netFD fixes the
TestTCPConcurrentAccept failures.
Additional changes add support for keepalive.
Brad Fitzpatrick [Wed, 22 Jan 2014 18:35:41 +0000 (10:35 -0800)]
syscall: use unsafe.Pointer in BSD kevent
Doesn't really matter for the most part, since the runtime-integrated
network poller uses its own kevent implementation, but for people using
the syscall directly, we should use an unsafe.Pointer for the precise GC
to retain the pointer arguments.
Also push down unsafe.Pointer a bit further in exec_linux.go, not
that there are any GC preemption points in the middle and sys
is still live anyway.
Dmitriy Vyukov [Wed, 22 Jan 2014 07:27:16 +0000 (11:27 +0400)]
runtime: remove locks from netpoll hotpaths
Introduces two-phase goroutine parking mechanism -- prepare to park, commit park.
This mechanism does not require backing mutex to protect wait predicate.
Use it in netpoll. See comment in netpoll.goc for details.
This slightly reduces contention between reader, writer and read/write io notifications;
and just eliminates a bunch of mutex operations from hotpaths, thus making then faster.
Dmitriy Vyukov [Wed, 22 Jan 2014 06:30:10 +0000 (10:30 +0400)]
runtime: fix and improve CPU profiling
- do not lose profiling signals when we have no mcache (possible for syscalls/cgo)
- do not lose any profiling signals on windows
- fix profiling of cgo programs on windows (they had no m->thread setup)
- properly setup tls in cgo programs on windows
- check _beginthread return value
Russ Cox [Wed, 22 Jan 2014 00:46:34 +0000 (19:46 -0500)]
liblink: remove use of linkmode on ARM
Now that liblink is compiled into the compilers and assemblers,
it must not refer to the "linkmode", since that is not known until
link time. This CL makes the ARM support no longer use linkmode,
which fixes a bug with cgo binaries that contain their own TLS
variables.
The x86 code must also remove linkmode; that is issue 7164.
Russ Cox [Tue, 21 Jan 2014 18:31:34 +0000 (13:31 -0500)]
cmd/gc: do not follow uintptr passed as function argument
The escape analysis works by tracing assignment paths from
variables that start with pointer type, or addresses of variables
(addresses are always pointers). It does allow non-pointers
in the path, so that in this code it sees x's value escape into y:
var x *[10]int
y := (*int)(unsafe.Pointer(uintptr(unsafe.Pointer(x))+32))
It must allow uintptr in order to see through this kind of
"pointer arithmetic".
It also traces such values if they end up as uintptrs passed to
functions. This used to be important because packages like
encoding/gob passed around uintptrs holding real pointers.
The introduction of precise collection of stacks has forced
code to be more honest about which declared stack variables
hold pointers and which do not. In particular, the garbage
collector no longer sees pointers stored in uintptr variables.
Because of this, packages like encoding/gob have been fixed.
There is not much point in the escape analysis accepting
uintptrs as holding pointers at call boundaries if the garbage
collector does not.
Excluding uintptr-valued arguments brings the escape
analysis in line with the garbage collector and has the
useful side effect of making arguments to syscall.Syscall
not appear to escape.
That is, this CL should yield the same benefits as
CL 45930043 (rolled back in CL 53870043), but it does
so by making uintptrs less special, not more.
Dmitriy Vyukov [Tue, 21 Jan 2014 09:06:57 +0000 (13:06 +0400)]
runtime: do not collect GC roots explicitly
Currently we collect (add) all roots into a global array in a single-threaded GC phase.
This hinders parallelism.
With this change we just kick off parallel for for number_of_goroutines+5 iterations.
Then parallel for callback decides whether it needs to scan stack of a goroutine
scan data segment, scan finalizers, etc. This eliminates the single-threaded phase entirely.
This requires to store all goroutines in an array instead of a linked list
(to allow direct indexing).
This CL also removes DebugScan functionality. It is broken because it uses
unbounded stack, so it can not run on g0. When it was working, I've found
it helpless for debugging issues because the two algorithms are too different now.
This change would require updating the DebugScan, so it's simpler to just delete it.
With 8 threads this change reduces GC pause by ~6%, while keeping cputime roughly the same.
Dmitriy Vyukov [Tue, 21 Jan 2014 07:20:23 +0000 (11:20 +0400)]
runtime: per-P defer pool
Instead of a per-goroutine stack of defers for all sizes,
introduce per-P defer pool for argument sizes 8, 24, 40, 56, 72 bytes.
For a program that starts 1e6 goroutines and then joins then:
old: rss=6.6g virtmem=10.2g time=4.85s
new: rss=4.5g virtmem= 8.2g time=3.48s
Dmitriy Vyukov [Tue, 21 Jan 2014 06:53:51 +0000 (10:53 +0400)]
runtime: zero 2-word memory blocks in-place
Currently for 2-word blocks we set the flag to clear the flag. Makes no sense.
In particular on 32-bits we call memclr always.
Dmitriy Vyukov [Tue, 21 Jan 2014 06:24:42 +0000 (10:24 +0400)]
runtime: ensure fair scheduling during frequent GCs
What was happenning is as follows:
Each writer goroutine always triggers GC during its scheduling quntum.
After GC goroutines are shuffled so that the timer goroutine is always second in the queue.
This repeats infinitely, causing timer goroutine starvation.
Fixes #7126.
Keith Randall [Fri, 17 Jan 2014 22:48:45 +0000 (14:48 -0800)]
runtime, cmd/gc: Get rid of vararg channel calls.
Vararg C calls present a problem for the GC because the
argument types are not derivable from the signature. Remove
them by passing pointers to channel elements instead of the
channel elements directly.
The compiler change is an ugly hack.
We can do better.
««« original CL description
syscall: mark arguments to Syscall as noescape
Heap arguments to "async" syscalls will break when/if we have moving GC anyway.
With this change is must not break until moving GC, because a user must
reference the object in Go to preserve liveness. Otherwise the code is broken already.
Reduces number of leaked params from 125 to 36 on linux.
Rob Pike [Fri, 17 Jan 2014 21:19:00 +0000 (13:19 -0800)]
syscall: allocate 64 bits of "basep" for Getdirentries
Recent crashes on 386 Darwin appear to be caused by this system call
smashing the stack. Phenomenology shows that allocating more data
here addresses the probem.
The guess is that since the actual system call is getdirentries64, 64 is
what we should allocate.
Dmitriy Vyukov [Fri, 17 Jan 2014 16:18:37 +0000 (20:18 +0400)]
syscall: mark arguments to Syscall as noescape
Heap arguments to "async" syscalls will break when/if we have moving GC anyway.
With this change is must not break until moving GC, because a user must
reference the object in Go to preserve liveness. Otherwise the code is broken already.
Reduces number of leaked params from 125 to 36 on linux.
Luke Curley [Fri, 17 Jan 2014 16:07:04 +0000 (11:07 -0500)]
crypto/cipher: improved cbc performance
decrypt: reduced the number of copy calls from 2n to 1.
encrypt: reduced the number of copy calls from n to 1.
Encryption is straight-forward: use dst instead of tmp when
xoring the block with the iv.
Decryption now loops backwards through the blocks abusing the
fact that the previous block's ciphertext (src) is the iv. This
means we don't need to copy the iv every time, in addition to
using dst instead of tmp like encryption.
Rob Pike [Thu, 16 Jan 2014 23:57:32 +0000 (15:57 -0800)]
net/rpc: fix inconsistency in documentation of Service.Register
Falsely claimed an old, no longer true condition that the first argument
must be a pointer.
Fixes #6697
Keith Randall [Thu, 16 Jan 2014 21:35:29 +0000 (13:35 -0800)]
reflect: Remove imprecise techniques from channel/select operations.
Reflect used to communicate to the runtime using interface words,
which is bad for precise GC because sometimes iwords hold a pointer
and sometimes they don't. This change rewrites channel and select
operations to always pass pointers to the runtime.
reflect.Select gets somewhat more expensive, as we now do an allocation
per receive case instead of one allocation whose size is the max of
all the received types. This seems unavoidable to get preciseness
(unless we move the allocation into selectgo, which is a much bigger
change).
Brad Fitzpatrick [Thu, 16 Jan 2014 19:43:52 +0000 (11:43 -0800)]
net/http: don't allow Content-Type or body on 204 and 1xx
Status codes 204, 304, and 1xx don't allow bodies. We already
had a function for this, but we were hard-coding just 304
(StatusNotModified) in a few places. Use the function
instead, and flesh out tests for all codes.
Fixes #6685
R=golang-codereviews, r
CC=golang-codereviews
https://golang.org/cl/53290044
Rob Pike [Thu, 16 Jan 2014 17:48:23 +0000 (09:48 -0800)]
fmt: fix bug printing large zero-padded hexadecimal
We forgot to include the width of "0x" when computing the crossover
from internal buffer to allocated buffer.
Also add a helper function to the test for formatting large zero-padded
test strings.
This CL makes the bitmaps a little more precise about variables
that have their address taken but for which the address does not
escape to the heap, so that the variables are kept in the stack frame
rather than allocated on the heap.
The code before this CL handled these variables by treating every
return statement as using every such variable and depending on
liveness analysis to essentially treat the variable as live during the
entire function. That approach has false positives and (worse) false
negatives. That is, it's both sloppy and buggy:
func f(b1, b2 bool) { // x live here! (sloppy)
if b2 {
print(0) // x live here! (sloppy)
return
}
var z **int
x := new(int)
*x = 42
z = &x
print(**z) // x live here (conservative)
if b2 {
print(1) // x live here (conservative)
return
}
for {
print(**z) // x not live here (buggy)
}
}
The first two liveness annotations (marked sloppy) are clearly
wrong: x cannot be live if it has not yet been declared.
The last liveness annotation (marked buggy) is also wrong:
x is live here as *z, but because there is no return statement
reachable from this point in the code, the analysis treats x as dead.
This CL changes the liveness calculation to mark such variables
live exactly at points in the code reachable from the variable
declaration. This keeps the conservative decisions but fixes
the sloppy and buggy ones.
The CL also detects ambiguously live variables, those that are
being marked live but may not actually have been initialized,
such as in this example:
func f(b1 bool) {
var z **int
if b1 {
x := new(int)
*x = 42
z = &x
} else {
y := new(int)
*y = 54
z = &y
}
print(**z) // x, y live here (conservative)
}
Since the print statement is reachable from the declaration of x,
x must conservatively be marked live. The same goes for y.
Although both x and y are marked live at the print statement,
clearly only one of them has been initialized. They are both
"ambiguously live".
These ambiguously live variables cause problems for garbage
collection: the collector cannot ignore them but also cannot
depend on them to be initialized to valid pointer values.
Ambiguously live variables do not come up too often in real code,
but recent changes to the way map and interface runtime functions
are invoked has created a large number of ambiguously live
compiler-generated temporary variables. The next CL will adjust
the analysis to understand these temporaries better, to make
ambiguously live variables fairly rare.
Once ambiguously live variables are rare enough, another CL will
introduce code at the beginning of a function to zero those
slots on the stack. At that point the garbage collector and the
stack copying routines will be able to depend on the guarantee that
if a slot is marked as live in a liveness bitmap, it is initialized.
Dmitriy Vyukov [Thu, 16 Jan 2014 08:17:00 +0000 (12:17 +0400)]
runtime: use lock-free ring for work queues
Use lock-free fixed-size ring for work queues
instead of an unbounded mutex-protected array.
The ring has single producer and multiple consumers.
If the ring overflows, work is put onto global queue.
benchmark old ns/op new ns/op delta
BenchmarkMatmult 7 5 -18.12%
BenchmarkMatmult-4 2 2 -18.98%
BenchmarkMatmult-16 1 0 -12.84%
Keith Randall [Wed, 15 Jan 2014 21:56:59 +0000 (13:56 -0800)]
reflect: add precise GC info for Call argument frame.
Give proper types to the argument/return areas
allocated for reflect calls. Avoid use of iword to
manipulate receivers, which may or may not be pointers.
Rob Pike [Wed, 15 Jan 2014 17:13:52 +0000 (09:13 -0800)]
cmd/pack: rewrite in Go
Replace the pack command, a C program, with a clean reimplementation in Go.
It does not need to reproduce the full feature set and it is no longer used by
the build chain, but has a role in looking inside archives created by the build
chain directly.
Since it's not in C, it is no longer build by dist, so remove it from cmd/dist and
make it a "tool" in cmd/go terminology.