Russ Cox [Thu, 16 Oct 2014 18:58:11 +0000 (14:58 -0400)]
runtime/pprof: disable new memory test
It cannot run 'go tool pprof'. There is no guarantee that's installed.
It needs to build a temporary pprof binary and run that.
It also needs to skip the test on systems that can't build and
run binaries, namely android and nacl.
See src/cmd/nm/nm_test.go's TestNM for a template.
Dmitriy Vyukov [Thu, 16 Oct 2014 18:11:26 +0000 (22:11 +0400)]
runtime: fix memory profiler
There are 3 issues:
1. Skip argument of callers is off by 3,
so that all allocations are deep inside of memory profiler.
2. Memory profiling statistics are not updated after runtime.GC.
3. Testing package does not update memory profiling statistics
before capturing the profile.
Also add an end-to-end test.
Fixes #8867.
Russ Cox [Thu, 16 Oct 2014 16:43:17 +0000 (12:43 -0400)]
cmd/gc: elide write barrier for x = x[0:y] and x = append(x, ...)
Both of these forms can avoid writing to the base pointer in x
(in the slice, always, and in the append, most of the time).
For Go 1.5, will need to change the compilation of x = x[0:y]
to avoid writing to the base pointer, so that the elision is safe,
and will need to change the compilation of x = append(x, ...)
to write to the base pointer (through a barrier) only when
growing the underlying array, so that the general elision is safe.
For Go 1.4, elide the write barrier always, a change that should
have equivalent performance characteristics but is much
simpler and therefore safer.
Adam Langley [Thu, 16 Oct 2014 00:54:04 +0000 (17:54 -0700)]
crypto/tls: support TLS_FALLBACK_SCSV as a server.
A new attack on CBC padding in SSLv3 was released yesterday[1]. Go only
supports SSLv3 as a server, not as a client. An easy fix is to change
the default minimum version to TLS 1.0 but that seems a little much
this late in the 1.4 process as it may break some things.
Thus this patch adds server support for TLS_FALLBACK_SCSV[2] -- a
mechanism for solving the fallback problem overall. Chrome has
implemented this since February and Google has urged others to do so in
light of yesterday's news.
With this change, clients can indicate that they are doing a fallback
connection and Go servers will be able to correctly reject them.
Russ Cox [Wed, 15 Oct 2014 23:33:15 +0000 (19:33 -0400)]
cmd/gc: simplify compiled code for explicit zeroing
Among other things, *x = T{} does not need a write barrier.
The changes here avoid an unnecessary copy even when
no pointers are involved, so it may have larger effects.
In 6g and 8g, avoid manually repeated STOSQ in favor of
writing explicit MOVs, under the theory that the MOVs
should have fewer dependencies and pipeline better.
Benchmarks compare best of 5 on a 2012 MacBook Pro Core i5
with TurboBoost disabled. Most improvements can be explained
by the changes in this CL.
The effect in Revcomp is real but harder to explain: none of
the instructions in the inner loop changed. I suspect loop
alignment but really have no idea.
Russ Cox [Wed, 15 Oct 2014 18:33:52 +0000 (14:33 -0400)]
cmd/gc: do not copy via temporary for writebarrierfat{2,3,4}
The general writebarrierfat needs a temporary for src,
because we need to pass the address of the temporary
to the writebarrierfat routine. But the new fixed-size
ones pass the value directly and don't need to introduce
the temporary.
Magnifies some of the effect of the custom write barrier change.
Comparing best of 5 with TurboBoost turned off,
on a 2012 Retina MacBook Pro Core i5.
Still not completely confident in these numbers,
but the fmt, regexp, and revcomp improvements seem real.
Russ Cox [Wed, 15 Oct 2014 18:24:18 +0000 (14:24 -0400)]
reflect: shorten value to 3 words
scalar is no longer needed, now that
interfaces always hold pointers.
Comparing best of 5 with TurboBoost turned off,
on a 2012 Retina MacBook Pro Core i5.
Still not completely confident in these numbers,
but the gob and template improvements seem real.
Russ Cox [Wed, 15 Oct 2014 17:12:16 +0000 (13:12 -0400)]
runtime: remove hand-generated ptr bitmaps for reflectcall
A Go prototype can be used instead now, and the compiler
will do a better job than we will doing it by hand.
(We got it wrong in amd64p32, causing the current build
breakage.)
The auto-prototype-matching only applies to functions
without an explicit package path, so the TEXT lines for
reflectcall and callXX are s/runtime·/·/.
LGTM=khr
R=khr
CC=golang-codereviews, iant, r
https://golang.org/cl/153600043
Brad Fitzpatrick [Wed, 15 Oct 2014 15:51:30 +0000 (17:51 +0200)]
net/http: don't send implicit gzip Accept-Encoding on Range requests
The http package by default adds "Accept-Encoding: gzip" to outgoing
requests, unless it's a bad idea, or the user requested otherwise.
Only when the http package adds its own implicit Accept-Encoding header
does the http package also transparently un-gzip the response.
If the user requested part of a document (e.g. bytes 40 to 50), it appears
that Github/Varnish send:
range(gzip(content), 40, 50)
And not:
gzip(range(content, 40, 50))
The RFC 2616 set of replacements (with the purpose of
clarifying ambiguities since 1999) has an RFC about Range
requests (http://tools.ietf.org/html/rfc7233) but does not
mention the interaction with encodings.
Regardless of whether range(gzip(content)) or gzip(range(content)) is
correct, this change prevents the Go package from asking for gzip
in requests if we're also asking for Range, avoiding the issue.
If the user cared, they can do it themselves. But Go transparently
un-gzipping a fragment of gzip is never useful.
Jens Frederich [Wed, 15 Oct 2014 03:24:58 +0000 (23:24 -0400)]
go/build: Return MultiplePackageError on importing a dir containing multiple packages
When the Import function in go/build encounters a directory
without any buildable Go source files, it returns a handy
NoGoError. Now if, instead it encounters multiple Go source files
from multiple packages, it returns a handy MultiplePackageError.
A new test for NoGoError and MultiplePackageError is also provided.
Russ Cox [Wed, 15 Oct 2014 03:24:32 +0000 (23:24 -0400)]
cmd/gc, runtime: fix race, nacl for writebarrier changes
The racewalk code was not updated for the new write barriers.
Make it more future-proof.
The new write barrier code assumed that +1 pointer would
be aligned properly for any type that might follow, but that's
not true on 32-bit systems where some types are 64-bit aligned.
The only system like that today is nacl/amd64p32.
Insert a dummy pointer so that the ambiguously typed
value is at +2 pointers, which is always max-aligned.
Rob Pike [Wed, 15 Oct 2014 03:03:35 +0000 (20:03 -0700)]
encoding/gob: make encoding structs a little faster
FieldByIndex never returns an invalid Value, so the validity
test can be avoided if the field is not indirect.
Alex Brainman [Wed, 15 Oct 2014 00:11:11 +0000 (11:11 +1100)]
runtime: handle all windows exception (second attempt)
includes undo of 22318cd31d7d and also:
- always use SetUnhandledExceptionFilter on windows-386;
- crash when receive EXCEPTION_BREAKPOINT in exception handler.
Assignments of 2-, 3-, and 4-word values were handled
by individual MOV instructions (and for scalars still are).
But if there are pointers involved, those assignments now
go through the write barrier routine. Before this CL, they
went to writebarrierfat, which calls memmove.
Memmove is too much overhead for these small
amounts of data.
Instead, call writebarrierfat{2,3,4}, which are specialized
for the specific amount of data being copied.
Today the write barrier does not care which words are
pointers, so size alone is enough to distinguish the cases.
If we keep these distinctions in Go 1.5 we will need to
expand them for all the pointer-vs-scalar possibilities,
so the current 3 functions will become 3+7+15 = 25,
still not a large burden (we deleted more morestack
functions than that when we dropped segmented stacks).
Adam Langley [Tue, 14 Oct 2014 01:35:53 +0000 (18:35 -0700)]
crypto/x509: continue to recognise MaxPathLen of zero as "no value".
In [1] the behaviour of encoding/asn1 with respect to marshaling
optional integers was changed. Previously, a zero valued integer would
be omitted when marshaling. After the change, if a default value was
set then the integer would only be omitted if it was the default value.
This changed the behaviour of crypto/x509 because
Certificate.MaxPathLen has a default value of -1 and thus zero valued
MaxPathLens would no longer be omitted when marshaling. This is
arguably a bug-fix -- a value of zero for MaxPathLen is valid and
meaningful and now could be expressed. However it broke users
(including Docker) who were not setting MaxPathLen at all.
This change again causes a zero-valued MaxPathLen to be omitted and
introduces a ZeroMathPathLen member that indicates that, yes, one
really does want a zero. This is ugly, but we value not breaking users.
Ian Lance Taylor [Mon, 13 Oct 2014 17:01:34 +0000 (10:01 -0700)]
reflect: generated unrolled GC bitmask directly
The code for a generated type is already generating an
unrolled GC bitmask. Rather than unrolling the the source
type bitmasks and copying them, just generate the required
bitmask directly. Don't mark it as an unrolled GC program,
since there is no need to do so.
Alex Brainman [Sat, 11 Oct 2014 11:01:04 +0000 (22:01 +1100)]
cmd/ld: do not assume that only pe section names start with '.'
Our current pe object reader assumes that every symbol starting with
'.' is section. It appeared to be true, until now gcc 4.9.1 generates
some symbols with '.' at the front. Change that logic to check other
symbol fields in addition to checking for '.'. I am not an expert
here, but it seems reasonable to me.
Added test, but it is only good, if tested with gcc 4.9.1. Otherwise
the test PASSes regardless.
Alex Brainman [Sat, 11 Oct 2014 10:34:10 +0000 (21:34 +1100)]
cmd/ld: correct pe section names if longer then 8 chars
gcc 4.9.1 generates pe sections with names longer then 8 charters.
From IMAGE_SECTION_HEADER definition:
Name
An 8-byte, null-padded UTF-8 string. There is no terminating null character
if the string is exactly eight characters long. For longer names, this
member contains a forward slash (/) followed by an ASCII representation
of a decimal number that is an offset into the string table.
Our current pe object file reader does not read string table when section
names starts with /. Do that, so (issue 8811 example)
c:\go\path\src\isssue8811>go build
# isssue8811
isssue8811/glfw(.text): isssue8811/glfw(/76): not defined
isssue8811/glfw(.text): undefined: isssue8811/glfw(/76)
becomes
c:\go\path\src\isssue8811>go build
# isssue8811
isssue8811/glfw(.text): isssue8811/glfw(.rdata$.refptr._glfwInitialized): not defined
isssue8811/glfw(.text): undefined: isssue8811/glfw(.rdata$.refptr._glfwInitialized)
Adam Langley [Fri, 10 Oct 2014 00:37:40 +0000 (17:37 -0700)]
encoding/asn1: fix explicitly tagged Times.
https://golang.org/cl/153770043/ tried to fix the case where a
implicitly tagged Time, that happened to have the same tag as
GENERALIZEDTIME, shouldn't be parsed as a GENERALIZEDTIME.
It did so, mistakenly, by testing whether params.tag != nil. But
explicitly tagged values also have a non-nil tag and there the inner
tag actually does encode the type of the value.
This change instead tests whether the tag class is UNIVERSAL before
assuming that the tag contains type information.
That was complete failure - builders are broken,
but original cl worked fine on my system.
I will need access to builders
to test this change properly.
««« original CL description
runtime: handle all windows exception
Keith Randall [Wed, 8 Oct 2014 22:57:20 +0000 (15:57 -0700)]
runtime: delay freeing of shrunk stacks until gc is done.
This change prevents confusion in the garbage collector.
The collector wants to make sure that every pointer it finds
isn't junk. Its criteria for junk is (among others) points
to a "free" span.
Because the stack shrinker modifies pointers in the heap,
there is a race condition between the GC scanner and the
shrinker. The GC scanner can see old pointers (pointers to
freed stacks). In particular this happens with SudoG.elem
pointers.
Normally this is not a problem, as pointers into stack spans
are ok. But if the freed stack is the last one in its span,
the span is marked as "free" instead of "contains stacks".
This change makes sure that even if the GC scanner sees
an old pointer, the span into which it points is still
marked as "contains stacks", and thus the GC doesn't
complain about it.
This change will make the GC pause a tiny bit slower, as
the stack freeing now happens in serial with the mark pause.
We could delay the freeing until the mutators start back up,
but this is the simplest change for now.
Ian Lance Taylor [Wed, 8 Oct 2014 22:48:46 +0000 (15:48 -0700)]
reflect: add tests for variadic method calls
These tests fail when using gccgo. In gccgo using Interface
on the value of a method function is implemented using a
variant of MakeFunc. That approach did not correctly handle
variadic functions.
LGTM=r
R=golang-codereviews, r
CC=golang-codereviews
https://golang.org/cl/151280043
Dmitriy Vyukov [Wed, 8 Oct 2014 09:51:12 +0000 (13:51 +0400)]
runtime: faster GC scan
The change contains 3 spot optimizations to scan loop:
1. Don't use byte vars, use uintptr's instead.
This seems to alleviate some codegen issue,
and alone accounts to a half of speedup.
2. Remove bitmap cache. Currently we cache only 1 byte,
so caching is not particularly effective anyway.
Removal of the cache simplifies code and positively affects regalloc.
3. Replace BitsMultiword switch with if and
do debug checks only in Debug mode.
I've benchmarked changes separately and ensured that
each of them provides speedup on top of the previous one.
This change as a whole fixes the unintentional regressions
of scan loop that were introduced during development cycle.
Fixes #8625.
Fixes #8565.
Russ Cox [Wed, 8 Oct 2014 03:17:31 +0000 (23:17 -0400)]
runtime: clear Defer.panic before removing from G.defer list
Another dangling stack pointer in a cached structure.
Same as SudoG.elem and SudoG.selectdone.
Definitely a fix, and the new test in freedefer makes the
crash reproducible, but probably not a complete fix.
I have seen one dangling pointer in a Defer.panic even
after this fix; I cannot see where it could be coming from.
I think this will fix the solaris build.
I do not think this will fix the occasional failure on the darwin build.
Keith Randall [Tue, 7 Oct 2014 22:21:00 +0000 (15:21 -0700)]
runtime: zero pointer-looking scalararg values
I have a CL which at every gc looks through data and bss
sections for nonpointer data (according to gc maps) that
looks like a pointer. These are potential missing roots.
The only thing it finds are begnign, storing stack pointers
into m0.scalararg[1] and never cleaning them up. Let's
clean them up now so the test CL passes all.bash cleanly.
The test CL can't be checked in because we might store
pointer-looking things in nonpointer data by accident.
Keith Randall [Tue, 7 Oct 2014 20:36:16 +0000 (13:36 -0700)]
runtime: update heap dump format for 1.4
We no longer have full type information in the heap, so
we can't dump that any more. Instead we dump ptr/noptr
maps so at least we can compute graph connectivity.
In addition, we still dump Iface/Eface types so together
with dwarf type info we might be able to reconstruct
types of most things in the heap.
Russ Cox [Tue, 7 Oct 2014 15:07:18 +0000 (11:07 -0400)]
runtime: crash if we see an invalid pointer into GC arena
This will help find bugs during the release freeze.
It's not clear it should be kept for the release itself.
That's issue 8861.
The most likely thing that would trigger this is stale
pointers that previously were ignored or caused memory
leaks. These were allowed due to the use of conservative
collection. Now that everything is precise, we should not
see them anymore.
The small number check reinforces what the stack copier
is already doing, catching the storage of integers in pointers.
It caught issue 8864.
The check is disabled if _cgo_allocate is linked into the binary,
which is to say if the binary is using SWIG to allocate untyped
Go memory. In that case, there are invalid pointers and there's
nothing we can do about it.
LGTM=rlh
R=golang-codereviews, dvyukov, rlh
CC=golang-codereviews, iant, khr, r
https://golang.org/cl/148470043
Russ Cox [Tue, 7 Oct 2014 15:06:51 +0000 (11:06 -0400)]
runtime: remove type-punning for Type.gc[0], gc[1]
Depending on flags&KindGCProg,
gc[0] and gc[1] are either pointers or inlined bitmap bits.
That's not compatible with a precise garbage collector:
it needs to be always pointers or never pointers.
Change the inlined bitmap case to store a pointer to an
out-of-line bitmap in gc[0]. The out-of-line bitmaps are
dedup'ed, so that for example all pointer types share the
same out-of-line bitmap.
Fixes #8864.
LGTM=r
R=golang-codereviews, dvyukov, r
CC=golang-codereviews, iant, khr, rlh
https://golang.org/cl/155820043
Jens Frederich [Tue, 7 Oct 2014 14:13:42 +0000 (07:13 -0700)]
net/http: fix authentication info leakage in Referer header (potential security risk)
http.Client calls URL.String() to fill in the Referer header, which may
contain authentication info. This patch removes authentication info from
the Referer header without introducing any API changes.
A new test for net/http is also provided.
This is the polished version of Alberto García Hierro's
https://golang.org/cl/9766046/
Brad Fitzpatrick [Mon, 6 Oct 2014 22:10:51 +0000 (15:10 -0700)]
strings: use fast path for IndexRune
Noticed while reviewing https://golang.org/cl/147690043/
I'd never seen anybody use IndexRune before, and
unsurprisingly it doesn't use the other fast paths in the
strings/bytes packages. IndexByte uses assembly.
Rob Pike [Mon, 6 Oct 2014 21:50:58 +0000 (14:50 -0700)]
go/build: do not consider "android.go" to be android-specific
A file name must have a non-empty underscore-separated
prefix before its suffix matches GOOS. This is what the
documentation already said but is not what the code did.
Fixes #8838.
This needs to be called out in the release notes.
The he single affected file
code.google.com/p/go.text/collate/tools/colcmp/darwin.go
could use a renaming but works because it has a build tag inside.
Russ Cox [Mon, 6 Oct 2014 18:18:09 +0000 (14:18 -0400)]
runtime: update docs, code for SetFinalizer
At last minute before 1.3 we relaxed SetFinalizer to avoid
crashes when you pass the result of a global alloc to it.
This avoids the crash but makes SetFinalizer a bit too relaxed.
Document that the finalizer of a global allocation may not run.
Tighten the SetFinalizer check to ignore a global allocation but
not ignore everything else.
Fixes #7656.
LGTM=r, iant
R=golang-codereviews, iant, r
CC=dvyukov, golang-codereviews, khr, rlh
https://golang.org/cl/145930043
Alex Brainman [Sun, 5 Oct 2014 02:15:13 +0000 (13:15 +1100)]
syscall: another attempt to keep windows syscall pointers live
This approach was suggested in
https://golang.org/cl/138250043/#msg15.
Unlike current version of mksyscall_windows.go,
new code could be used in go.sys and other external
repos without help from asm.
LGTM=iant
R=golang-codereviews, iant, r
CC=golang-codereviews
https://golang.org/cl/143160046