Keith Randall [Wed, 8 Oct 2014 22:57:20 +0000 (15:57 -0700)]
runtime: delay freeing of shrunk stacks until gc is done.
This change prevents confusion in the garbage collector.
The collector wants to make sure that every pointer it finds
isn't junk. Its criteria for junk is (among others) points
to a "free" span.
Because the stack shrinker modifies pointers in the heap,
there is a race condition between the GC scanner and the
shrinker. The GC scanner can see old pointers (pointers to
freed stacks). In particular this happens with SudoG.elem
pointers.
Normally this is not a problem, as pointers into stack spans
are ok. But if the freed stack is the last one in its span,
the span is marked as "free" instead of "contains stacks".
This change makes sure that even if the GC scanner sees
an old pointer, the span into which it points is still
marked as "contains stacks", and thus the GC doesn't
complain about it.
This change will make the GC pause a tiny bit slower, as
the stack freeing now happens in serial with the mark pause.
We could delay the freeing until the mutators start back up,
but this is the simplest change for now.
Ian Lance Taylor [Wed, 8 Oct 2014 22:48:46 +0000 (15:48 -0700)]
reflect: add tests for variadic method calls
These tests fail when using gccgo. In gccgo using Interface
on the value of a method function is implemented using a
variant of MakeFunc. That approach did not correctly handle
variadic functions.
LGTM=r
R=golang-codereviews, r
CC=golang-codereviews
https://golang.org/cl/151280043
Dmitriy Vyukov [Wed, 8 Oct 2014 09:51:12 +0000 (13:51 +0400)]
runtime: faster GC scan
The change contains 3 spot optimizations to scan loop:
1. Don't use byte vars, use uintptr's instead.
This seems to alleviate some codegen issue,
and alone accounts to a half of speedup.
2. Remove bitmap cache. Currently we cache only 1 byte,
so caching is not particularly effective anyway.
Removal of the cache simplifies code and positively affects regalloc.
3. Replace BitsMultiword switch with if and
do debug checks only in Debug mode.
I've benchmarked changes separately and ensured that
each of them provides speedup on top of the previous one.
This change as a whole fixes the unintentional regressions
of scan loop that were introduced during development cycle.
Fixes #8625.
Fixes #8565.
Russ Cox [Wed, 8 Oct 2014 03:17:31 +0000 (23:17 -0400)]
runtime: clear Defer.panic before removing from G.defer list
Another dangling stack pointer in a cached structure.
Same as SudoG.elem and SudoG.selectdone.
Definitely a fix, and the new test in freedefer makes the
crash reproducible, but probably not a complete fix.
I have seen one dangling pointer in a Defer.panic even
after this fix; I cannot see where it could be coming from.
I think this will fix the solaris build.
I do not think this will fix the occasional failure on the darwin build.
Keith Randall [Tue, 7 Oct 2014 22:21:00 +0000 (15:21 -0700)]
runtime: zero pointer-looking scalararg values
I have a CL which at every gc looks through data and bss
sections for nonpointer data (according to gc maps) that
looks like a pointer. These are potential missing roots.
The only thing it finds are begnign, storing stack pointers
into m0.scalararg[1] and never cleaning them up. Let's
clean them up now so the test CL passes all.bash cleanly.
The test CL can't be checked in because we might store
pointer-looking things in nonpointer data by accident.
Keith Randall [Tue, 7 Oct 2014 20:36:16 +0000 (13:36 -0700)]
runtime: update heap dump format for 1.4
We no longer have full type information in the heap, so
we can't dump that any more. Instead we dump ptr/noptr
maps so at least we can compute graph connectivity.
In addition, we still dump Iface/Eface types so together
with dwarf type info we might be able to reconstruct
types of most things in the heap.
Russ Cox [Tue, 7 Oct 2014 15:07:18 +0000 (11:07 -0400)]
runtime: crash if we see an invalid pointer into GC arena
This will help find bugs during the release freeze.
It's not clear it should be kept for the release itself.
That's issue 8861.
The most likely thing that would trigger this is stale
pointers that previously were ignored or caused memory
leaks. These were allowed due to the use of conservative
collection. Now that everything is precise, we should not
see them anymore.
The small number check reinforces what the stack copier
is already doing, catching the storage of integers in pointers.
It caught issue 8864.
The check is disabled if _cgo_allocate is linked into the binary,
which is to say if the binary is using SWIG to allocate untyped
Go memory. In that case, there are invalid pointers and there's
nothing we can do about it.
LGTM=rlh
R=golang-codereviews, dvyukov, rlh
CC=golang-codereviews, iant, khr, r
https://golang.org/cl/148470043
Russ Cox [Tue, 7 Oct 2014 15:06:51 +0000 (11:06 -0400)]
runtime: remove type-punning for Type.gc[0], gc[1]
Depending on flags&KindGCProg,
gc[0] and gc[1] are either pointers or inlined bitmap bits.
That's not compatible with a precise garbage collector:
it needs to be always pointers or never pointers.
Change the inlined bitmap case to store a pointer to an
out-of-line bitmap in gc[0]. The out-of-line bitmaps are
dedup'ed, so that for example all pointer types share the
same out-of-line bitmap.
Fixes #8864.
LGTM=r
R=golang-codereviews, dvyukov, r
CC=golang-codereviews, iant, khr, rlh
https://golang.org/cl/155820043
Jens Frederich [Tue, 7 Oct 2014 14:13:42 +0000 (07:13 -0700)]
net/http: fix authentication info leakage in Referer header (potential security risk)
http.Client calls URL.String() to fill in the Referer header, which may
contain authentication info. This patch removes authentication info from
the Referer header without introducing any API changes.
A new test for net/http is also provided.
This is the polished version of Alberto García Hierro's
https://golang.org/cl/9766046/
Brad Fitzpatrick [Mon, 6 Oct 2014 22:10:51 +0000 (15:10 -0700)]
strings: use fast path for IndexRune
Noticed while reviewing https://golang.org/cl/147690043/
I'd never seen anybody use IndexRune before, and
unsurprisingly it doesn't use the other fast paths in the
strings/bytes packages. IndexByte uses assembly.
Rob Pike [Mon, 6 Oct 2014 21:50:58 +0000 (14:50 -0700)]
go/build: do not consider "android.go" to be android-specific
A file name must have a non-empty underscore-separated
prefix before its suffix matches GOOS. This is what the
documentation already said but is not what the code did.
Fixes #8838.
This needs to be called out in the release notes.
The he single affected file
code.google.com/p/go.text/collate/tools/colcmp/darwin.go
could use a renaming but works because it has a build tag inside.
Russ Cox [Mon, 6 Oct 2014 18:18:09 +0000 (14:18 -0400)]
runtime: update docs, code for SetFinalizer
At last minute before 1.3 we relaxed SetFinalizer to avoid
crashes when you pass the result of a global alloc to it.
This avoids the crash but makes SetFinalizer a bit too relaxed.
Document that the finalizer of a global allocation may not run.
Tighten the SetFinalizer check to ignore a global allocation but
not ignore everything else.
Fixes #7656.
LGTM=r, iant
R=golang-codereviews, iant, r
CC=dvyukov, golang-codereviews, khr, rlh
https://golang.org/cl/145930043
Alex Brainman [Sun, 5 Oct 2014 02:15:13 +0000 (13:15 +1100)]
syscall: another attempt to keep windows syscall pointers live
This approach was suggested in
https://golang.org/cl/138250043/#msg15.
Unlike current version of mksyscall_windows.go,
new code could be used in go.sys and other external
repos without help from asm.
LGTM=iant
R=golang-codereviews, iant, r
CC=golang-codereviews
https://golang.org/cl/143160046
Rob Pike [Sat, 4 Oct 2014 03:27:08 +0000 (20:27 -0700)]
fmt: print &map like &slice and &struct
It was inconsistent.
Also test these better.
Also document the default format for types.
This wasn't written down.
Rob Pike [Fri, 3 Oct 2014 20:23:35 +0000 (13:23 -0700)]
fmt: part 2 of the great flag rebuild: make %+v work in formatters
Apply a similar transformation to %+v that we did to %#v, making it
a top-level setting separate from the + flag itself. This fixes the
appearance of flags in Formatters and cleans up the code too,
probably making it a little faster.
Russ Cox [Fri, 3 Oct 2014 16:44:20 +0000 (12:44 -0400)]
cmd/cc, runtime: disallow structs without tags
Structs without tags have no unique name to use in the
Go definitions generated from the C types.
This caused issue 8812, fixed by CL 149260043.
Avoid future problems by requiring struct tags.
Update runtime as needed.
(There is no other C code in the tree.)
LGTM=bradfitz, iant
R=golang-codereviews, bradfitz, dave, iant
CC=golang-codereviews, khr, r
https://golang.org/cl/150360043
Rob Pike [Thu, 2 Oct 2014 21:16:58 +0000 (14:16 -0700)]
fmt: make the %#v verb a special flag
The %#v verb is special: it says all values below need to print as %#v.
However, for some situations the # flag has other meanings and this
causes some issues, particularly in how Formatters work. Since %#v
dominates all formatting, translate it into actual state of the formatter
and decouple it from the # flag itself within the calculations (although
it must be restored when methods are doing the work.)
The result is cleaner code and correct handling of # for Formatters.
TODO: Apply the same thinking to the + flag in a followup CL.
Also, the wasString return value in handleMethods is always false,
so eliminate it.
Robert Griesemer [Thu, 2 Oct 2014 20:02:25 +0000 (13:02 -0700)]
math/big: math.Exp should return result >= 0 for |m| > 0
The documentation states that Exp(x, y, m)
computes x**y mod |m| for m != nil && m > 0.
In math.big, Mod is the Euclidean modulus,
which is always >= 0.
Rob Pike [Wed, 1 Oct 2014 22:25:56 +0000 (15:25 -0700)]
doc/go_faq.html: explain the policy about unused imports a little better
This new text won't stop the whining but it might focus the whining a little more.
Rob Pike [Wed, 1 Oct 2014 21:35:12 +0000 (21:35 +0000)]
fmt: fix internal unknownType function
This thing should never be called, but before 151960044 it was being called, incorrectly.
This is now just a precaution but let's pretend it
Fixes #8843
even though that was fixed by 151960044.
The test case was already there and ran, another mystery.
Rob Pike [Wed, 1 Oct 2014 20:18:44 +0000 (13:18 -0700)]
net/rpc: shut down connection if gob has error
The nicest solution would be to buffer the message and only write
it if it encodes correctly, but that adds considerable memory and
CPU overhead for a very rare condition. Instead, we just shut
down the connection if this happens.
Fixes #7689.
Paul van Brouwershaven [Tue, 30 Sep 2014 20:38:48 +0000 (13:38 -0700)]
x509: Fixed ASN.1 encoding in CRL Distribution Points extension
The ASN.1 encoding of the CRL Distribution Points extension showed an invalid false 'IsCompound' which caused a display problem in the Windows certificate viewer.
The internal parse and format functions in both packages
were almost identical - made them identical by adding an
extra parameter, and documented them as identical.
Eventually we should find a nice way to factor these functions
out, but we cannot do this now while in prep for 1.4.
Robert Griesemer [Tue, 30 Sep 2014 18:44:29 +0000 (11:44 -0700)]
spec: clarify variable declaration type rules
Not a language change.
Several inaccuracies were fixed:
1) A variable declaration may declare more than just one
variable.
2) Variable initialization follows the rules of assignments,
including n:1 assignments. The existing wording implied a 1:1
or n:n rule and generally was somewhat unspecific.
3) The rules for variable declarations with no types and
untyped initialization expressions had minor holes (issue 8088).
4) Clarified the special cases of assignments of untyped values
(we don't just have untyped constants, but also untyped bools,
e.g. from comparisons). The new wording is more direct.
To that end, introduced the notion of an untyped constant's
"default type" so that the same concept doesn't have to be
repeatedly introduced.
Fixes #8088.
LGTM=iant, r, rsc
R=r, rsc, iant, ken
CC=golang-codereviews
https://golang.org/cl/142320043
cmd/go: sometimes name tmp test binary test.test.exe on Windows
Right now it is always pkgname.test.exe, but if pkgname is
patch or install or setup or update, Windows thinks that
running it will install new software, so it pops up a dialog
box asking for more permission.
Renaming the binary avoids the Windows security check.
This only applies to the binary that the Go command writes
to its temporary work directory. If the user runs 'go test -c'
or any of the other ways to generate a test binary, it will
continue to use pkgname.test.exe.
Fixes #8711.
LGTM=bradfitz
R=golang-codereviews, r
CC=alex.brainman, bradfitz, golang-codereviews, iant
https://golang.org/cl/146580043
This is a new implementation of pprof,
written in Go instead of in Perl.
It was written primarily by Raul Silvera and
is in use for profiling programs of all languages
inside Google.
The internal structure is a bit package-heavy,
but it matches the copy used inside Google, and
since it is in an internal directory, we can make
changes to it later if we need to.
The only "new" file here is src/cmd/pprof/pprof.go,
which stitches together the Google pprof and the
Go command libraries for object file access.
I am explicitly NOT interested in style or review
comments on the rest of the files
(that is, src/cmd/pprof/internal/...).
Those are intended to stay as close to the Google
copies as possible, like we did with the pprof Perl script.
Still to do:
- Basic tests.
- Real command documentation.
- Hook up disassemblers.
LGTM=r
R=r, bradfitz, alex.brainman, dave
CC=golang-codereviews
https://golang.org/cl/153750043
cmd/objdump: move armasm, x86asm into internal packages
For Go 1.3 these external packages were collapsed into
large single-file implementations stored in the cmd/objdump
directory.
For Go 1.4 we want pprof to be able to link against them too,
so move them into cmd/internal, where they can be shared.
The new files are copied from the repo in the file path (rsc.io/...).
Those repos were code reviewed during development
(mainly by crawshaw and minux), because we knew the
main repo would use them.
Keith Randall [Tue, 30 Sep 2014 15:51:02 +0000 (08:51 -0700)]
runtime: fix scanning of gc work buffer
GC types were not being generated for the garbage collector
work buffer. The markfor object was being collected as a result.
This broke amd64p32 and maybe plan9 builds. Why it didn't break
every build I'm not sure...
Keith Randall [Tue, 30 Sep 2014 04:21:36 +0000 (21:21 -0700)]
runtime: initialize traceback variables earlier
Our traceback code needs to know the PC of several special
functions, including goexit, mcall, etc. Make sure that
these PCs are initialized before any traceback occurs.
net/http: make Transport.CloseIdleConnections also close pending dials
See comment 4 of https://code.google.com/p/go/issues/detail?id=8483#c4:
"So if a user creates a http.Client, issues a bunch of
requests and then wants to shutdown it and all opened connections;
what is she intended to do? The report suggests that just waiting for
all pending requests and calling CloseIdleConnections won't do, as
there can be new racing connections. Obviously she can't do what
you've done in the test, as it uses the unexported function. If this
happens periodically, it can lead to serious resource leaks (the
transport is also preserved alive). Am I missing something?"
This CL tracks the user's intention to close all idle
connections (CloseIdleConnections sets it true; and making a
new request sets it false). If a pending dial finishes and
nobody wants it, before it's retained for a future caller, the
"wantIdle" bool is checked and it's closed if the user has
called CloseIdleConnections without a later call to make a new
request.
Dmitri Shuralyov [Tue, 30 Sep 2014 00:04:48 +0000 (17:04 -0700)]
go/format, cmd/gofmt: fix issues with partial Go code with indent
Fixes #5551.
Fixes #4449.
Adds tests for both issues.
Note that the two issues occur only when formatting partial Go code
with indent.
The best way to understand the change is as follows: I took the code
of cmd/gofmt and go/format, combined it into one unified code that
does not suffer from either 4449 nor 5551, and then applied that code
to both cmd/gofmt and go/format.
As a result, there is now much more identical code between the two
packages, making future code deduplication easier (it was not possible
to do that now without adding public APIs, which I was advised not to
do at this time).
More specifically, I took the parse() of cmd/gofmt which correctly
preserves comments (issue 5551) and modified it to fix issue where
it would sometimes modify literal values (issue 4449).
I ended up removing the matchSpace() function because it no longer
needed to do some of its work (insert indent), and a part of its work
had to be done in advance (determining the indentation of first code
line), because that calculation is required for cfg.Fprint() to run.
adjustIndent is used to adjust the indent of cfg.Fprint() to compensate
for the body of wrapper func being indented by one level. This allows
to get rid of the bytes.Replace text manipulation of inner content,
which was problematic and sometimes altered raw string literals (issue
4449). This means that sometimes the value of cfg.Indent is negative,
but that works as expected.
So now the algorithm for formatting partial Go code is:
1. Determine and prepend leading space of original source.
2. Determine and prepend indentation of first code line.
3. Format and write partial Go code (with all of its leading &
trailing space trimmed).
4. Determine and append trailing space of original source.
Tom Linford [Mon, 29 Sep 2014 23:51:49 +0000 (09:51 +1000)]
x509: add root certs for android.
On android, root certificates appear to be stored in the folder
/system/etc/security/cacerts, which has many certs in several
different files. This change adds a new array of directories in
which certs can be found.
To test this, I simply tried making a request with the http
library to an HTTPS URL on an android emulator and manually
verified that it worked.
Ian Lance Taylor [Mon, 29 Sep 2014 20:32:14 +0000 (13:32 -0700)]
cmd/yacc: fix handling of tokens that don't start with letters
CL 149110043 changed yacc to no longer keep a leading space
for quoted tokens. That is OK by itself but unfortunately
yacc was relying on that leading space to notice which tokens
it should not output as const declarations.
Add a few such tokens to expr.y, although it won't make any
immediate difference as we seem to have no tests for yacc.