Russ Cox [Mon, 27 Oct 2014 22:59:02 +0000 (18:59 -0400)]
test: make maplinear more robust
The test just doubled a certain number of times
and then gave up. On a mostly fast but occasionally
slow machine this may never make the test run
long enough to see the linear growth.
Change test to keep doubling until the first round
takes at least a full second, to reduce the effect of
occasional scheduling or other jitter.
The failure we saw had a time for the first round
of around 100ms.
Note that this test still passes once it sees a linear
effect, even with a very small total time.
The timeout here only applies to how long the execution
must be to support a reported failure.
Austin Clements [Mon, 27 Oct 2014 21:12:48 +0000 (17:12 -0400)]
runtime: fix endianness assumption when decoding ftab
The ftab ends with a half functab record consisting only of
the 'entry' field followed by a uint32 giving the offset of
the next table. Previously, symtabinit assumed it could read
this uint32 as a uintptr. Since this is unsafe on big endian,
explicitly read the offset as a uint32.
Ian Lance Taylor [Mon, 27 Oct 2014 15:46:18 +0000 (08:46 -0700)]
net: if a DNS lookup times out, forget that it is in flight
Before this CL, if the system resolver does a very slow DNS
lookup for a particular host, all subsequent requests for that
host will hang waiting for that lookup to complete. That is
more or less expected when Dial is called with no deadline.
When Dial has a deadline, though, we can accumulate a large
number of goroutines waiting for that slow DNS lookup. Try to
avoid this problem by restarting the DNS lookup when it is
redone after a deadline is passed.
This CL also avoids creating an extra goroutine purely to
handle the deadline.
No test because we would have to simulate a slow DNS lookup
followed by a fast DNS lookup.
Emil Hessman [Mon, 27 Oct 2014 01:43:14 +0000 (12:43 +1100)]
misc/makerelease/windows: fix 404 help URL in installer
ARPHELPLINK yields 404; update the URL.
While here, also prefix the ARPREADME and ARPURLINFOABOUT URL's with the HTTP scheme to make 'em clickable links in the Add or Remove Programs listing.
Rob Pike [Fri, 24 Oct 2014 16:37:25 +0000 (09:37 -0700)]
unsafe: document that unsafe programs are not protected
The compatibility guideline needs to be clear about this even
though it means adding a clause that was not there from the
beginning. It has always been understood, so this isn't really
a change in policy, just in its expression.
Russ Cox [Fri, 24 Oct 2014 14:27:39 +0000 (10:27 -0400)]
cmd/gc: synthesize zeroed value for non-assignment context
CL 157910047 introduced code to turn a node representing
a zeroed composite literal into N, the nil Node* pointer
(which represents any zero, not the Go literal nil).
That's great for assignments like x = T{}, but it doesn't work
when T{} is used in a value context like T{}.v or x == T{}.
Fix those.
Should have no effect on performance; confirmed.
The deltas below are noise (compare ns/op):
Russ Cox [Fri, 24 Oct 2014 03:44:47 +0000 (23:44 -0400)]
encoding/csv: for Postgres, unquote empty strings, quote \.
In theory both of these lines encode the same three fields:
a,,c
a,"",c
However, Postgres defines that when importing CSV, the unquoted
version is treated as NULL (missing), while the quoted version is
treated as a string value (empty string). If the middle field is supposed to
be an integer value, the first line can be imported (NULL is okay), but
the second line cannot (empty string is not).
Postgres's import command (COPY FROM) has an option to force
the unquoted empty to be interpreted as a string but it does not
have an option to force the quoted empty to be interpreted as a NULL.
From http://www.postgresql.org/docs/9.0/static/sql-copy.html:
The CSV format has no standard way to distinguish a NULL
value from an empty string. PostgreSQL's COPY handles this
by quoting. A NULL is output as the NULL parameter string
and is not quoted, while a non-NULL value matching the NULL
parameter string is quoted. For example, with the default
settings, a NULL is written as an unquoted empty string,
while an empty string data value is written with double
quotes (""). Reading values follows similar rules. You can
use FORCE_NOT_NULL to prevent NULL input comparisons for
specific columns.
Therefore printing the unquoted empty is more flexible for
imports into Postgres than printing the quoted empty.
In addition to making the output more useful with Postgres, not
quoting empty strings makes the output smaller and easier to read.
It also matches the behavior of Microsoft Excel and Google Drive.
Since we are here and making concessions for Postgres, handle this
case too (again quoting the Postgres docs):
Because backslash is not a special character in the CSV
format, \., the end-of-data marker, could also appear as a
data value. To avoid any misinterpretation, a \. data value
appearing as a lone entry on a line is automatically quoted
on output, and on input, if quoted, is not interpreted as
the end-of-data marker. If you are loading a file created by
another application that has a single unquoted column and
might have a value of \., you might need to quote that value
in the input file.
Dmitriy Vyukov [Wed, 22 Oct 2014 16:23:49 +0000 (20:23 +0400)]
sync: release Pool memory during second and later GCs
Pool memory was only being released during the first GC after the first Put.
Put assumes that p.local != nil means p is on the allPools list.
poolCleanup (called during each GC) removed each pool from allPools
but did not clear p.local, so each pool was cleared by exactly one GC
and then never cleared again.
This bug was introduced late in the Go 1.3 release cycle.
Dave Cheney [Wed, 22 Oct 2014 01:30:15 +0000 (12:30 +1100)]
runtime/cgo: encode BLX directly, fixes one clang build error on arm
Fixes #8348.
Trying to work around clang's dodgy support for .arch by reverting to the external assembler didn't work out so well. Minux had a much better solution to encode the instructions we need as .word directives which avoids .arch altogether.
I've confirmed with gdb that this form produces the expected machine code
Jens Frederich [Tue, 21 Oct 2014 23:56:26 +0000 (10:56 +1100)]
misc/makerelease: set version number in Windows installer
Set correct version number at Windows installer based on
Go's Mercurial tag.
Name | Version
------------------------------------------------
Go Programming Language amd64 go1.3.3 | 1.3.3
Go Programming Language amd64 go1.2rc3 | 1.2
Go Programming Language amd64 go1.2beta1 | 1.2
Some platforms are still not working even with the integrated assembler disabled, will have to find another solution.
««« original CL description
cmd/cgo: disable clang's integrated assembler
Fixes #8348.
Clang's internal assembler (introduced by default in clang 3.4) understands the .arch directive, but doesn't change the default value of -march. This causes the build to fail when we use BLX (armv5 and above) when clang is compiled for the default armv4t architecture (which appears to be the default on all the distros I've used).
This is probably a clang bug, so work around it for the time being by disabling the integrated assembler when compiling the cgo assembly shim.
This CL also includes a small change to ldelf.c which was required as clang 3.4 and above generate more weird symtab entries.
Dave Cheney [Mon, 20 Oct 2014 23:28:39 +0000 (23:28 +0000)]
cmd/cgo: disable clang's integrated assembler
Fixes #8348.
Clang's internal assembler (introduced by default in clang 3.4) understands the .arch directive, but doesn't change the default value of -march. This causes the build to fail when we use BLX (armv5 and above) when clang is compiled for the default armv4t architecture (which appears to be the default on all the distros I've used).
This is probably a clang bug, so work around it for the time being by disabling the integrated assembler when compiling the cgo assembly shim.
This CL also includes a small change to ldelf.c which was required as clang 3.4 and above generate more weird symtab entries.
Keith Randall [Mon, 20 Oct 2014 22:48:42 +0000 (15:48 -0700)]
runtime: fix flaky TestBlockProfile test
It has been failing periodically on Solaris/x64.
Change blockevent so it always records an event if we called
SetBlockProfileRate(1), even if the time delta is negative or zero.
Hopefully this will fix the test on Solaris.
Caveat: I don't actually know what the Solaris problem is, this
is just an educated guess.
LGTM=dave
R=dvyukov, dave
CC=golang-codereviews
https://golang.org/cl/159150043
Ian Lance Taylor [Mon, 20 Oct 2014 17:43:43 +0000 (10:43 -0700)]
reflect: allocate correct type in assignTo and cvtT2I
I came across this while debugging a GC problem in gccgo.
There is code in assignTo and cvtT2I that handles assignment
to all interface values. It allocates an empty interface even
if the real type is a non-empty interface. The fields are
then set for a non-empty interface, but the memory is recorded
as holding an empty interface. This means that the GC has
incorrect information.
This is extremely unlikely to fail, because the code in the GC
that handles empty interfaces looks like this:
In the current runtime the condition is always true--if
KindDirectIface is set, then KindNoPointers is clear--and we
always want to set obj = eface->data. So the question is what
happens when we incorrectly store a non-empty interface value
in memory marked as an empty interface. In that case
eface->type will not be a *rtype as we expect, but will
instead be a pointer to an Itab. We are going to use this
pointer to look at a *rtype kind field. The *rtype struct
starts out like this:
type rtype struct {
size uintptr
hash uint32 // hash of type; avoids computation in hash tables
_ uint8 // unused/padding
align uint8 // alignment of variable with this type
fieldAlign uint8 // alignment of struct field with this type
kind uint8 // enumeration for C
An Itab always has at least two pointers, so on a
little-endian 64-bit system the kind field will be the high
byte of the second pointer. This will normally be zero, so
the test of typ->kind will succeed, which is what we want.
On a 32-bit system it might be possible to construct a failing
case by somehow getting the Itab for an interface with one
method to be immediately followed by a word that is all ones.
The effect would be that the test would sometimes fail and the
GC would not mark obj, leading to an invalid dangling
pointer. I have not tried to construct this test.
I noticed this in gccgo, where this error is much more likely
to cause trouble for a rather random reason: gccgo uses a
different layout of rtype, and in gccgo the kind field happens
to be the low byte of a pointer, not the high byte.
Daniel Morsing [Mon, 20 Oct 2014 14:59:10 +0000 (15:59 +0100)]
cmd/gc: emit code for extern = <N>
https://golang.org/cl/152700045/ made it possible for struct literals assigned to globals to use <N> as the RHS. Normally, this is to zero out variables on first use. Because globals are already zero (or their linker initialized value), we just ignored this.
Now that <N> can occur from non-initialization code, we need to emit this code. We don't use <N> for initialization of globals any more, so this shouldn't cause any excessive zeroing.
Rob Pike [Mon, 20 Oct 2014 14:33:08 +0000 (07:33 -0700)]
encoding/gob: add custom decoder buffer for performance
As we did with encoding, provide a trivial byte reader for
faster decoding. We can also reduce some of the copying
by doing the allocation all at once using a slightly different
interface from byte buffers.
James Robinson [Mon, 20 Oct 2014 01:58:02 +0000 (12:58 +1100)]
compress/flate: add Reset() to allow reusing large buffers to compress multiple buffers
This adds a Reset() to compress/flate's decompressor and plumbs that through
to compress/zlib and compress/gzip's Readers so callers can avoid large
allocations when performing many inflate operations. In particular this
preserves the allocation of the decompressor.hist buffer, which is 32kb and
overwritten as needed while inflating.
On the benchmark described in issue 6317, produces the following speedup on
my 2.3ghz Intel Core i7 MBP with go version devel +6b696a34e0af Sun Aug 03
15:14:59 2014 -0700 darwin/amd64:
blocked.text w/out patch vs blocked.text w/ patch:
benchmark old ns/op new ns/op delta
BenchmarkGunzip 83715775337927917687 -5.30%
benchmark old allocs new allocs delta
BenchmarkGunzip 176818 148519 -16.00%
benchmark old bytes new bytes delta
BenchmarkGunzip 29218493612739528 -95.64%
flat.text vs blocked.text w/patch:
benchmark old ns/op new ns/op delta
BenchmarkGunzip 79394478277927917687 -0.15%
benchmark old allocs new allocs delta
BenchmarkGunzip 90702 148519 +63.74%
benchmark old bytes new bytes delta
BenchmarkGunzip 995952812739528 +27.91%
Similar speedups to those bradfitz saw in https://golang.org/cl/13416045.
Rob Pike [Sun, 19 Oct 2014 17:33:22 +0000 (10:33 -0700)]
flag: disallow setting flags multiple times
This is a day 1 error in the flag package: It did not check
that a flag was set at most once on the command line.
Because user-defined flags may have more general
properties, the check applies only to the standard flag
types in this package: bool, string, etc.
Keith Randall [Sun, 19 Oct 2014 04:02:49 +0000 (21:02 -0700)]
runtime: dequeue the correct SudoG
select {
case <- c:
case <- c:
}
In this case, c.recvq lists two SudoGs which have the same G.
So we can't use the G as the key to dequeue the correct SudoG,
as that key is ambiguous. Dequeueing the wrong SudoG ends up
freeing a SudoG that is still in c.recvq.
The fix is to use the actual SudoG pointer as the key.
Rob Pike [Sat, 18 Oct 2014 18:22:05 +0000 (11:22 -0700)]
text/template: fix bug in pipelined variadics
Simple bug in argument processing: The final arg may
be the pipeline value, in which case it gets bound to the
fixed argument section. The code got that wrong. Easy
to fix.
Rob Pike [Sat, 18 Oct 2014 03:51:15 +0000 (20:51 -0700)]
encoding/gob: use simple append-only buffer for encoding
Bytes buffers have more API and are a little slower. Since appending
is a key part of the path in encode, using a faster implementation
speeds things up measurably.
The couple of positive swings are likely garbage-collection related
since memory allocation looks different in the benchmark now.
I am not concerned by them.
Rob Pike [Fri, 17 Oct 2014 19:37:41 +0000 (12:37 -0700)]
encoding/gob: custom array/slice decoders
Use go generate to write better loops for decoding arrays,
just as we did for encoding. It doesn't help as much,
relatively speaking, but it's still noticeable.
Dmitriy Vyukov [Fri, 17 Oct 2014 17:28:47 +0000 (21:28 +0400)]
runtime/pprof: fix memory profiler test
Don't use cmd/pprof as it is not necessary installed
and does not work on nacl and plan9.
Instead just look at the raw profile.
Russ Cox [Fri, 17 Oct 2014 17:10:42 +0000 (13:10 -0400)]
cmd/gc: don't use static init to initialize small structs, fields
Better to avoid the memory loads and just use immediate constants.
This especially applies to zeroing, which was being done by
copying zeros from elsewhere in the binary, even if the value
was going to be completely initialized with non-zero values.
The zero writes were optimized away but the zero loads from
the data segment were not.
Russ Cox [Fri, 17 Oct 2014 16:54:31 +0000 (12:54 -0400)]
reflect: a few microoptimizations
Replace i < 0 || i >= x with uint(i) >= uint(x).
Shorten a few other code sequences.
Move the kind bits to the bottom of the flag word, to avoid shifts.
Rob Pike [Fri, 17 Oct 2014 16:00:07 +0000 (09:00 -0700)]
encoding/gob: speed up encoding of arrays and slices
We borrow a trick from the fmt package and avoid reflection
to walk the elements when possible. We could push further with
unsafe (and we may) but this is a good start.
Decode can benefit similarly; it will be done separately.
Use go generate (engen.go) to produce the helper functions
(enc_helpers.go).
Russ Cox [Thu, 16 Oct 2014 18:58:11 +0000 (14:58 -0400)]
runtime/pprof: disable new memory test
It cannot run 'go tool pprof'. There is no guarantee that's installed.
It needs to build a temporary pprof binary and run that.
It also needs to skip the test on systems that can't build and
run binaries, namely android and nacl.
See src/cmd/nm/nm_test.go's TestNM for a template.
Dmitriy Vyukov [Thu, 16 Oct 2014 18:11:26 +0000 (22:11 +0400)]
runtime: fix memory profiler
There are 3 issues:
1. Skip argument of callers is off by 3,
so that all allocations are deep inside of memory profiler.
2. Memory profiling statistics are not updated after runtime.GC.
3. Testing package does not update memory profiling statistics
before capturing the profile.
Also add an end-to-end test.
Fixes #8867.
Russ Cox [Thu, 16 Oct 2014 16:43:17 +0000 (12:43 -0400)]
cmd/gc: elide write barrier for x = x[0:y] and x = append(x, ...)
Both of these forms can avoid writing to the base pointer in x
(in the slice, always, and in the append, most of the time).
For Go 1.5, will need to change the compilation of x = x[0:y]
to avoid writing to the base pointer, so that the elision is safe,
and will need to change the compilation of x = append(x, ...)
to write to the base pointer (through a barrier) only when
growing the underlying array, so that the general elision is safe.
For Go 1.4, elide the write barrier always, a change that should
have equivalent performance characteristics but is much
simpler and therefore safer.
Adam Langley [Thu, 16 Oct 2014 00:54:04 +0000 (17:54 -0700)]
crypto/tls: support TLS_FALLBACK_SCSV as a server.
A new attack on CBC padding in SSLv3 was released yesterday[1]. Go only
supports SSLv3 as a server, not as a client. An easy fix is to change
the default minimum version to TLS 1.0 but that seems a little much
this late in the 1.4 process as it may break some things.
Thus this patch adds server support for TLS_FALLBACK_SCSV[2] -- a
mechanism for solving the fallback problem overall. Chrome has
implemented this since February and Google has urged others to do so in
light of yesterday's news.
With this change, clients can indicate that they are doing a fallback
connection and Go servers will be able to correctly reject them.
Russ Cox [Wed, 15 Oct 2014 23:33:15 +0000 (19:33 -0400)]
cmd/gc: simplify compiled code for explicit zeroing
Among other things, *x = T{} does not need a write barrier.
The changes here avoid an unnecessary copy even when
no pointers are involved, so it may have larger effects.
In 6g and 8g, avoid manually repeated STOSQ in favor of
writing explicit MOVs, under the theory that the MOVs
should have fewer dependencies and pipeline better.
Benchmarks compare best of 5 on a 2012 MacBook Pro Core i5
with TurboBoost disabled. Most improvements can be explained
by the changes in this CL.
The effect in Revcomp is real but harder to explain: none of
the instructions in the inner loop changed. I suspect loop
alignment but really have no idea.
Russ Cox [Wed, 15 Oct 2014 18:33:52 +0000 (14:33 -0400)]
cmd/gc: do not copy via temporary for writebarrierfat{2,3,4}
The general writebarrierfat needs a temporary for src,
because we need to pass the address of the temporary
to the writebarrierfat routine. But the new fixed-size
ones pass the value directly and don't need to introduce
the temporary.
Magnifies some of the effect of the custom write barrier change.
Comparing best of 5 with TurboBoost turned off,
on a 2012 Retina MacBook Pro Core i5.
Still not completely confident in these numbers,
but the fmt, regexp, and revcomp improvements seem real.
Russ Cox [Wed, 15 Oct 2014 18:24:18 +0000 (14:24 -0400)]
reflect: shorten value to 3 words
scalar is no longer needed, now that
interfaces always hold pointers.
Comparing best of 5 with TurboBoost turned off,
on a 2012 Retina MacBook Pro Core i5.
Still not completely confident in these numbers,
but the gob and template improvements seem real.
Russ Cox [Wed, 15 Oct 2014 17:12:16 +0000 (13:12 -0400)]
runtime: remove hand-generated ptr bitmaps for reflectcall
A Go prototype can be used instead now, and the compiler
will do a better job than we will doing it by hand.
(We got it wrong in amd64p32, causing the current build
breakage.)
The auto-prototype-matching only applies to functions
without an explicit package path, so the TEXT lines for
reflectcall and callXX are s/runtime·/·/.
LGTM=khr
R=khr
CC=golang-codereviews, iant, r
https://golang.org/cl/153600043
Brad Fitzpatrick [Wed, 15 Oct 2014 15:51:30 +0000 (17:51 +0200)]
net/http: don't send implicit gzip Accept-Encoding on Range requests
The http package by default adds "Accept-Encoding: gzip" to outgoing
requests, unless it's a bad idea, or the user requested otherwise.
Only when the http package adds its own implicit Accept-Encoding header
does the http package also transparently un-gzip the response.
If the user requested part of a document (e.g. bytes 40 to 50), it appears
that Github/Varnish send:
range(gzip(content), 40, 50)
And not:
gzip(range(content, 40, 50))
The RFC 2616 set of replacements (with the purpose of
clarifying ambiguities since 1999) has an RFC about Range
requests (http://tools.ietf.org/html/rfc7233) but does not
mention the interaction with encodings.
Regardless of whether range(gzip(content)) or gzip(range(content)) is
correct, this change prevents the Go package from asking for gzip
in requests if we're also asking for Range, avoiding the issue.
If the user cared, they can do it themselves. But Go transparently
un-gzipping a fragment of gzip is never useful.
Jens Frederich [Wed, 15 Oct 2014 03:24:58 +0000 (23:24 -0400)]
go/build: Return MultiplePackageError on importing a dir containing multiple packages
When the Import function in go/build encounters a directory
without any buildable Go source files, it returns a handy
NoGoError. Now if, instead it encounters multiple Go source files
from multiple packages, it returns a handy MultiplePackageError.
A new test for NoGoError and MultiplePackageError is also provided.
Russ Cox [Wed, 15 Oct 2014 03:24:32 +0000 (23:24 -0400)]
cmd/gc, runtime: fix race, nacl for writebarrier changes
The racewalk code was not updated for the new write barriers.
Make it more future-proof.
The new write barrier code assumed that +1 pointer would
be aligned properly for any type that might follow, but that's
not true on 32-bit systems where some types are 64-bit aligned.
The only system like that today is nacl/amd64p32.
Insert a dummy pointer so that the ambiguously typed
value is at +2 pointers, which is always max-aligned.
Rob Pike [Wed, 15 Oct 2014 03:03:35 +0000 (20:03 -0700)]
encoding/gob: make encoding structs a little faster
FieldByIndex never returns an invalid Value, so the validity
test can be avoided if the field is not indirect.
Alex Brainman [Wed, 15 Oct 2014 00:11:11 +0000 (11:11 +1100)]
runtime: handle all windows exception (second attempt)
includes undo of 22318cd31d7d and also:
- always use SetUnhandledExceptionFilter on windows-386;
- crash when receive EXCEPTION_BREAKPOINT in exception handler.
Assignments of 2-, 3-, and 4-word values were handled
by individual MOV instructions (and for scalars still are).
But if there are pointers involved, those assignments now
go through the write barrier routine. Before this CL, they
went to writebarrierfat, which calls memmove.
Memmove is too much overhead for these small
amounts of data.
Instead, call writebarrierfat{2,3,4}, which are specialized
for the specific amount of data being copied.
Today the write barrier does not care which words are
pointers, so size alone is enough to distinguish the cases.
If we keep these distinctions in Go 1.5 we will need to
expand them for all the pointer-vs-scalar possibilities,
so the current 3 functions will become 3+7+15 = 25,
still not a large burden (we deleted more morestack
functions than that when we dropped segmented stacks).
Adam Langley [Tue, 14 Oct 2014 01:35:53 +0000 (18:35 -0700)]
crypto/x509: continue to recognise MaxPathLen of zero as "no value".
In [1] the behaviour of encoding/asn1 with respect to marshaling
optional integers was changed. Previously, a zero valued integer would
be omitted when marshaling. After the change, if a default value was
set then the integer would only be omitted if it was the default value.
This changed the behaviour of crypto/x509 because
Certificate.MaxPathLen has a default value of -1 and thus zero valued
MaxPathLens would no longer be omitted when marshaling. This is
arguably a bug-fix -- a value of zero for MaxPathLen is valid and
meaningful and now could be expressed. However it broke users
(including Docker) who were not setting MaxPathLen at all.
This change again causes a zero-valued MaxPathLen to be omitted and
introduces a ZeroMathPathLen member that indicates that, yes, one
really does want a zero. This is ugly, but we value not breaking users.
Ian Lance Taylor [Mon, 13 Oct 2014 17:01:34 +0000 (10:01 -0700)]
reflect: generated unrolled GC bitmask directly
The code for a generated type is already generating an
unrolled GC bitmask. Rather than unrolling the the source
type bitmasks and copying them, just generate the required
bitmask directly. Don't mark it as an unrolled GC program,
since there is no need to do so.