Shenghou Ma [Thu, 11 Jul 2013 20:35:53 +0000 (04:35 +0800)]
cmd/cgo: silence two gcc warnings for *.cgo2.c
1. "int e;" is unused, generating "unused variable" error.
2. a->e was typed void *[2], but was accessed with *(int *)(a->e), this
generated "dereferencing type-punned pointer will break strict-aliasing rules" error.
runtime: fix CPU underutilization
runtime.newproc/ready are deliberately sloppy about waking new M's,
they only ensure that there is at least 1 spinning M.
Currently to compensate for that, schedule() checks if the current P
has local work and there are no spinning M's, it wakes up another one.
It does not work if goroutines do not call schedule.
With this change a spinning M wakes up another M when it finds work to do.
It's also not ideal, but it fixes the underutilization.
A proper check would require to know the exact number of runnable G's,
but it's too expensive to maintain.
Fixes #5586.
This is reincarnation of cl/9776044 with the bug fixed.
The bug was due to code added after cl/9776044 was created:
if(tick - (((uint64)tick*0x4325c53fu)>>36)*61 == 0 && runtime·sched.runqsize > 0) {
runtime·lock(&runtime·sched);
gp = globrunqget(m->p, 1);
runtime·unlock(&runtime·sched);
}
If M gets gp from global runq here, it does not reset m->spinning.
runtime: adjust traceback for new preemptive reality
Currently it crashes as follows:
fatal error: unknown pc
...
goroutine 71698 [runnable]:
runtime.racegoend()
src/pkg/runtime/race.c:171
runtime.goexit()
src/pkg/runtime/proc.c:1276 +0x9
created by runtime_test.testConcurrentReadsAfterGrowth
src/pkg/runtime/map_test.go:264 +0x332
Robert Griesemer [Wed, 10 Jul 2013 04:12:53 +0000 (21:12 -0700)]
spec: define notion of named type
The notion of a named type is crucial for the definition
of type identity, assignability, definitions of methods.
Explicitly introduce the notion with an extra sentence.
Rob Pike [Tue, 9 Jul 2013 23:52:36 +0000 (09:52 +1000)]
cmd/go: fix a couple of bugs in coverage tooling
Merging a couple of CLs into one, since they collided in my client
and I'm lazy.
1) Fix up output in "go test -cover" case.
We need to tell the testing package the name of the package being tested
and the name of the package being covered. It can then sort out the report.
2) Filter out the _test.go files from coverage processing. We want to measure
what the tests cover, not what's covered in the tests,
The coverage for encoding/gob goes from 82.2% to 88.4%.
There may be a cleaner way to do this - suggestions welcome - but ça suffit.
Shenghou Ma [Sun, 7 Jul 2013 15:06:30 +0000 (23:06 +0800)]
cmd/go: fix "go get -u" for git repositories.
CL 10869046 changed cmd/go to checkout master branch, so
for "go get -u" to work, it must "git pull" instead of
"git fetch". Added "--ff-only" so that it won't accidentally
overwrite user changes.
Adam Langley [Tue, 2 Jul 2013 23:58:56 +0000 (19:58 -0400)]
crypto/tls: implement TLS 1.2.
This does not include AES-GCM yet. Also, it assumes that the handshake and
certificate signature hash are always SHA-256, which is true of the ciphersuites
that we currently support.
Shenghou Ma [Tue, 2 Jul 2013 16:33:38 +0000 (00:33 +0800)]
runtime: fix runtime.sigreturn_tramp for NetBSD/ARM
using m->tls[0] to save ucontext pointer is not re-entry safe, and
the old code didn't set it before the early return when signal is
received on non-Go threads.
so misc/cgo/test used to hang when testing issue 5337.
Daniel Morsing [Tue, 2 Jul 2013 15:12:08 +0000 (17:12 +0200)]
cmd/gc: fix issue with method wrappers not having escape analysis run on them.
Escape analysis needs the right curfn value on a dclfunc node, otherwise it will not analyze the function.
When generating method value wrappers, we forgot to set the curfn correctly.
Jeff R. Allen [Tue, 2 Jul 2013 01:42:29 +0000 (21:42 -0400)]
time: prevent a panic from leaving the timer mutex held
When deleting a timer, a panic due to nil deref
would leave a lock held, possibly leading to a deadlock
in a defer. Instead return false on a nil timer.
This CL provides stable in-place sorting by use of
bottom up merge sort with in-place merging done by
the SymMerge algorithm from P.-S. Kim and A. Kutzner.
The additional space needed for stable sorting (in the form of
stack space) is logarithmic in the inputs size n.
Number of calls to Less and Swap grow like O(n * log n) and
O(n * log n * log n):
Stable sorting random data uses significantly more calls
to Swap than the unstable quicksort implementation (5 times more
on n=100, 10 times more on n=1e4 and 23 times more on n=1e8).
The number of calls to Less is practically the same for Sort and
Stable.
Stable sorting 1 million random integers takes 5 times longer
than using Sort.
Andrew Gerrand [Mon, 1 Jul 2013 22:44:25 +0000 (08:44 +1000)]
cmd/godoc: set up playground for examples that are already visible
This fixes an issue where linking directly to an example makes it not
runnable and visible only in a tiny window. To see the bug in action, link: http://golang.org/pkg/strings/#example_Map
R=golang-dev, r
CC=golang-dev
https://golang.org/cl/10679050
runtime: fix memory leaks due to defers
fn can clearly hold a closure in memory.
argp/pc point into stack and so can hold
in memory a block that was previously
a large stack serment.
Depending on net/http means depending on cgo.
When the tree is in a shaky state it's nice to see sync/atomic
pass even if cgo or net causes broken binaries.
R=golang-dev, dave, r
CC=golang-dev
https://golang.org/cl/10753044
Russ Cox [Fri, 28 Jun 2013 20:37:07 +0000 (13:37 -0700)]
runtime: shorten hash lookup stack frames
On amd64 the frames are very close to the limit for a
nosplit (textflag 7) function, in part because the C compiler
does not make any attempt to reclaim space allocated for
completely registerized variables. Avoid a few short-lived
variables to reclaim two words.
R=golang-dev, r
CC=golang-dev
https://golang.org/cl/10758043
Dmitriy Vyukov [Fri, 28 Jun 2013 14:37:06 +0000 (18:37 +0400)]
runtime: introduce GODEBUG env var
Currently it replaces GOGCTRACE env var (GODEBUG=gctrace=1).
The plan is to extend it with other type of debug tracing,
e.g. GODEBUG=gctrace=1,schedtrace=100.
Dmitriy Vyukov [Fri, 28 Jun 2013 13:52:17 +0000 (17:52 +0400)]
runtime: preempt goroutines for GC
The last patch for preemptive scheduler,
with this change stoptheworld issues preemption
requests every 100us.
Update #543.
Russ Cox [Thu, 27 Jun 2013 20:51:06 +0000 (16:51 -0400)]
runtime: use gp->sched.sp for stack overflow check
On x86 it is a few words lower on the stack than m->morebuf.sp
so it is a more precise check. Enabling the check requires recording
a valid gp->sched in reflect.call too. This is a good thing in general,
since it will make stack traces during reflect.call work better, and it
may be useful for preemption too.
Dmitriy Vyukov [Thu, 27 Jun 2013 20:49:53 +0000 (00:49 +0400)]
runtime: fix goroutine status corruption
runtime.entersyscall() sets g->status = Gsyscall,
then calls runtime.lock() which causes stack split.
runtime.newstack() resets g->status to Grunning.
This will lead to crash during GC (world is not stopped) or GC will scan stack incorrectly.
On my 64-bit machine, despite being 32-bit code, fixed-base
multiplications are 7.1x faster and arbitary multiplications are 2.6x
faster.
It is difficult to review this change. However, the code is essentially
the same as code that has been open-sourced in Chromium. There it has
been successfully performing P-256 operations for several months on
many machines so the arithmetic of the code should be sound.
««« original CL description
runtime: fix CPU underutilization
runtime.newproc/ready are deliberately sloppy about waking new M's,
they only ensure that there is at least 1 spinning M.
Currently to compensate for that, schedule() checks if the current P
has local work and there are no spinning M's, it wakes up another one.
It does not work if goroutines do not call schedule.
With this change a spinning M wakes up another M when it finds work to do.
It's also not ideal, but it fixes the underutilization.
A proper check would require to know the exact number of runnable G's,
but it's too expensive to maintain.
Fixes #5586.
Dmitriy Vyukov [Thu, 27 Jun 2013 16:52:12 +0000 (20:52 +0400)]
runtime: fix CPU underutilization
runtime.newproc/ready are deliberately sloppy about waking new M's,
they only ensure that there is at least 1 spinning M.
Currently to compensate for that, schedule() checks if the current P
has local work and there are no spinning M's, it wakes up another one.
It does not work if goroutines do not call schedule.
With this change a spinning M wakes up another M when it finds work to do.
It's also not ideal, but it fixes the underutilization.
A proper check would require to know the exact number of runnable G's,
but it's too expensive to maintain.
Fixes #5586.
Russ Cox [Thu, 27 Jun 2013 15:32:01 +0000 (11:32 -0400)]
runtime: record proper goroutine state during stack split
Until now, the goroutine state has been scattered during the
execution of newstack and oldstack. It's all there, and those routines
know how to get back to a working goroutine, but other pieces of
the system, like stack traces, do not. If something does interrupt
the newstack or oldstack execution, the rest of the system can't
understand the goroutine. For example, if newstack decides there
is an overflow and calls throw, the stack tracer wouldn't dump the
goroutine correctly.
For newstack to save a useful state snapshot, it needs to be able
to rewind the PC in the function that triggered the split back to
the beginning of the function. (The PC is a few instructions in, just
after the call to morestack.) To make that possible, we change the
prologues to insert a jmp back to the beginning of the function
after the call to morestack. That is, the prologue used to be roughly:
TEXT myfunc
check for split
jmpcond nosplit
call morestack
nosplit:
sub $xxx, sp
Now an extra instruction is inserted after the call:
TEXT myfunc
start:
check for split
jmpcond nosplit
call morestack
jmp start
nosplit:
sub $xxx, sp
The jmp is not executed directly. It is decoded and simulated by
runtime.rewindmorestack to discover the beginning of the function,
and then the call to morestack returns directly to the start label
instead of to the jump instruction. So logically the jmp is still
executed, just not by the cpu.
The prologue thus repeats in the case of a function that needs a
stack split, but against the cost of the split itself, the extra few
instructions are noise. The repeated prologue has the nice effect of
making a stack split double-check that the new stack is big enough:
if morestack happens to return on a too-small stack, we'll now notice
before corruption happens.
The ability for newstack to rewind to the beginning of the function
should help preemption too. If newstack decides that it was called
for preemption instead of a stack split, it now has the goroutine state
correctly paused if rescheduling is needed, and when the goroutine
can run again, it can return to the start label on its original stack
and re-execute the split check.
Here is an example of a split stack overflow showing the full
trace, without any special cases in the stack printer.
(This one was triggered by making the split check incorrect.)
Alex Brainman [Thu, 27 Jun 2013 00:11:30 +0000 (10:11 +1000)]
syscall: handle empty environment variable values properly on windows
Setenv("AN_ENV_VAR", "") deletes AN_ENV_VAR instead of setting it
to "" at this moment. Also Getenv("AN_ENV_VAR") returns "not found",
if AN_ENV_VAR is "". Change it, so they behave like unix.
Rob Pike [Wed, 26 Jun 2013 20:14:11 +0000 (13:14 -0700)]
crypto/sha512: provide top-level Sum512 and Sum384 functions
Makes it easy to ask the simple question, what is the hash of this data?
Also fix the commentary and prints in Sum256.
Russ Cox [Wed, 26 Jun 2013 18:31:12 +0000 (14:31 -0400)]
cmd/go: proper rebuild of affected packages during go test
With this CL, go test -short -cover std successfully builds and
runs all the standard package tests. The tests that look a file
line numbers (log and runtime/debug) fail, because cover is
not inserting //line directives. Everything else passes.
ok cmd/api 0.038s coverage: 66.6% of statements
? cmd/cgo [no test files]
ok cmd/fix 0.043s coverage: 27.2% of statements
ok cmd/go 0.063s coverage: 2.4% of statements
? cmd/godoc [no test files]
ok cmd/gofmt 0.085s coverage: 61.3% of statements
? cmd/yacc [no test files]
ok archive/tar 0.023s coverage: 74.2% of statements
ok archive/zip 0.075s coverage: 71.8% of statements
ok bufio 0.149s coverage: 88.2% of statements
ok bytes 0.135s coverage: 90.4% of statements
ok compress/bzip2 0.087s coverage: 85.1% of statements
ok compress/flate 0.632s coverage: 79.3% of statements
ok compress/gzip 0.027s coverage: 76.7% of statements
ok compress/lzw 0.141s coverage: 71.2% of statements
ok compress/zlib 1.123s coverage: 77.2% of statements
ok container/heap 0.020s coverage: 85.8% of statements
ok container/list 0.021s coverage: 92.5% of statements
ok container/ring 0.030s coverage: 86.5% of statements
? crypto [no test files]
ok crypto/aes 0.054s coverage: 54.3% of statements
ok crypto/cipher 0.027s coverage: 68.8% of statements
ok crypto/des 0.041s coverage: 83.8% of statements
ok crypto/dsa 0.027s coverage: 33.1% of statements
ok crypto/ecdsa 0.048s coverage: 48.7% of statements
ok crypto/elliptic 0.030s coverage: 91.6% of statements
ok crypto/hmac 0.019s coverage: 83.3% of statements
ok crypto/md5 0.020s coverage: 78.7% of statements
ok crypto/rand 0.057s coverage: 20.8% of statements
ok crypto/rc4 0.092s coverage: 70.8% of statements
ok crypto/rsa 0.261s coverage: 80.8% of statements
ok crypto/sha1 0.019s coverage: 83.9% of statements
ok crypto/sha256 0.021s coverage: 89.0% of statements
ok crypto/sha512 0.023s coverage: 88.7% of statements
ok crypto/subtle 0.027s coverage: 83.9% of statements
ok crypto/tls 0.833s coverage: 79.7% of statements
ok crypto/x509 0.961s coverage: 74.9% of statements
? crypto/x509/pkix [no test files]
ok database/sql 0.033s coverage: 75.0% of statements
ok database/sql/driver 0.020s coverage: 46.2% of statements
ok debug/dwarf 0.023s coverage: 71.5% of statements
ok debug/elf 0.035s coverage: 58.2% of statements
ok debug/gosym 0.022s coverage: 1.8% of statements
ok debug/macho 0.023s coverage: 63.7% of statements
ok debug/pe 0.024s coverage: 50.5% of statements
ok encoding/ascii85 0.021s coverage: 89.7% of statements
ok encoding/asn1 0.022s coverage: 77.9% of statements
ok encoding/base32 0.022s coverage: 91.4% of statements
ok encoding/base64 0.020s coverage: 90.7% of statements
ok encoding/binary 0.022s coverage: 66.2% of statements
ok encoding/csv 0.022s coverage: 88.5% of statements
ok encoding/gob 0.064s coverage: 82.2% of statements
ok encoding/hex 0.019s coverage: 86.3% of statements
ok encoding/json 0.047s coverage: 77.3% of statements
ok encoding/pem 0.026s coverage: 80.5% of statements
ok encoding/xml 0.039s coverage: 85.0% of statements
ok errors 0.022s coverage: 100.0% of statements
ok expvar 0.048s coverage: 72.0% of statements
ok flag 0.019s coverage: 86.9% of statements
ok fmt 0.062s coverage: 91.2% of statements
ok go/ast 0.028s coverage: 46.3% of statements
ok go/build 0.190s coverage: 75.4% of statements
ok go/doc 0.095s coverage: 76.7% of statements
ok go/format 0.036s coverage: 79.8% of statements
ok go/parser 0.075s coverage: 82.0% of statements
ok go/printer 0.733s coverage: 88.6% of statements
ok go/scanner 0.031s coverage: 86.5% of statements
ok go/token 0.062s coverage: 79.7% of statements
? hash [no test files]
ok hash/adler32 0.029s coverage: 49.0% of statements
ok hash/crc32 0.020s coverage: 64.2% of statements
ok hash/crc64 0.021s coverage: 53.5% of statements
ok hash/fnv 0.018s coverage: 75.5% of statements
ok html 0.022s coverage: 4.5% of statements
ok html/template 0.087s coverage: 83.9% of statements
ok image 0.108s coverage: 67.1% of statements
ok image/color 0.026s coverage: 20.1% of statements
ok image/draw 0.049s coverage: 69.6% of statements
ok image/gif 0.019s coverage: 65.2% of statements
ok image/jpeg 0.197s coverage: 78.6% of statements
ok image/png 0.055s coverage: 56.5% of statements
ok index/suffixarray 0.027s coverage: 82.4% of statements
ok io 0.037s coverage: 83.4% of statements
ok io/ioutil 0.022s coverage: 70.1% of statements
FAIL log 0.020s
ok log/syslog 2.063s coverage: 71.1% of statements
ok math 0.023s coverage: 76.5% of statements
ok math/big 0.235s coverage: 79.2% of statements
ok math/cmplx 0.020s coverage: 66.5% of statements
ok math/rand 0.031s coverage: 69.9% of statements
ok mime 0.022s coverage: 83.0% of statements
ok mime/multipart 0.389s coverage: 76.1% of statements
ok net 2.219s coverage: 58.0% of statements
ok net/http 4.744s coverage: 82.9% of statements
ok net/http/cgi 0.593s coverage: 68.5% of statements
ok net/http/cookiejar 0.038s coverage: 90.3% of statements
ok net/http/fcgi 0.047s coverage: 37.6% of statements
ok net/http/httptest 0.068s coverage: 68.9% of statements
ok net/http/httputil 0.058s coverage: 52.8% of statements
? net/http/pprof [no test files]
ok net/mail 0.025s coverage: 80.3% of statements
ok net/rpc 0.063s coverage: 71.5% of statements
ok net/rpc/jsonrpc 0.047s coverage: 81.3% of statements
ok net/smtp 0.032s coverage: 74.1% of statements
ok net/textproto 0.023s coverage: 66.0% of statements
ok net/url 0.020s coverage: 78.2% of statements
ok os 4.729s coverage: 73.3% of statements
ok os/exec 39.620s coverage: 65.1% of statements
ok os/signal 0.541s coverage: 89.9% of statements
ok os/user 0.022s coverage: 62.2% of statements
ok path 0.018s coverage: 90.8% of statements
ok path/filepath 10.834s coverage: 88.4% of statements
ok reflect 0.055s coverage: 83.2% of statements
ok regexp 0.084s coverage: 75.5% of statements
ok regexp/syntax 0.547s coverage: 85.2% of statements
ok runtime 4.755s coverage: 75.9% of statements
? runtime/cgo [no test files]
FAIL runtime/debug 0.018s
ok runtime/pprof 0.368s coverage: 8.5% of statements
? runtime/race [no test files]
ok sort 0.059s coverage: 97.7% of statements
ok strconv 0.315s coverage: 95.6% of statements
ok strings 0.147s coverage: 96.1% of statements
ok sync 0.083s coverage: 56.7% of statements
ok sync/atomic 0.035s coverage: 0.0% of statements
ok syscall 0.043s coverage: 24.0% of statements
ok testing 0.018s coverage: 24.0% of statements
? testing/iotest [no test files]
ok testing/quick 0.062s coverage: 83.2% of statements
ok text/scanner 0.020s coverage: 91.5% of statements
ok text/tabwriter 0.021s coverage: 90.4% of statements
ok text/template 0.052s coverage: 81.1% of statements
ok text/template/parse 0.024s coverage: 86.1% of statements
ok time 2.431s coverage: 88.8% of statements
ok unicode 0.024s coverage: 92.1% of statements
ok unicode/utf16 0.017s coverage: 97.3% of statements
ok unicode/utf8 0.019s coverage: 97.4% of statements
? unsafe [no test files]
R=golang-dev, r
CC=golang-dev
https://golang.org/cl/10586043
Rob Pike [Wed, 26 Jun 2013 17:48:04 +0000 (10:48 -0700)]
cmd/go: log compilation errors when scanning directories and packages
Before, some packages disappear silently if the package cannot be imported,
such as if the import statement is unparseable.
Before:
% ls src
foo issue
% go list ./...
_/home/r/bug/src/foo
%
After:
% go list ./...
src/issue/issue.go:3:5: expected 'STRING', found newline
_/home/r/bug/src/foo
%
Russ Cox [Tue, 25 Jun 2013 21:23:33 +0000 (17:23 -0400)]
cmd/ld: fix line numbers when using fieldtrack
USEFIELD is a special kind of NOP, so treat it like a NOP
when generating the pc-ln table.
There are more invasive fixes that could be applied here.
I am going for minimum number of lines changed.
The smallest test case we know of is five distinct Go files
in four packages, and the bug only happens with
GOEXPERIMENT=fieldtrack enabled, which we don't
normally build with, so the test would never run
meaningfully anyway.
Russ Cox [Tue, 25 Jun 2013 21:23:21 +0000 (17:23 -0400)]
codereview: force hg update after hg pull -u during hg sync
If you hg update your client to an earlier CL, then
hg sync will move you back to tip if it pulls anything in,
but it will leave you where you are if it doesn't pull anything in.
That's confusing: make hg sync always update to tip.
Ian Lance Taylor [Tue, 25 Jun 2013 17:44:25 +0000 (10:44 -0700)]
lib9: avoid all -Wconversion warnings
Built after adding -Wconversion to the list of compiler
arguments used when building. I believe these are all OK
assuming we will not change the API. There is no effort to
detect overflow due to very long strings.
R=golang-dev, dave, rsc, r
CC=golang-dev
https://golang.org/cl/10195044
Dmitriy Vyukov [Tue, 25 Jun 2013 16:27:19 +0000 (20:27 +0400)]
sync: fix race instrumentation of WaitGroup
Currently more than 1 gorutine can execute raceWrite() in Wait()
in the following scenario:
1. goroutine 1 executes first check of wg.counter, sees that it's == 0
2. goroutine 2 executes first check of wg.counter, sees that it's == 0
3. goroutine 2 locks the mutex, sees that he is the first waiter and executes raceWrite()
4. goroutine 2 block on the semaphore
5. goroutine 3 executes Done() and unblocks goroutine 2
6. goroutine 1 lock the mutex, sees that he is the first waiter and executes raceWrite()
It produces the following false report:
WARNING: DATA RACE
Write by goroutine 35:
sync.raceWrite()
src/pkg/sync/race.go:41 +0x33
sync.(*WaitGroup).Wait()
src/pkg/sync/waitgroup.go:103 +0xae
command-line-arguments_test.TestNoRaceWaitGroupMultipleWait2()
src/pkg/runtime/race/testdata/waitgroup_test.go:156 +0x19a
testing.tRunner()
src/pkg/testing/testing.go:361 +0x108