Tim Möhlmann [Fri, 10 Jan 2020 10:06:53 +0000 (12:06 +0200)]
database/sql: add method Err on sql.Row
The Row.Err method is intended to assist wrapping sql.DB.
Because sql.Row is a struct with private fields,
a wrapper in an existing code base cannot easily provide users
with a different implementation without large rewrites.
Adding this method allows query level errors to be handled
centrally.
Fixes #35804
Change-Id: I94e6329de89a7ee1284ce9ef76af4363d2d081f9
Reviewed-on: https://go-review.googlesource.com/c/go/+/214317 Reviewed-by: Daniel Theophanes <kardianos@gmail.com>
Run-TryBot: Daniel Theophanes <kardianos@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Kyle Nusbaum [Wed, 18 Mar 2020 19:17:50 +0000 (19:17 +0000)]
context: prevent creation of invalid contexts
This commit makes it impossible to create derived contexts with nil parents.
Previously it was possible to create derived contexts with nil parents, and
invalid contexts could propogate through the program. Eventually this can
cause a panic downstream, which is difficult to trace back to the source
of the error.
Although `WithCancel` and `WithDeadline` already panic if `parent` is `nil`, this adds explicit checks to give a useful message in the panic.
Fixes #37908
Change-Id: I70fd01f6539c1b0da0e775fc5457e32e7075e52c
GitHub-Last-Rev: 1b7dadd7db9ba42952644ad5e9a49591d6a5191f
GitHub-Pull-Request: golang/go#37898
Reviewed-on: https://go-review.googlesource.com/c/go/+/223777
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Michael Anthony Knyszek [Mon, 17 Jun 2019 19:03:09 +0000 (19:03 +0000)]
runtime: don't hold worldsema across mark phase
This change makes it so that worldsema isn't held across the mark phase.
This means that various operations like ReadMemStats may now stop the
world during the mark phase, reducing latency on such operations.
Only three such operations are still no longer allowed to occur during
marking: GOMAXPROCS, StartTrace, and StopTrace.
For the former it's because any change to GOMAXPROCS impacts GC mark
background worker scheduling and the details there are tricky.
For the latter two it's because tracing needs to observe consistent GC
start and GC end events, and if StartTrace or StopTrace may stop the
world during marking, then it's possible for it to see a GC end event
without a start or GC start event without an end, respectively.
To ensure that GOMAXPROCS and StartTrace/StopTrace cannot proceed until
marking is complete, the runtime now holds a new semaphore, gcsema,
across the mark phase just like it used to with worldsema.
This change is being landed once more after being reverted in the Go
1.14 release cycle, since CL 215157 allows it to have a positive
effect on system performance.
For the benchmark BenchmarkReadMemStatsLatency in the runtime, which
measures ReadMemStats latencies while the GC is exercised, the tail of
these latencies reduced dramatically on an 8-core machine:
name old 50%tile-ns new 50%tile-ns delta
ReadMemStatsLatency-8 4.40M ±74% 0.12M ± 2% -97.35% (p=0.008 n=5+5)
name old 90%tile-ns new 90%tile-ns delta
ReadMemStatsLatency-8 102M ± 6% 0M ±14% -99.79% (p=0.008 n=5+5)
name old 99%tile-ns new 99%tile-ns delta
ReadMemStatsLatency-8 147M ±18% 4M ±57% -97.43% (p=0.008 n=5+5)
Michael Anthony Knyszek [Wed, 19 Feb 2020 20:54:25 +0000 (20:54 +0000)]
runtime: add ReadMemStats latency benchmark
This change adds a benchmark to the runtime which measures ReadMemStats
latencies. It generates allocations with lots of pointers to keep the GC
busy while hitting ReadMemStats and measuring the time it takes to
complete.
Michael Anthony Knyszek [Thu, 16 Jan 2020 21:39:42 +0000 (21:39 +0000)]
runtime: preempt dedicated background mark workers for STW
Currently, dedicated background mark workers are essentially always
non-preemptible.
This change makes it so that dedicated background mark workers park if
their preemption flag is set and someone is trying to STW, allowing them
to do so.
This change prepares us for allowing a STW to happen (and happen
promptly) during GC marking in a follow-up change.
Jay Conrod [Mon, 10 Feb 2020 19:42:02 +0000 (14:42 -0500)]
cmd/go/internal/modfetch/zip_sum_test: remove dead versions
* gitlab.com/yumeko/MumbleEmu: the go-import tag now points to
gitlab.com/yumeko/mumbleemu, but the module path hasn't changed
in go.mod.
* github.com/openshift/api: tag v3.9.0 was deleted.
* github.com/AlexStocks/log4go: tag v1.0.5 was deleted.
* github.com/belogik/goes: repository is no longer available.
* llvm.org/llvm: server times out and disconnects after 30-40 mins.
Also, fix a typo in an error message.
With these versions removed, zip_sum_test passes.
Updates #35290
Change-Id: Id3bdb8675a5582f88a6ff4c12dd7d1abe31aa56f
Reviewed-on: https://go-review.googlesource.com/c/go/+/218917
Run-TryBot: Jay Conrod <jayconrod@google.com> Reviewed-by: Bryan C. Mills <bcmills@google.com>
Cherry Zhang [Tue, 17 Mar 2020 00:08:00 +0000 (20:08 -0400)]
runtime: don't send preemption signal if there is a signal pending
If multiple threads call preemptone to preempt the same M, it may
send many signals to the same M such that it hardly make
progress, causing live-lock problem. Only send a signal if there
isn't already one pending.
Bryan C. Mills [Tue, 17 Mar 2020 19:17:22 +0000 (15:17 -0400)]
cmd/go: add a missing curly-brace in the 'stale' command format string
The missing brace made the 'stale' command a no-op in the non-error case.
Fix the 'short' skip in install_cross_gobin (it was backward) and
update it to no longer check staleness of a not-necessarily-stale
target and to no longer expect to be able to install into GOROOT/pkg.
(This was missed in #30316 because that part of the test was
erroneously skipped in non-short mode.)
Change-Id: I6a276fec5fa5e5da3fe0daf0c2b5086116ed7c1a
Reviewed-on: https://go-review.googlesource.com/c/go/+/223747
Run-TryBot: Bryan C. Mills <bcmills@google.com> Reviewed-by: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Daniel Theophanes [Fri, 26 Apr 2019 21:45:46 +0000 (14:45 -0700)]
database/sql: process all Session Resets synchronously
Adds a new interface, driver.ConnectionValidator, to allow
drivers to signal they should not be used again,
separatly from the session resetter interface.
This is done now that the session reset is done
after the connection is put into the connection pool.
Previous behavior attempted to run Session Resets
in a background worker. This implementation had two
problems: untested performance gains for additional
complexity, and failures when the pool size
exceeded the connection reset channel buffer size.
Rodolfo Carvalho [Tue, 17 Mar 2020 10:41:45 +0000 (10:41 +0000)]
testing: use "exit code" in documentation consistently
The documentation for m.Run says it returns an "exit code" to pass to
os.Exit. The argument to os.Exit is named "code".
While "exit code", "exit status" and "exit status code" are all valid ways
to refer to the same concept, prefer to stick to one form for consistency
and to avoid confusing users.
Change-Id: If76ee3fab5cc99c79e05ac1a4e413790a9c93d60
GitHub-Last-Rev: 85a081d2f03c2cf9e8e519916986c59c86aebf57
GitHub-Pull-Request: golang/go#37899
Reviewed-on: https://go-review.googlesource.com/c/go/+/223778 Reviewed-by: Gabriel Aszalos <gabriel.aszalos@gmail.com> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Gabriel Aszalos <gabriel.aszalos@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Bryan C. Mills [Tue, 17 Mar 2020 14:50:40 +0000 (10:50 -0400)]
cmd/go: replace TestCgoDependsOnSyscall with a simpler script test
The existing test attempted to remove '_race' binaries from
GOROOT/pkg, which could not only fail if GOROOT is read-only, but also
interfere with other tests run in parallel.
Updates #30316
Updates #37573
Updates #17751
Change-Id: Id7e2286ab67f8333baf4d52244b7f4476aa93a46
Reviewed-on: https://go-review.googlesource.com/c/go/+/223745
Run-TryBot: Bryan C. Mills <bcmills@google.com> Reviewed-by: Jay Conrod <jayconrod@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Changkun Ou [Sun, 16 Feb 2020 13:42:29 +0000 (14:42 +0100)]
testing: do not require os.Exit in TestMain
If TestMain reports a wrong exit code to os.Exit, the test will be
exited with exist code inconsist with test results.
This CL eliminates the requirement of calling os.Exit in TestMain.
Now, m.Run records the execution status of its test, the outer
main func will call os.Exit with that exit code if TestMain does
not call os.Exit.
If TestMain does not call m.Run, the outer main func remain calls
os.Exit(0) as before.
Fixes #34129
Change-Id: I9598023e03b0a6260f0217f34df41c231c7d6489
Reviewed-on: https://go-review.googlesource.com/c/go/+/219639 Reviewed-by: Ian Lance Taylor <iant@golang.org>
Ian Lance Taylor [Fri, 24 Jan 2020 04:38:20 +0000 (20:38 -0800)]
runtime: add goroutines returned by poller to local run queue
In Go 1.13, when the network poller found a list of ready goroutines,
they were added to the global run queue. The timer goroutine would
typically sleep in a futex with a timeout, and when the timeout
expired the timer goroutine would either be handed off to an idle P
or added to the global run queue. The effect was that on a busy system
with no idle P's goroutines waiting for timeouts and goroutines waiting
for the network would start at the same priority.
That changed on tip with the new timer code. Now timer functions are
invoked directly from a P, and it happens that the functions used
by time.Sleep and time.After and time.Ticker add the newly ready
goroutines to the local run queue. When a P looks for work it will
prefer goroutines on the local run queue; in fact it will only
occasionally look at the global run queue, and even when it does it
will just pull one goroutine off. So on a busy system with both active
timers and active network connections the system can noticeably prefer
to run goroutines waiting for timers rather than goroutines waiting
for the network.
This CL undoes that change by, when possible, adding goroutines
waiting for the network to the local run queue of the P that checked.
This doesn't affect network poller checks done by sysmon, but it
does affect network poller checks done as each P enters the scheduler.
This CL also makes injecting a list into either the local or global run
queue more efficient, using bulk operations rather than individual ones.
Change-Id: I85a66ad74e4fc3b458256fb7ab395d06f0d2ffac
Reviewed-on: https://go-review.googlesource.com/c/go/+/216198 Reviewed-by: Michael Knyszek <mknyszek@google.com>
Paschalis Tsilias [Mon, 3 Feb 2020 08:47:41 +0000 (10:47 +0200)]
time: fix time.Before to reuse t.sec(), u.sec()
Fixes #36987
Change-Id: I91ea1a42f75302de5256a22d382ab7f1b307a498
Reviewed-on: https://go-review.googlesource.com/c/go/+/217360
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Rework instruction generation so that multiple instructions are generated
from a single obj.Prog, rather than the current approach where obj.Progs
are rewritten. This allows the original obj.Prog to remain intact, before
being converted into an architecture specific instruction form.
This simplifies the code and removes a level of indirection that results
from trying to manipulate obj.Prog.To/obj.Prog.From into forms that match
the instruction encoding. Furthermore, the errors reported make more sense
since it matches up with the actual assembly that was parsed.
Note that the CALL/JMP/JALR type sequences have not yet been migrated to
this framework and will likely be converted at a later time.
Joel Sing [Wed, 5 Feb 2020 17:47:52 +0000 (04:47 +1100)]
cmd/internal/obj/riscv: store relocation details in obj.Prog.RestArgs
Store the relocation offset and symbol in obj.Prog.RestArgs, rather than
overloading obj.Prog.From and having to deal with invalid offsets
potentially existing when the instruction is encoded.
Robert Kuska [Fri, 13 Mar 2020 18:21:19 +0000 (19:21 +0100)]
net/http: use atomicBool for inShutdown
This removes the TODO leftover by replacing the original int32 for
atomicBool that mimicks atomic operations for boolean.
Change-Id: I1b2cac0c9573c890c7315e9906ce6bfccee3d770
Reviewed-on: https://go-review.googlesource.com/c/go/+/223357
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Brian Kessler [Fri, 12 Apr 2019 04:54:18 +0000 (22:54 -0600)]
math/cmplx: implement Payne-Hanek range reduction
Tan has poles along the real axis. In order to accurately calculate
the value near these poles, a range reduction by Pi is performed and
the result calculated via a Taylor series. The prior implementation
of range reduction used Cody-Waite range reduction in three parts.
This fails when x is too large to accurately calculate the partial
products in the summation accurately. Above this threshold, Payne-Hanek
range reduction using a multiple precision value of 1/Pi is required.
Additionally, the threshold used in math/trig_reduce.go for Payne-Hanek
range reduction was not set conservatively enough. The prior threshold
ensured that catastrophic failure did not occur where the argument x
would not actually be reduced below Pi/4. However, errors in reduction
begin to occur at values much lower when z = ((x - y*PI4A) - y*PI4B) - y*PI4C
is not exact because y*PI4A cannot be exactly represented as a float64.
reduceThreshold is lowered to the proper value.
Fixes #31566
Change-Id: I0f39a4171a5be44f64305f18dc57f6c29f19dba7
Reviewed-on: https://go-review.googlesource.com/c/go/+/172838 Reviewed-by: Rob Pike <r@golang.org>
Bryan C. Mills [Wed, 11 Mar 2020 19:14:38 +0000 (15:14 -0400)]
context: deflake time-sensitive tests
Many of tests in this package assumed reasonable scheduling latency.
Unfortunately, scheduling latency on builders and CI systems is not
always reasonable.
Rather than expecting that a timeout is detected within a fixed short
interval, we can use (*testing.T).Deadline to portably scale the time
we're willing to wait to something appropriate to the builder.
Some of the tests also included arbitrary-duration sleeps, which are
no longer needed after CL 196521; we can remove those instead of
extending them.
Promptness of timeouts is also an important property, but testing that
property is better suited to benchmarks than to tests proper: unlike
tests, we generally expect benchmarks to be run in a quiet,
low-contention environment.
Fixes #13956
Change-Id: I0797e2267fb778c8ad94add56d797de9e2c885e5
Reviewed-on: https://go-review.googlesource.com/c/go/+/223019
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Ian Lance Taylor [Thu, 12 Mar 2020 04:51:09 +0000 (21:51 -0700)]
runtime: don't crash on mlock failure
Instead, note that mlock has failed, start trying the mitigation of
touching the signal stack before sending a preemption signal, and,
if the program crashes, mention the possible problem and a wiki page
describing the issue (https://golang.org/wiki/LinuxKernelSignalVectorBug).
Tested on a kernel in the buggy version range, but with the patch,
by using `ulimit -l 0`.
Fixes #37436
Change-Id: I072aadb2101496dffd655e442fa5c367dad46ce8
Reviewed-on: https://go-review.googlesource.com/c/go/+/223121
Run-TryBot: Ian Lance Taylor <iant@golang.org> Reviewed-by: Austin Clements <austin@google.com> Reviewed-by: Keith Randall <khr@golang.org>
Bryan C. Mills [Fri, 13 Mar 2020 16:51:09 +0000 (12:51 -0400)]
cmd/go/internal/modload: suppress the 'go mod init' hint for GOROOT more aggressively
Previously, we suppressed a `to create a module there, run: … go mod
init' warning only if the config file itself (such as .git/config) was
found in GOROOT. However, our release tarballs don't include the
.git/config, so that case was not encountered, and the warning could
occur based on a config file found in some parent directory (outside
of GOROOT entirely).
Instead, skip the directory walk completely if the working directory
is anywhere in GOROOT.
Fixes #34191
Change-Id: I9f774901bfbb53b700407c4882f37d6339d023fe
Reviewed-on: https://go-review.googlesource.com/c/go/+/223340
Run-TryBot: Bryan C. Mills <bcmills@google.com> Reviewed-by: Jay Conrod <jayconrod@google.com>
shortcircuitBlock contained a loop to handle blocks like
b: <- p q
v = Phi true false
If v -> t u
in a single execution.
This change makes shortcircuitBlock do it in two instead,
one for each constant phi arg.
Motivation: Upcoming changes will expand the range of
blocks that the shortcircuit pass can handle.
Those changes need to understand what the CFG
will look like after the rewrite in shortcircuitBlock.
Making shortcircuitBlock do only a single CFG
modification at a time significantly simplifies that code.
In theory, this is less efficient, but not measurably so.
There is minor, unimportant churn in the generated code.
This commit extends the -spectre flag to cmd/asm and adds
a new Spectre mitigation mode "ret", which enables the use
of retpolines.
Retpolines prevent speculation about the target of an indirect
jump or call and are described in more detail here:
https://support.google.com/faqs/answer/7625886
Change-Id: I4f2cb982fa94e44d91e49bd98974fd125619c93a
Reviewed-on: https://go-review.googlesource.com/c/go/+/222661 Reviewed-by: Keith Randall <khr@golang.org>
Russ Cox [Mon, 6 Jan 2020 15:31:39 +0000 (10:31 -0500)]
cmd/compile: add spectre mitigation mode enabled by -spectre
This commit adds a new cmd/compile flag -spectre,
which accepts a comma-separated list of possible
Spectre mitigations to apply, or the empty string (none),
or "all". The only known mitigation right now is "index",
which uses conditional moves to ensure that x86-64 CPUs
do not speculate past index bounds checks.
Speculating past index bounds checks may be problematic
on systems running privileged servers that accept requests
from untrusted users who can execute their own programs
on the same machine. (And some more constraints that
make it even more unlikely in practice.)
The cases this protects against are analogous to the ones
Microsoft explains in the "Array out of bounds load/store feeding ..."
sections here:
https://docs.microsoft.com/en-us/cpp/security/developer-guidance-speculative-execution?view=vs-2019#array-out-of-bounds-load-feeding-an-indirect-branch
Change-Id: Ib7532d7e12466b17e04c4e2075c2a456dc98f610
Reviewed-on: https://go-review.googlesource.com/c/go/+/222660 Reviewed-by: Keith Randall <khr@golang.org>
zikaeroh [Thu, 12 Mar 2020 00:02:50 +0000 (17:02 -0700)]
cmd/cover: skip function declarations with blank names
Function declarations with blank ("_") names do not introduce a binding,
and therefore cannot be referenced or executed (in fact, they do not
make it into the final compiled binary at all). As such, counters
defined while annotating their bodies will always be zero.
These types of functions are commonly used to create compile-time
checks (e.g., stringer) which are not expected to be executed.
Skip over these functions when annotating a file, preventing the unused
counters from being generated and appearing as uncovered lines in
coverage reports.
Fixes #36264
Change-Id: I6b516cf43c430a6248d68d5f483a3902253fbdab
Reviewed-on: https://go-review.googlesource.com/c/go/+/223117 Reviewed-by: Bryan C. Mills <bcmills@google.com>
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Lynn Boger [Tue, 10 Mar 2020 14:08:02 +0000 (10:08 -0400)]
cmd/internal/obj/ppc64: clean up some opcodes
This does some clean up of the ppc64 opcodes to remove names
from the opcode list that don't actually assemble. At one time
names were added to this list to represent opcode "classes" to
organize other opcodes that have the same set of operand
combinations. Since this is not documented, it is confusing as
to which opcodes can be used in an asm file and which can't, and
which opcodes should be supported in the disassembler. It is
clearer for the user if the list of Go opcodes are all opcodes
that can be assembled with names that match the ppc64 opcode
where possible.
I found this when trying to use Go opcode XXLAND in an asm file
which seems like it should map to ppc64 xxland but when used it
gets this error:
go tool asm test_xxland.s
asm: bad r/r, r/r/r or r/r/r/r opcode XXLAND
asm: assembly failed
This change removes the opcodes that are only used for opcode
"classes" and fixes the case statement where they are referenced.
This also fixes XXLAND and XXPERM which are opcodes that should
assemble to their corresponding ppc64 opcode but do not.
Ian Lance Taylor [Thu, 12 Mar 2020 05:03:50 +0000 (22:03 -0700)]
runtime: leave cleantimers early if G is being preempted
The cleantimers can run for a while in some unlikely cases.
If the GC is trying to preempt the G, it is forced to wait as the
G is holding timersLock. To avoid introducing a GC delay,
return from cleantimers if the G has a preemption request.
Fixes #37779
Change-Id: Id9a567f991e26668e2292eefc39e2edc56efa4e0
Reviewed-on: https://go-review.googlesource.com/c/go/+/223122
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Michael Knyszek <mknyszek@google.com> Reviewed-by: Cherry Zhang <cherryyz@google.com>
David Chase [Wed, 11 Mar 2020 17:36:42 +0000 (13:36 -0400)]
cmd/objdump: guard against out-of-range lines from directives.
//line bogo.go:9999999 will cause 'go tool objdump' to crash
unless bogo.go has that many lines. Guard the array index
and return innocuous values (nil, nil) from the file cache.
Walt Della [Sun, 2 Feb 2020 05:58:37 +0000 (21:58 -0800)]
cmd/go: improve pseudo-version timestamp error
The previous "invalid pseudo-version: does not match version-control
timestamp" error message used a different timestamp format than the
format used in go.mod and go.sum. For cut-and-paste-ability this patch
makes the two consistent.
Fixes #36974
Change-Id: I21f344ab9898cc584c0bcf4a75d74275a703c650
Reviewed-on: https://go-review.googlesource.com/c/go/+/217437
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com> Reviewed-by: Michael Matloob <matloob@golang.org> Reviewed-by: Jay Conrod <jayconrod@google.com>
Rob Pike [Thu, 12 Mar 2020 00:25:39 +0000 (11:25 +1100)]
io: add a comment about how to turn a Reader into ByteReader
Offered as an alternative to CL 221380, which was more
tutorial than necessary.
Update #37344
Change-Id: Ide673b0b97983c2c2319a9311dc3d0a10567e6c4
Reviewed-on: https://go-review.googlesource.com/c/go/+/223097 Reviewed-by: Ian Lance Taylor <iant@golang.org>
Robert Griesemer [Thu, 5 Mar 2020 22:14:54 +0000 (14:14 -0800)]
cmd/compile/internal/syntax: various cleanups following CL 221603
1) Introduced setLit method to uniformly set the scanner state for
literals instead of directly manipulating the scanner fields.
2) Use a local variable 'ok' to track validity of literals instead
of relying on the side-effect of error reporters setting s.bad.
More code but clearer because it is local and explicit.
3) s/litname/baseName/ and use this function uniformly, also for
escapes. Consequently we now report always "hexadecimal" and
not "hex" (in the case of invalid escapes).
4) Added TestDirectives verifying that we get the correct directive
string (even if that string contains '%').
Verified that lines/s parsing performance is unchanged by comparing
go test -run StdLib -fast -skip "syntax/(scanner|scanner_test)\.go"
before and after (no relevant difference).
Change-Id: I143e4648fdaa31d1c365fb794a1cae4bc1c3f5ba
Reviewed-on: https://go-review.googlesource.com/c/go/+/222258
Run-TryBot: Robert Griesemer <gri@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Matthew Dempsky <mdempsky@google.com>
Jay Conrod [Tue, 25 Feb 2020 16:00:08 +0000 (11:00 -0500)]
cmd/go: extract module zip files in place
Previously, we extracted module zip files to temporary directories
with random names, then renamed them to their final locations. This
failed with ERROR_ACCESS_DENIED on Windows if any file in the
temporary was open. Antivirus programs did this occasionally. Retrying
the rename did not work (CL 220978).
With this change, we extract module zip files in place. We create a
.partial file alongside the .lock file to indicate a directory is not
fully populated, and we delete this at the end of the process.
Updates #36568
Change-Id: I75c09df879a602841f3459322c021896292b2fdb
Reviewed-on: https://go-review.googlesource.com/c/go/+/221157
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com> Reviewed-by: Michael Matloob <matloob@golang.org>
Jay Conrod [Fri, 28 Feb 2020 21:31:19 +0000 (16:31 -0500)]
cmd/go: make module zip extraction more robust
Currently, we extract module zip files to temporary directories, then
atomically rename them into place. On Windows, this can fail with
ERROR_ACCESS_DENIED if another process (antivirus) has files open
before the rename. In CL 220978, we repeated the rename operation in a
loop over 500 ms, but this didn't solve the problem for everyone.
A better solution will extract module zip files to their permanent
locations in the cache and will keep a ".partial" marker file,
indicating when a module hasn't been fully extracted (CL 221157).
This approach is not safe if current versions of Go access the module
cache concurrently, since the module directory is detected with a
single os.Stat.
In the interim, this CL makes two changes:
1. Flaky file system operations are repeated over 2000 ms to reduce
the chance of this error occurring.
2. cmd/go will now check for .partial files created by future
versions. If a .partial file is found, it will lock the lock file,
then remove the .partial file and directory if needed.
After some time has passed and Go versions lacking this CL are no
longer supported, we can start extracting module zip files in place.
Updates #36568
Change-Id: I467ee11aa59a90b63cf0e3e761c4fec89d57d3b6
Reviewed-on: https://go-review.googlesource.com/c/go/+/221820
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Michael Matloob <matloob@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
Russ Cox [Wed, 11 Mar 2020 04:02:04 +0000 (00:02 -0400)]
cmd/compile: fix buggy AMD64 rewrite from CL 213058
CL 213058's "bonus optimization I noticed while working on this"
turns out to be buggy. It would be correct for CMP, but not TEST.
Fix it to use TEST semantics instead.
This was breaking compilation with the upcoming Spectre mode.
Change-Id: If2d4c3798ed182f35f0244febe74e68c61e4c61b
Reviewed-on: https://go-review.googlesource.com/c/go/+/222853 Reviewed-by: Keith Randall <khr@golang.org>
Bryan C. Mills [Wed, 11 Mar 2020 13:39:54 +0000 (09:39 -0400)]
runtime: skip TestSignalIgnoreSIGTRAP on known-flaky OpenBSD builders
This test is flaky, and the cause is suspected to be an OpenBSD kernel bug.
Since there is no obvious workaround on the Go side, skip the test on
builders whose versions are known to be affected.
Fixes #17496
Change-Id: Ifa70061eb429e1d949f0fa8a9e25d177afc5c488
Reviewed-on: https://go-review.googlesource.com/c/go/+/222856
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
This change forces CFGs to take the full width of their column
and allows them to be as tall as necessary.
In my (recent) experience, this makes them far less likely to
be cropped, which makes them much more useful.
On rare occasions, this can lead to gigantic CFGs,
but if you've bothered to explicitly request a CFG,
this is still better than an irrevocably truncated CFG.
erifan01 [Sun, 23 Dec 2018 02:15:44 +0000 (02:15 +0000)]
strings, bytes: improve IndexAny and LastIndexAny performance
For the case of a pattern containing multi-byte rune, the time complexity of the
previous algorithm is O(nm), and if both input arguments are long, the search
performance will be poor. This CL improves the searching performance for these
cases by using IndexRune, which is mainly implemented with IndexByte and Index.
As IndexByte and Index are specially optimized with some powerful instructions
for short patterns (an UTF8 rune is 1 to 4 bytes), so they can help to reduce the
runtime complexity of IndexAny and LastIndexAny.
Another optimization method is using hash table, however, the actual test results
show that using indexrune is better, and the space complexity is lower.
There are two fast paths in IndexAny and LastIndexAny for cases where the length
of the input arguements are 1, and their locations are not exactly the same, which
is determined based on the actual test results.
Change-Id: Ie05e306f8b184b989701868cb161ce8b3f18203b
Reviewed-on: https://go-review.googlesource.com/c/go/+/156998
Run-TryBot: eric fang <eric.fang@arm.com>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Dan Scales [Thu, 5 Mar 2020 20:46:04 +0000 (12:46 -0800)]
runtime: fix problem with repeated panic/recover/re-panics and open-coded defers
In the open-code defer implementation, we add defer struct entries to the defer
chain on-the-fly at panic time to represent stack frames that contain open-coded
defers. This allows us to process non-open-coded and open-coded defers in the
correct order. Also, we need somewhere to be able to store the 'started' state of
open-coded defers. However, if a recover succeeds, defers will now be processed
inline again (unless another panic happens). Any defer entry representing a frame
with open-coded defers will become stale once we run the corresponding defers
inline and exit the associated stack frame. So, we need to remove all entries for
open-coded defers at recover time.
The current code was only removing the top-most open-coded defer from the defer
chain during recovery. However, with recursive functions that do repeated
panic-recover-repanic, multiple stale entries can accumulate on the chain. So, we
just adjust the loop to process the entire chain. Since this is at panic/recover
case, it is fine to scan through the entire chain (which should usually have few
elements in it, since most defers are open-coded).
The added test fails with a SEGV without the fix, because it tries to run a stale
open-code defer entry (and the stack has changed).
Fixes #37664.
Change-Id: I8e3da5d610b5e607411451b66881dea887f7484d
Reviewed-on: https://go-review.googlesource.com/c/go/+/222420
Run-TryBot: Dan Scales <danscales@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Keith Randall <khr@golang.org>
Than McIntosh [Fri, 7 Feb 2020 19:00:26 +0000 (14:00 -0500)]
cmd/link: fix for package name attr testpoint in dwarf_test.go
Tighten up a testpoint that looks for the compile unit
DW_AT_go_package_name attribute. The linker code that injects this
attribute was accidentally broken on the dev.link branch, but in a way
that wasn't detected by the test (attr was generated, but always with
an empty string). The new test will fail if the attr is an empty
string, or if we can't find the attribute for the runtime package.
Change-Id: I8b065e7eb3486646364d0eaf48a73db6acffbd18
Reviewed-on: https://go-review.googlesource.com/c/go/+/218483
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Jeremy Faller <jeremy@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com>
Bryan C. Mills [Thu, 5 Mar 2020 21:17:45 +0000 (16:17 -0500)]
cmd/go/internal/lockedfile: use a retry loop to suppress EDEADLK on AIX and Solaris
AIX, Solaris, and Illumos all appear to implement fcntl deadlock
detection at the granularity of processes. However, we are acquiring
and releasing file locks on individual goroutines running
concurrently: our locking occurs at a much finer granularity. As a
result, these platforms occasionally fail with EDEADLK errors, when
they detect locks that would be _misordered_ in a single-threaded
program but are safely _unordered_ in a multi-threaded context.
To work around the spurious errors, we treat EDEADLK as always
spurious, and retry the failing system call with a bounded exponential
backoff. This approach may introduce substantial latency since we no
longer benefit from kernel-scheduled wakeups in case of collisions,
but high-latency operations seem better than spurious failures.
Updates #33974
Updates #35618
Fixes #32817
Change-Id: I58b2c6a0f143bce55d6460fd4ddc3db83577ada7
Reviewed-on: https://go-review.googlesource.com/c/go/+/222277 Reviewed-by: Jay Conrod <jayconrod@google.com>
Keith Randall [Fri, 6 Mar 2020 22:01:26 +0000 (14:01 -0800)]
runtime: make typehash match compiler generated hashes exactly
If typehash (used by reflect) does not match the built-in map's hash,
then problems occur. If a map is built using reflect, and then
assigned to a variable of map type, the hash function can change. That
causes very bad things.
This issue is rare. MapOf consults a cache of all types that occur in
the binary before making a new one. To make a true new map type (with
a hash function derived from typehash) that map type must not occur in
the binary anywhere. But to cause the bug, we need a variable of that
type in order to assign to it. The only way to make that work is to
use a named map type for the variable, so it is distinct from the
unnamed version that MapOf looks for.
smasher164 [Thu, 27 Feb 2020 08:44:13 +0000 (03:44 -0500)]
cmd/vet: add ifaceassert and stringintconv checks
This change re-vendors x/tools to add the ifaceassert and stringintconv
checks to cmd/vet.
Fixes #32479.
Updates #4483.
Change-Id: I6bd30b0a3278592dfab4bd247036404ddaff09e4
Reviewed-on: https://go-review.googlesource.com/c/go/+/221339
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
Keith Randall [Thu, 30 Jan 2020 18:17:01 +0000 (10:17 -0800)]
cmd/compile: insert complicated x86 addressing modes as a separate pass
Use a separate compiler pass to introduce complicated x86 addressing
modes. Loads in the normal architecture rules (for x86 and all other
platforms) can have constant offsets (AuxInt values) and symbols (Aux
values), but no more.
The complex addressing modes (x+y, x+2*y, etc.) are introduced in a
separate pass that combines loads with LEAQx ops.
Organizing rewrites this way simplifies the number of rewrites
required, as there are lots of different rule orderings that have to
be specified to ensure these complex addressing modes are always found
if they are possible.
smasher164 [Thu, 27 Feb 2020 07:42:28 +0000 (02:42 -0500)]
std,cmd: update x/net and github.com/google/pprof
Re-vendor x/net/dns/dnsmessage, x/net/route, and github.com/google/pprof
(commit 1ebb73c). The updated dependencies fix the string(int)
conversions, in preparation for the vet warning.
Updates #32479.
Change-Id: I023a4e30415d060f8b403b9943fe911f6d19f2e9
Reviewed-on: https://go-review.googlesource.com/c/go/+/221337
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
cmd/compile: use only bit patterns in isNonNegative
CL 212777 added a check to isNonNegative
to return true for unsigned values.
However, the SSA backend isn't type safe
enough for that to be sound.
The other checks in isNonNegative
look only at the pattern of bits.
Remove the type-based check.
Keith Randall [Mon, 9 Mar 2020 19:46:45 +0000 (12:46 -0700)]
cmd/compile: remove -largemodel flag from docs
It does nothing (it can't even be parsed).
Change-Id: I29abdddea1955d2ad93a97696f6542fa47cdb954
Reviewed-on: https://go-review.googlesource.com/c/go/+/222672 Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Neven Sajko [Sat, 7 Mar 2020 17:23:20 +0000 (17:23 +0000)]
internal/xcoff: fix wrong bit masking comparisons
I do not know much about xcoff, but this was probably the intended
behavior. (The comparison is tautologically false, as is.)
Also note: does any other code even depend on the changed code existing?
Maybe it should just be removed, as I did not find any uses of fields
that are written to if the branch condition tests true.
Change-Id: I1f23d33764df40e87f3e64460d63f6efc51a2a78
GitHub-Last-Rev: 268909130fd7fb3993fcf7004143ec48dbfe5e2a
GitHub-Pull-Request: golang/go#37733
Reviewed-on: https://go-review.googlesource.com/c/go/+/222478
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Clément Chigot <clement.chigot%atos.net@gtempaccount.com> Reviewed-by: Ian Lance Taylor <iant@golang.org>