]> Cypherpunks repositories - gostls13.git/log
gostls13.git
17 months agomath/rand, math/rand/v2: use ChaCha8 for global rand
Russ Cox [Sun, 6 Aug 2023 03:26:28 +0000 (13:26 +1000)]
math/rand, math/rand/v2: use ChaCha8 for global rand

Move ChaCha8 code into internal/chacha8rand and use it to implement
runtime.rand, which is used for the unseeded global source for
both math/rand and math/rand/v2. This also affects the calculation of
the start point for iteration over very very large maps (when the
32-bit fastrand is not big enough).

The benefit is that misuse of the global random number generators
in math/rand and math/rand/v2 in contexts where non-predictable
randomness is important for security reasons is no longer a
security problem, removing a common mistake among programmers
who are unaware of the different kinds of randomness.

The cost is an extra 304 bytes per thread stored in the m struct
plus 2-3ns more per random uint64 due to the more sophisticated
algorithm. Using PCG looks like it would cost about the same,
although I haven't benchmarked that.

Before this, the math/rand and math/rand/v2 global generator
was wyrand (https://github.com/wangyi-fudan/wyhash).
For math/rand, using wyrand instead of the Mitchell/Reeds/Thompson
ALFG was justifiable, since the latter was not any better.
But for math/rand/v2, the global generator really should be
at least as good as one of the well-studied, specific algorithms
provided directly by the package, and it's not.

(Wyrand is still reasonable for scheduling and cache decisions.)

Good randomness does have a cost: about twice wyrand.

Also rationalize the various runtime rand references.

goos: linux
goarch: amd64
pkg: math/rand/v2
cpu: AMD Ryzen 9 7950X 16-Core Processor
                        │ bbb48afeb7.amd64 │           5cf807d1ea.amd64           │
                        │      sec/op      │    sec/op     vs base                │
ChaCha8-32                     1.862n ± 2%    1.861n ± 2%        ~ (p=0.825 n=20)
PCG_DXSM-32                    1.471n ± 1%    1.460n ± 2%        ~ (p=0.153 n=20)
SourceUint64-32                1.636n ± 2%    1.582n ± 1%   -3.30% (p=0.000 n=20)
GlobalInt64-32                 2.087n ± 1%    3.663n ± 1%  +75.54% (p=0.000 n=20)
GlobalInt64Parallel-32        0.1042n ± 1%   0.2026n ± 1%  +94.48% (p=0.000 n=20)
GlobalUint64-32                2.263n ± 2%    3.724n ± 1%  +64.57% (p=0.000 n=20)
GlobalUint64Parallel-32       0.1019n ± 1%   0.1973n ± 1%  +93.67% (p=0.000 n=20)
Int64-32                       1.771n ± 1%    1.774n ± 1%        ~ (p=0.449 n=20)
Uint64-32                      1.863n ± 2%    1.866n ± 1%        ~ (p=0.364 n=20)
GlobalIntN1000-32              3.134n ± 3%    4.730n ± 2%  +50.95% (p=0.000 n=20)
IntN1000-32                    2.489n ± 1%    2.489n ± 1%        ~ (p=0.683 n=20)
Int64N1000-32                  2.521n ± 1%    2.516n ± 1%        ~ (p=0.394 n=20)
Int64N1e8-32                   2.479n ± 1%    2.478n ± 2%        ~ (p=0.743 n=20)
Int64N1e9-32                   2.530n ± 2%    2.514n ± 2%        ~ (p=0.193 n=20)
Int64N2e9-32                   2.501n ± 1%    2.494n ± 1%        ~ (p=0.616 n=20)
Int64N1e18-32                  3.227n ± 1%    3.205n ± 1%        ~ (p=0.101 n=20)
Int64N2e18-32                  3.647n ± 1%    3.599n ± 1%        ~ (p=0.019 n=20)
Int64N4e18-32                  5.135n ± 1%    5.069n ± 2%        ~ (p=0.034 n=20)
Int32N1000-32                  2.657n ± 1%    2.637n ± 1%        ~ (p=0.180 n=20)
Int32N1e8-32                   2.636n ± 1%    2.636n ± 1%        ~ (p=0.763 n=20)
Int32N1e9-32                   2.660n ± 2%    2.638n ± 1%        ~ (p=0.358 n=20)
Int32N2e9-32                   2.662n ± 2%    2.618n ± 2%        ~ (p=0.064 n=20)
Float32-32                     2.272n ± 2%    2.239n ± 2%        ~ (p=0.194 n=20)
Float64-32                     2.272n ± 1%    2.286n ± 2%        ~ (p=0.763 n=20)
ExpFloat64-32                  3.762n ± 1%    3.744n ± 1%        ~ (p=0.171 n=20)
NormFloat64-32                 3.706n ± 1%    3.655n ± 2%        ~ (p=0.066 n=20)
Perm3-32                       32.93n ± 3%    34.62n ± 1%   +5.13% (p=0.000 n=20)
Perm30-32                      202.9n ± 1%    204.0n ± 1%        ~ (p=0.482 n=20)
Perm30ViaShuffle-32            115.0n ± 1%    114.9n ± 1%        ~ (p=0.358 n=20)
ShuffleOverhead-32             112.8n ± 1%    112.7n ± 1%        ~ (p=0.692 n=20)
Concurrent-32                  2.107n ± 0%    3.725n ± 1%  +76.75% (p=0.000 n=20)

goos: darwin
goarch: arm64
pkg: math/rand/v2
                       │ bbb48afeb7.arm64 │           5cf807d1ea.arm64            │
                       │      sec/op      │    sec/op     vs base                 │
ChaCha8-8                     2.480n ± 0%    2.429n ± 0%    -2.04% (p=0.000 n=20)
PCG_DXSM-8                    2.531n ± 0%    2.530n ± 0%         ~ (p=0.877 n=20)
SourceUint64-8                2.534n ± 0%    2.533n ± 0%         ~ (p=0.732 n=20)
GlobalInt64-8                 2.172n ± 1%    4.794n ± 0%  +120.67% (p=0.000 n=20)
GlobalInt64Parallel-8        0.4320n ± 0%   0.9605n ± 0%  +122.32% (p=0.000 n=20)
GlobalUint64-8                2.182n ± 0%    4.770n ± 0%  +118.58% (p=0.000 n=20)
GlobalUint64Parallel-8       0.4307n ± 0%   0.9583n ± 0%  +122.51% (p=0.000 n=20)
Int64-8                       4.107n ± 0%    4.104n ± 0%         ~ (p=0.416 n=20)
Uint64-8                      4.080n ± 0%    4.080n ± 0%         ~ (p=0.052 n=20)
GlobalIntN1000-8              2.814n ± 2%    5.643n ± 0%  +100.50% (p=0.000 n=20)
IntN1000-8                    4.141n ± 0%    4.139n ± 0%         ~ (p=0.140 n=20)
Int64N1000-8                  4.140n ± 0%    4.140n ± 0%         ~ (p=0.313 n=20)
Int64N1e8-8                   4.140n ± 0%    4.139n ± 0%         ~ (p=0.103 n=20)
Int64N1e9-8                   4.139n ± 0%    4.140n ± 0%         ~ (p=0.761 n=20)
Int64N2e9-8                   4.140n ± 0%    4.140n ± 0%         ~ (p=0.636 n=20)
Int64N1e18-8                  5.266n ± 0%    5.326n ± 1%    +1.14% (p=0.001 n=20)
Int64N2e18-8                  6.052n ± 0%    6.167n ± 0%    +1.90% (p=0.000 n=20)
Int64N4e18-8                  8.826n ± 0%    9.051n ± 0%    +2.55% (p=0.000 n=20)
Int32N1000-8                  4.127n ± 0%    4.132n ± 0%    +0.12% (p=0.000 n=20)
Int32N1e8-8                   4.126n ± 0%    4.131n ± 0%    +0.12% (p=0.000 n=20)
Int32N1e9-8                   4.127n ± 0%    4.132n ± 0%    +0.12% (p=0.000 n=20)
Int32N2e9-8                   4.132n ± 0%    4.131n ± 0%         ~ (p=0.017 n=20)
Float32-8                     4.109n ± 0%    4.105n ± 0%         ~ (p=0.379 n=20)
Float64-8                     4.107n ± 0%    4.106n ± 0%         ~ (p=0.867 n=20)
ExpFloat64-8                  5.339n ± 0%    5.383n ± 0%    +0.82% (p=0.000 n=20)
NormFloat64-8                 5.735n ± 0%    5.737n ± 1%         ~ (p=0.856 n=20)
Perm3-8                       26.65n ± 0%    26.80n ± 1%    +0.58% (p=0.000 n=20)
Perm30-8                      194.8n ± 1%    197.0n ± 0%    +1.18% (p=0.000 n=20)
Perm30ViaShuffle-8            156.6n ± 0%    157.6n ± 1%    +0.61% (p=0.000 n=20)
ShuffleOverhead-8             124.9n ± 0%    125.5n ± 0%    +0.52% (p=0.000 n=20)
Concurrent-8                  2.434n ± 3%    5.066n ± 0%  +108.09% (p=0.000 n=20)

goos: linux
goarch: 386
pkg: math/rand/v2
cpu: AMD Ryzen 9 7950X 16-Core Processor
                        │ bbb48afeb7.386 │            5cf807d1ea.386             │
                        │     sec/op     │    sec/op     vs base                 │
ChaCha8-32                  11.295n ± 1%    4.748n ± 2%   -57.96% (p=0.000 n=20)
PCG_DXSM-32                  7.693n ± 1%    7.738n ± 2%         ~ (p=0.542 n=20)
SourceUint64-32              7.658n ± 2%    7.622n ± 2%         ~ (p=0.344 n=20)
GlobalInt64-32               3.473n ± 2%    7.526n ± 2%  +116.73% (p=0.000 n=20)
GlobalInt64Parallel-32      0.3198n ± 0%   0.5444n ± 0%   +70.22% (p=0.000 n=20)
GlobalUint64-32              3.612n ± 0%    7.575n ± 1%  +109.69% (p=0.000 n=20)
GlobalUint64Parallel-32     0.3168n ± 0%   0.5403n ± 0%   +70.51% (p=0.000 n=20)
Int64-32                     7.673n ± 2%    7.789n ± 1%         ~ (p=0.122 n=20)
Uint64-32                    7.773n ± 1%    7.827n ± 2%         ~ (p=0.920 n=20)
GlobalIntN1000-32            6.268n ± 1%    9.581n ± 1%   +52.87% (p=0.000 n=20)
IntN1000-32                  10.33n ± 2%    10.45n ± 1%         ~ (p=0.233 n=20)
Int64N1000-32                10.98n ± 2%    11.01n ± 1%         ~ (p=0.401 n=20)
Int64N1e8-32                 11.19n ± 2%    10.97n ± 1%         ~ (p=0.033 n=20)
Int64N1e9-32                 11.06n ± 1%    11.08n ± 1%         ~ (p=0.498 n=20)
Int64N2e9-32                 11.10n ± 1%    11.01n ± 2%         ~ (p=0.995 n=20)
Int64N1e18-32                15.23n ± 2%    15.04n ± 1%         ~ (p=0.973 n=20)
Int64N2e18-32                15.89n ± 1%    15.85n ± 1%         ~ (p=0.409 n=20)
Int64N4e18-32                18.96n ± 2%    19.34n ± 2%         ~ (p=0.048 n=20)
Int32N1000-32                10.46n ± 2%    10.44n ± 2%         ~ (p=0.480 n=20)
Int32N1e8-32                 10.46n ± 2%    10.49n ± 2%         ~ (p=0.951 n=20)
Int32N1e9-32                 10.28n ± 2%    10.26n ± 1%         ~ (p=0.431 n=20)
Int32N2e9-32                 10.50n ± 2%    10.44n ± 2%         ~ (p=0.249 n=20)
Float32-32                   13.80n ± 2%    13.80n ± 2%         ~ (p=0.751 n=20)
Float64-32                   23.55n ± 2%    23.87n ± 0%         ~ (p=0.408 n=20)
ExpFloat64-32                15.36n ± 1%    15.29n ± 2%         ~ (p=0.316 n=20)
NormFloat64-32               13.57n ± 1%    13.79n ± 1%    +1.66% (p=0.005 n=20)
Perm3-32                     45.70n ± 2%    46.99n ± 2%    +2.81% (p=0.001 n=20)
Perm30-32                    399.0n ± 1%    403.8n ± 1%    +1.19% (p=0.006 n=20)
Perm30ViaShuffle-32          349.0n ± 1%    350.4n ± 1%         ~ (p=0.909 n=20)
ShuffleOverhead-32           322.3n ± 1%    323.8n ± 1%         ~ (p=0.410 n=20)
Concurrent-32                3.331n ± 1%    7.312n ± 1%  +119.50% (p=0.000 n=20)

For #61716.

Change-Id: Ibdddeed85c34d9ae397289dc899e04d4845f9ed2
Reviewed-on: https://go-review.googlesource.com/c/go/+/516860
Reviewed-by: Michael Pratt <mpratt@google.com>
Reviewed-by: Filippo Valsorda <filippo@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

17 months agomath/rand/v2: add ChaCha8
Russ Cox [Thu, 30 Nov 2023 17:25:43 +0000 (12:25 -0500)]
math/rand/v2: add ChaCha8

This is a replay of CL 516859, after its rollback in CL 543895,
with big-endian systems fixed and the tests disabled on RISC-V
since the compiler is broken there (#64285).

ChaCha8 provides a cryptographically strong generator
alongside PCG, so that people who want stronger randomness
have access to that. On systems with 128-bit vector math
assembly (amd64 and arm64), ChaCha8 runs at about the same
speed as PCG (25% slower on amd64, 2% faster on arm64).

Fixes #64284.

Change-Id: I6290bb8ace28e1aff9a61f805dbe380ccdf25b94
Reviewed-on: https://go-review.googlesource.com/c/go/+/546020
Reviewed-by: Filippo Valsorda <filippo@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

17 months agointernal/trace/v2: handle thread ID reuse correctly
Michael Anthony Knyszek [Wed, 29 Nov 2023 22:18:32 +0000 (22:18 +0000)]
internal/trace/v2: handle thread ID reuse correctly

To begin with, CL 545515 made the trace parser tolerant of
GoCreateSyscall having a P, but that was wrong. Because dropm trashes
the M's syscalltick, that case should never be possible. So the first
thing this change does is it rewrites the test that CL introduced to
expect a failure instead of a success.

What I'd misinterpreted as a case that should be allowed was actually
the same as the other issues causing #64060, which is that the parser
doesn't correctly implement what happens to Ps when a thread calls back
into Go on non-pthread platforms, and what happens when a thread dies
on pthread platorms (or more succinctly, what the runtime does when it
calls dropm).

Specifically, the GoDestroySyscall event implies that if any P is still
running on that M when it's called, that the P stops running. This is
what is intended by the runtime trashing the M's syscalltick; when it
calls back into Go, the tracer models that thread as obtaining a new P
from scratch.

Handling this incorrectly manifests in one of two ways.

On pthread platforms, GoDestroySyscall is only emitted when a C thread
that previously called into Go is destroyed. However, that thread ID can
be reused. Because we have no thread events, whether it's the same
thread or not is totally ambiguous to the tracer. Therefore, the tracer
may observe a thread that previously died try to start running with a
new P under the same identity. The association to the old P is still
intact because the ID is the same, and the tracer gets confused -- it
appears as if two Ps are running on the same M!

On non-pthread platforms, GoDestroySyscall is emitted on every return to
C from Go code. In this case, the same thread with the same identity is
naturally going to keep calling back into Go. But again, since the
runtime trashes syscalltick in dropm, it's always going to acquire a P
from the tracer's perspective. But if this is a different P than before,
just like the pthread case, the parser is going to get confused, since
it looks like two Ps are running on the same M!

The case that CL 545515 actually handled was actually the non-pthread
case, specifically where the same P is reacquired by an M calling back
into Go. In this case, if we tolerate having a P, then what we'll
observe is the M stealing its own P from itself, then running with it.

Now that we know what the problem is, how do we fix it? This change
addresses the problem by emitting an extra event when encountering a
GoDestroySyscall with an active P in its context. In this case, it emits
an additional ProcSteal event to steal from itself, indicating that the
P stopped running. This removes any association between that M and that
P, resolving any ambiguity in the tracer.

There's one other minor detail that needs to be worked out, and that's
what happens to any *real* ProcSteal event that stole the P we're now
emitting an extra ProcSteal event for. Since, this event is going to
look for an M that may have moved on already and the P at this point is
already idle. Luckily, we have *exactly* the right fix for this. The
handler for GoDestroySyscall now moves any active P it has to the
ProcSyscallAbandoned state, indicating that we've lost information about
the P and that it should be treated as already idle. Conceptually this
all makes sense: this is a P in _Psyscall that has been abandoned by the
M it was previously bound to.

It's unfortunate how complicated this has all ended up being, but we can
take a closer look at that in the future.

Fixes #64060.

Change-Id: Ie9e6eb9cf738607617446e3487392643656069a2
Reviewed-on: https://go-review.googlesource.com/c/go/+/546096
Reviewed-by: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Run-TryBot: Michael Knyszek <mknyszek@google.com>
Auto-Submit: Michael Knyszek <mknyszek@google.com>

17 months agoruntime: add the disablethp GODEBUG setting
Michael Anthony Knyszek [Tue, 5 Dec 2023 17:31:34 +0000 (17:31 +0000)]
runtime: add the disablethp GODEBUG setting

Go 1.21.1 and Go 1.22 have ceased working around an issue with Linux
kernel defaults for transparent huge pages that can result in excessive
memory overheads. (https://bugzilla.kernel.org/show_bug.cgi?id=93111)

Many Linux distributions disable huge pages altogether these days, so
this problem isn't quite as far-reaching as it used to be. Also, the
problem only affects Go programs with very particular memory usage
patterns.

That being said, because the runtime used to actively deal with this
problem (but with some unpredictable behavior), it's preventing users
that don't have a lot of control over their execution environment from
upgrading to Go beyond Go 1.20.

This change adds a GODEBUG to smooth over the transition. The GODEBUG
setting disables transparent huge pages for all heap memory on Linux,
which is much more predictable than restoring the old behavior.

Fixes #64332.

Change-Id: I73b1894337f0f0b1a5a17b90da1221e118e0b145
Reviewed-on: https://go-review.googlesource.com/c/go/+/547475
Reviewed-by: Michael Pratt <mpratt@google.com>
Auto-Submit: Michael Knyszek <mknyszek@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

17 months agodoc/go1.22: document enabling PIE by default on darwin/amd64
Cherry Mui [Tue, 5 Dec 2023 17:30:19 +0000 (12:30 -0500)]
doc/go1.22: document enabling PIE by default on darwin/amd64

Updates #61229.
For #61422.

Change-Id: I6cf8169c1e310e0de734250dbe04fb36e14728d9
Reviewed-on: https://go-review.googlesource.com/c/go/+/547455
Reviewed-by: Than McIntosh <thanm@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

17 months agodoc: add release note for math/big.Rat.FloatPrec
Robert Griesemer [Mon, 4 Dec 2023 18:45:50 +0000 (10:45 -0800)]
doc: add release note for math/big.Rat.FloatPrec

For #50489.

Change-Id: I4544a24327196eb3ed62af64ae5ddb1f60441d12
Reviewed-on: https://go-review.googlesource.com/c/go/+/546357
Reviewed-by: Alan Donovan <adonovan@google.com>
Auto-Submit: Robert Griesemer <gri@google.com>
Reviewed-by: Robert Griesemer <gri@google.com>
TryBot-Bypass: Robert Griesemer <gri@google.com>

17 months agolog/slog: correct doc examples for Handler.WithGroup
Jes Cok [Tue, 5 Dec 2023 04:12:45 +0000 (04:12 +0000)]
log/slog: correct doc examples for Handler.WithGroup

Fixes #64519

Change-Id: Iaea5de74402b77eb5a42979bd3baa1276b13d585
GitHub-Last-Rev: 5080a257b2d7aa7c4333d05bdef0b5dcbb0449ed
GitHub-Pull-Request: golang/go#64551
Reviewed-on: https://go-review.googlesource.com/c/go/+/547215
Reviewed-by: Bryan Mills <bcmills@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jes Cok <xigua67damn@gmail.com>
TryBot-Result: Gopher Robot <gobot@golang.org>

17 months agonet/http: limit chunked data overhead
Damien Neil [Tue, 7 Nov 2023 18:47:56 +0000 (10:47 -0800)]
net/http: limit chunked data overhead

The chunked transfer encoding adds some overhead to
the content transferred. When writing one byte per
chunk, for example, there are five bytes of overhead
per byte of data transferred: "1\r\nX\r\n" to send "X".

Chunks may include "chunk extensions",
which we skip over and do not use.
For example: "1;chunk extension here\r\nX\r\n".

A malicious sender can use chunk extensions to add
about 4k of overhead per byte of data.
(The maximum chunk header line size we will accept.)

Track the amount of overhead read in chunked data,
and produce an error if it seems excessive.

Fixes #64433
Fixes CVE-2023-39326

Change-Id: I40f8d70eb6f9575fb43f506eb19132ccedafcf39
Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/2076135
Reviewed-by: Tatiana Bradley <tatianabradley@google.com>
Reviewed-by: Roland Shoemaker <bracewell@google.com>
Reviewed-on: https://go-review.googlesource.com/c/go/+/547335
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

17 months agomaps: fix aliasing problems with Clone
Keith Randall [Fri, 1 Dec 2023 06:59:04 +0000 (22:59 -0800)]
maps: fix aliasing problems with Clone

Make sure to alloc+copy large keys and values instead of aliasing them,
when they might be updated by a future assignment.

Fixes #64474

Change-Id: Ie2226a81cf3897e4e2ee24472f2966d397ace53f
Reviewed-on: https://go-review.googlesource.com/c/go/+/546515
Reviewed-by: Cherry Mui <cherryyz@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Keith Randall <khr@google.com>
Reviewed-by: Mauri de Souza Meneguzzo <mauri870@gmail.com>
17 months agoruntime: require cgo for cgo gdb test
Michael Pratt [Mon, 4 Dec 2023 22:47:52 +0000 (17:47 -0500)]
runtime: require cgo for cgo gdb test

Fixes #64542.

Change-Id: Ic0dc37dfefac83fd3aab6dacfec58b8ee9105fbd
Reviewed-on: https://go-review.googlesource.com/c/go/+/547135
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Michael Pratt <mpratt@google.com>
Auto-Submit: Michael Pratt <mpratt@google.com>
Commit-Queue: Michael Pratt <mpratt@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
17 months agodoc: add release notes for PGO improvements
Michael Pratt [Mon, 4 Dec 2023 18:31:21 +0000 (13:31 -0500)]
doc: add release notes for PGO improvements

For #61577.
For #61422.

Change-Id: I575bf657fb36bd7103c73620bb2371d0f490af20
Reviewed-on: https://go-review.googlesource.com/c/go/+/546935
Reviewed-by: Cherry Mui <cherryyz@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

17 months agocmd/compile: update PGO profile
Cherry Mui [Sat, 2 Dec 2023 21:37:17 +0000 (16:37 -0500)]
cmd/compile: update PGO profile

Since the default.pgo profile is collected, there has been a lot
of development and refactoring in the compiler. It's time to
refresh the compiler's PGO profile. As we are in the freeze, there
will probably not be many major changes to the compiler, so this
should be good for Go 1.22.

Profile collected by running the cmd/compile/profile.sh script on
the linux-amd64-perf gomote.

Benchmark results on Linux/AMD64:

         │  nopgo.txt   │               old.txt               │              new.txt               │
         │    sec/op    │    sec/op     vs base               │   sec/op     vs base               │
Template    127.4m ± 0%    125.3m ± 0%  -1.65% (p=0.000 n=20)   123.5m ± 0%  -3.07% (p=0.000 n=20)
Unicode    105.94m ± 0%   100.77m ± 0%  -4.87% (p=0.000 n=20)   99.26m ± 0%  -6.30% (p=0.000 n=20)
GoTypes     798.2m ± 1%    784.4m ± 0%  -1.73% (p=0.000 n=20)   769.4m ± 1%  -3.61% (p=0.000 n=20)
Compiler    85.10m ± 0%    82.03m ± 1%  -3.61% (p=0.000 n=20)   79.77m ± 1%  -6.26% (p=0.000 n=20)
SSA          6.054 ± 0%     5.945 ± 0%  -1.81% (p=0.000 n=20)    5.811 ± 0%  -4.03% (p=0.000 n=20)
Flate       75.37m ± 1%    74.11m ± 0%  -1.66% (p=0.000 n=20)   73.02m ± 1%  -3.12% (p=0.000 n=20)
GoParser    144.6m ± 1%    141.7m ± 1%  -1.95% (p=0.000 n=20)   140.5m ± 1%  -2.83% (p=0.000 n=20)
Reflect     312.9m ± 1%    304.9m ± 1%  -2.56% (p=0.000 n=20)   300.4m ± 0%  -4.00% (p=0.000 n=20)
Tar         115.8m ± 0%    113.6m ± 0%  -1.88% (p=0.000 n=20)   111.9m ± 1%  -3.33% (p=0.000 n=20)
XML         145.9m ± 1%    143.6m ± 0%  -1.58% (p=0.000 n=20)   141.3m ± 1%  -3.15% (p=0.000 n=20)
geomean     224.1m         218.9m       -2.34%                  215.2m       -3.98%

This also shows that a stale profile (the old one) still brings a
speedup, but smaller, as expected.

As before, the profile is transferrable. Benchmark results on
Darwin/ARM64:

         │  nopgo.txt   │              old.txt               │               new.txt               │
         │    sec/op    │   sec/op     vs base               │    sec/op     vs base               │
Template   71.31m ± 34%   68.97m ± 1%  -3.28% (p=0.000 n=20)   68.66m ± 38%  -3.71% (p=0.030 n=20)
Unicode    58.70m ±  1%   55.64m ± 1%  -5.20% (p=0.000 n=20)   53.05m ±  1%  -9.61% (p=0.000 n=20)
GoTypes    422.9m ±  0%   412.7m ± 0%  -2.43% (p=0.000 n=20)   406.0m ±  1%  -4.01% (p=0.000 n=20)
Compiler   45.89m ±  1%   42.67m ± 2%  -7.00% (p=0.000 n=20)   42.96m ±  1%  -6.38% (p=0.000 n=20)
SSA         3.190 ±  0%    3.090 ± 0%  -3.15% (p=0.000 n=20)    3.007 ±  1%  -5.74% (p=0.000 n=20)
Flate      42.02m ±  1%   41.09m ± 1%  -2.20% (p=0.000 n=20)   40.58m ±  1%  -3.43% (p=0.000 n=20)
GoParser   75.75m ±  1%   73.84m ± 1%  -2.52% (p=0.000 n=20)   72.66m ±  1%  -4.08% (p=0.000 n=20)
Reflect    163.7m ±  1%   158.1m ± 1%  -3.39% (p=0.000 n=20)   155.3m ±  1%  -5.10% (p=0.000 n=20)
Tar        62.03m ±  1%   60.15m ± 0%  -3.02% (p=0.000 n=20)   59.74m ±  1%  -3.69% (p=0.000 n=20)
XML        79.38m ±  1%   77.32m ± 1%  -2.59% (p=0.000 n=20)   75.05m ±  2%  -5.45% (p=0.000 n=20)
geomean    120.9m         116.6m       -3.49%                  114.6m        -5.14%

For #60234.

Change-Id: I4ead6158f835c2e69c0f51ea24d044b82a7320cb
Reviewed-on: https://go-review.googlesource.com/c/go/+/546337
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Michael Pratt <mpratt@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

17 months agodoc: add release notes for runtime/pprof changes
Michael Anthony Knyszek [Thu, 30 Nov 2023 21:14:52 +0000 (21:14 +0000)]
doc: add release notes for runtime/pprof changes

For #50891.
For #61015.

For #61422.

Change-Id: I30d580814ac02fe9f3fbd1a101b2cc05947a9aaa
Reviewed-on: https://go-review.googlesource.com/c/go/+/546475
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

17 months agodoc: add release notes for the new execution tracer
Michael Anthony Knyszek [Thu, 30 Nov 2023 19:11:58 +0000 (19:11 +0000)]
doc: add release notes for the new execution tracer

For #60773.
For #62627.
For #63960.

For #61422.

Change-Id: I3c933f7522f65cd36d11d38a268556d92c8053f9
Reviewed-on: https://go-review.googlesource.com/c/go/+/546026
Reviewed-by: Michael Pratt <mpratt@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Michael Knyszek <mknyszek@google.com>

17 months agoruntime: let the fault thread to crash the process
zzkcode [Thu, 30 Nov 2023 11:39:29 +0000 (11:39 +0000)]
runtime: let the fault thread to crash the process

Let the fault thread to crash the program to make sure while gdb coredump file could see the correct backtrace in the number one thread in gdb.

Fixes #63277.

Change-Id: Ie4473f76f0feba596091433918bcd35a4ff7e11b
GitHub-Last-Rev: f4615c23f663a2f0794ca9e5c86fc2f0cc8552d7
GitHub-Pull-Request: golang/go#63666
Reviewed-on: https://go-review.googlesource.com/c/go/+/536895
Reviewed-by: Michael Pratt <mpratt@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

17 months agonet/http: avoid leaking io.Copy goroutines (and hijacked connections) in TestTranspor...
Bryan C. Mills [Fri, 1 Dec 2023 21:27:43 +0000 (16:27 -0500)]
net/http: avoid leaking io.Copy goroutines (and hijacked connections) in TestTransportNoReuseAfterEarlyResponse

Fixes #64252 (maybe).

Change-Id: Iba2a403a9347be4206f14acb11591dc2eb7f9fb8
Reviewed-on: https://go-review.googlesource.com/c/go/+/546616
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Bryan Mills <bcmills@google.com>

17 months agodoc: add release notes for allocation headers
Michael Anthony Knyszek [Thu, 30 Nov 2023 19:36:46 +0000 (19:36 +0000)]
doc: add release notes for allocation headers

For #61422.

Change-Id: Ia7cb2721e52d9ccd932a49e0c331868eb7749f94
Reviewed-on: https://go-review.googlesource.com/c/go/+/546027
Reviewed-by: Cherry Mui <cherryyz@google.com>
Auto-Submit: Michael Knyszek <mknyszek@google.com>
Reviewed-by: Keith Randall <khr@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

17 months agoruntime: check if mp.curg is nil before readgstatus in writeProcStatusForP
Michael Anthony Knyszek [Fri, 1 Dec 2023 21:05:34 +0000 (21:05 +0000)]
runtime: check if mp.curg is nil before readgstatus in writeProcStatusForP

CL 546025 failed to check if mp.curg is nil, causing all sorts of
failures. It can very well be nil in this context.

For #64318.
Fixes #64507.

Change-Id: I4a95c3fa16d5e1dee8041394c4bdb9c6ba04f032
Reviewed-on: https://go-review.googlesource.com/c/go/+/546636
Reviewed-by: Michael Pratt <mpratt@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

17 months agotesting: add regression tests for reentrant calls to T.Run
Bryan C. Mills [Thu, 30 Nov 2023 16:00:56 +0000 (11:00 -0500)]
testing: add regression tests for reentrant calls to T.Run

These tests represent two patterns of usage, found in Google-internal
tests, that deadlocked after CL 506755.

TestConcurrentRun is a minor variation on TestParallelSub, with the
additional expectation that the concurrent calls to Run (without
explicit calls to Parallel) proceed without blocking. It replaces
TestParallelSub.

TestParentRun is similar, but instead of calling Run concurrently it
calls Run from within the subtest body. It almost certainly represents
an accidental misuse of T.Run, but since that pattern used to run to
completion we don't want to break it accidentally. (Perhaps it should
be diagnosed with a vet check instead?)

While we are testing concurrency, this also cleans up
TestConcurrentCleanup to use a clearer synchronization pattern.

Fixes #64402.

Change-Id: I14fc7e7085a994c284509eac28190c3a8feb04cd
Reviewed-on: https://go-review.googlesource.com/c/go/+/546019
Auto-Submit: Bryan Mills <bcmills@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Alan Donovan <adonovan@google.com>
17 months agodoc: document changes to slices package
Keith Randall [Fri, 1 Dec 2023 20:54:00 +0000 (12:54 -0800)]
doc: document changes to slices package

Change-Id: I4e3f9812463251acd312372dd8a4f8c2cea9e289
Reviewed-on: https://go-review.googlesource.com/c/go/+/545996
Reviewed-by: Eli Bendersky <eliben@google.com>
Reviewed-by: Keith Randall <khr@google.com>
TryBot-Bypass: Keith Randall <khr@google.com>

17 months agodoc: remove encoding/binary.NativeEndian release note placeholder
Keith Randall [Fri, 1 Dec 2023 20:37:40 +0000 (12:37 -0800)]
doc: remove encoding/binary.NativeEndian release note placeholder

This was released in 1.21, release noted in CL 524675.
(I think it was missed originally and then and backported,
but then somehow incorrectly made it into the 1.22 dragnet.)

Change-Id: I45f3182f14f77f8b92a3b7d5ef0011b71fd3c176
Reviewed-on: https://go-review.googlesource.com/c/go/+/546675
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Keith Randall <khr@google.com>
TryBot-Bypass: Keith Randall <khr@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
17 months agogo/types, types2: add testcase for alias issue
Robert Griesemer [Thu, 30 Nov 2023 20:52:01 +0000 (12:52 -0800)]
go/types, types2: add testcase for alias issue

Using the new type Alias node, this issue is now fixed.
Add a test case.

Fixes #50729.

Change-Id: I22a4cf31b83de497e052989ca2054227e65e9937
Reviewed-on: https://go-review.googlesource.com/c/go/+/546455
Run-TryBot: Robert Griesemer <gri@google.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Auto-Submit: Robert Griesemer <gri@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Robert Griesemer <gri@google.com>
17 months agocmd/link: update flag doc
Cherry Mui [Fri, 1 Dec 2023 19:04:35 +0000 (14:04 -0500)]
cmd/link: update flag doc

Update the go doc for linker flags. Remove flags that no longer
exist. Also remove flags that are intended for debugging the
linker from user docs. Add -aslr to the doc.

The -n flag does nothing except print a nearly useless message on
XCOFF linking. Deprecate it.

Fixes #64476.

Change-Id: I518c9c6cc009eae50b7c11308348524ad6a62b69
Reviewed-on: https://go-review.googlesource.com/c/go/+/546615
Reviewed-by: Than McIntosh <thanm@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

17 months agocmd/compile: correct code generation for right shifts on riscv64
Joel Sing [Sat, 25 Nov 2023 15:48:05 +0000 (02:48 +1100)]
cmd/compile: correct code generation for right shifts on riscv64

The code generation on riscv64 will currently result in incorrect
assembly when a 32 bit integer is right shifted by an amount that
exceeds the size of the type. In particular, this occurs when an
int32 or uint32 is cast to a 64 bit type and right shifted by a
value larger than 31.

Fix this by moving the SRAW/SRLW conversion into the right shift
rules and removing the SignExt32to64/ZeroExt32to64. Add additional
rules that rewrite to SRAIW/SRLIW when the shift is less than the
size of the type, or replace/eliminate the shift when it exceeds
the size of the type.

Add SSA tests that would have caught this issue. Also add additional
codegen tests to ensure that the resulting assembly is what we
expect in these overflow cases.

Fixes #64285

Change-Id: Ie97b05668597cfcb91413afefaab18ee1aa145ec
Reviewed-on: https://go-review.googlesource.com/c/go/+/545035
Reviewed-by: Russ Cox <rsc@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: M Zhuo <mzh@golangcn.org>
Reviewed-by: Mark Ryan <markdryan@rivosinc.com>
Run-TryBot: Joel Sing <joel@sing.id.au>
TryBot-Result: Gopher Robot <gobot@golang.org>

17 months agoruntime: emit the correct P status from a safepoint in the v2 tracer
Michael Anthony Knyszek [Thu, 30 Nov 2023 18:13:51 +0000 (18:13 +0000)]
runtime: emit the correct P status from a safepoint in the v2 tracer

The GoSyscallBegin event is a signal for both the P and the G to enter a
syscall state for the trace parser. (Ps can't have their own event
because it's too hard to model. As soon as the P enters _Psyscall it can
get stolen out of it.) But there's a window in time between when that
event is emitted and when the P enters _Psyscall where the P's status
can get emitted. In this window the tracer will emit the wrong status:
Running instead of Syscall. Really any call into the tracer could emit a
status event for the P, but in this particular case it's when running a
safepoint function that explicitly emits an event for the P's status.

The fix is straightforward. The source-of-truth on syscall status is the
G's status, so the function that emits the P's status just needs to
check the status of any G attached to it. If it's in _Gsyscall, then the
tracer should emit a Syscall status for the P if it's in _Prunning.

Fixes #64318.

Change-Id: I3b0fb0d41ff578e62810b04fa5a3ef73e2929b0a
Reviewed-on: https://go-review.googlesource.com/c/go/+/546025
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
17 months agodoc: add release notes about bootstrap toolchain version
Michael Anthony Knyszek [Thu, 30 Nov 2023 21:02:24 +0000 (21:02 +0000)]
doc: add release notes about bootstrap toolchain version

For #61422.

Change-Id: Ie0d3c05752d29aa65983add85935d2505560ec55
Reviewed-on: https://go-review.googlesource.com/c/go/+/546029
Auto-Submit: Michael Knyszek <mknyszek@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
17 months agocmd/compile: fix typecheck range over rune literal
Cuong Manh Le [Thu, 30 Nov 2023 17:16:24 +0000 (00:16 +0700)]
cmd/compile: fix typecheck range over rune literal

With range over int, the rune literal in range expression will be left
as untyped rune, but idealType is not handling this case, causing ICE.

Fixing this by setting the concrete type for untyped rune expresison.

Fixes #64471

Change-Id: I07a151c54ea1d9e1b92e4d96cdfb6e73dca13862
Reviewed-on: https://go-review.googlesource.com/c/go/+/546296
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Cuong Manh Le <cuong.manhle.vn@gmail.com>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

17 months agobytes: add a boundary test for Count of 1 byte
Keith Randall [Thu, 5 Oct 2023 19:20:11 +0000 (12:20 -0700)]
bytes: add a boundary test for Count of 1 byte

This bottoms out in internal/bytealg.Count, which does a fair amount
of avx shenanigans. Make sure we're not reading out of bounds.

Change-Id: Ied0e461f536f75acc24c6797f7fc74e3f3901c10
Reviewed-on: https://go-review.googlesource.com/c/go/+/533116
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
Reviewed-by: Jorropo <jorropo.pgm@gmail.com>
Reviewed-by: Achille Roussel <achille.roussel@gmail.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Keith Randall <khr@google.com>
17 months agocmd/compile: fix memcombine pass for big endian, > 1 byte elements
Keith Randall [Thu, 30 Nov 2023 18:04:16 +0000 (10:04 -0800)]
cmd/compile: fix memcombine pass for big endian, > 1 byte elements

The shift amounts were wrong in this case, leading to miscompilation
of load combining.

Also the store combining was not triggering when it should.

Fixes #64468

Change-Id: Iaeb08972c5fc1d6f628800334789c6af7216e87b
Reviewed-on: https://go-review.googlesource.com/c/go/+/546355
Reviewed-by: David Chase <drchase@google.com>
Reviewed-by: Mauri de Souza Meneguzzo <mauri870@gmail.com>
Reviewed-by: Keith Randall <khr@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

17 months agotesting: remove tests from the running log while they are waiting on parallel subtests
Bryan C. Mills [Wed, 29 Nov 2023 21:13:07 +0000 (16:13 -0500)]
testing: remove tests from the running log while they are waiting on parallel subtests

The parallel subtests are themselves removed from the running map
while they are blocked on calls to t.Parallel, so it is misleading to
log their parents as if they are running when we know they cannot be
making any kind of meaningful progress.

Fixes #64404.

Change-Id: Iaad11d5d4f4c86d775d36e5285c49629dccddd74
Reviewed-on: https://go-review.googlesource.com/c/go/+/546018
Run-TryBot: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Bryan Mills <bcmills@google.com>
Reviewed-by: Alan Donovan <adonovan@google.com>
17 months agodebug/elf: return error in DynValue for invalid dynamic section size
Jes Cok [Wed, 29 Nov 2023 16:23:18 +0000 (16:23 +0000)]
debug/elf: return error in DynValue for invalid dynamic section size

This is a follow-up to CL 536400.

Fixes #64446

Change-Id: I35646732f62cb1937fd448f94ea518544d4295d4
GitHub-Last-Rev: 55db18a909fd44e6b2f2b98fd1a44ad01bb37932
GitHub-Pull-Request: golang/go#64448
Reviewed-on: https://go-review.googlesource.com/c/go/+/545835
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Run-TryBot: Jes Cok <xigua67damn@gmail.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Than McIntosh <thanm@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>

17 months agogo/types: set correct Var.scopePos for parameters/results
Alan Donovan [Mon, 20 Nov 2023 22:22:37 +0000 (17:22 -0500)]
go/types: set correct Var.scopePos for parameters/results

Previously, its value was unset (NoPos), but the correct
value is a point after the signature (FuncType.End) and
before the body.

Also, fix a bug in Scope.Innermost whereby it would return
the wrong (outer) scope when the query position was in
the FuncType portion of a Func{Decl,Lit}.
The fix is to set the scope's pos/end to those of the
complete Func{Decl,Lit}. This is now documented at
Info.Scopes, along with other missing information.

Also, fix a bug in the go/types (but not types2) scope
test, in which comments were discarded by the parser,
causing the entire test to be a no-op (!).

Also, make failures of TestScopeLookupParent more
informative.

Also, add a release note about the change in behavior.

Fixes #64292
Fixes #64295

Change-Id: Ib681f59d1b0b43de977666db08302d7524d3305f
Reviewed-on: https://go-review.googlesource.com/c/go/+/544035
Reviewed-by: Robert Findley <rfindley@google.com>
Reviewed-by: Robert Griesemer <gri@google.com>
Auto-Submit: Robert Griesemer <gri@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Run-TryBot: Robert Griesemer <gri@google.com>

17 months agoruntime/pprof: retry vmmap invocation if it failed due to a reported temporary resour...
Cosmos Nicolaou [Wed, 29 Nov 2023 19:04:57 +0000 (11:04 -0800)]
runtime/pprof: retry vmmap invocation if it failed due to a reported temporary resource shortage

As per #62352 the invocation of vmmap may fail (very rarely) due to
a temporary lack of resources on the test runner machine. This PR
allows for retrying the invocation a fixed number of times before
giving up. This is because we suspect the failure is due to
sensible to retry.

Fixes: #62352
Change-Id: I51aa66b949753d8127cc307181b6ef32e91d5b05
Reviewed-on: https://go-review.googlesource.com/c/go/+/545935
Auto-Submit: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
17 months agoall: update vendored dependencies
Dmitri Shuralyov [Wed, 29 Nov 2023 20:42:53 +0000 (15:42 -0500)]
all: update vendored dependencies

The Go 1.22 code freeze has recently started. This is a time to update
all golang.org/x/... module versions that contribute packages to the
std and cmd modules in the standard library to latest master versions.

Generated with:

go install golang.org/x/build/cmd/updatestd@latest
go install golang.org/x/tools/cmd/bundle@latest
updatestd -goroot=$(pwd) -branch=master

For #36905.

Change-Id: I76525261b9a954ed21a3bd3cb6c4a12e6c031d80
Cq-Include-Trybots: luci.golang.try:gotip-windows-amd64-longtest,gotip-linux-amd64-longtest,gotip-linux-386-longtest
Reviewed-on: https://go-review.googlesource.com/c/go/+/546055
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>

17 months agotypes2, go/types: fix type checking of ~[]E passing to unsafe builtins
Cuong Manh Le [Tue, 28 Nov 2023 02:50:25 +0000 (09:50 +0700)]
types2, go/types: fix type checking of ~[]E passing to unsafe builtins

Fixes #64406

Change-Id: I58002ad722a229fe6db0be08d745fbad86048c6d
Reviewed-on: https://go-review.googlesource.com/c/go/+/545395
Reviewed-by: Robert Griesemer <gri@google.com>
Run-TryBot: Robert Griesemer <gri@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
Auto-Submit: Robert Griesemer <gri@google.com>

17 months agodoc: release notes for GOARM hardfloat/softfloat change
Keith Randall [Tue, 28 Nov 2023 21:09:42 +0000 (13:09 -0800)]
doc: release notes for GOARM hardfloat/softfloat change

See CL 514907

Change-Id: Ieba2d7737115c66990b0ea7629033e787a99be93
Reviewed-on: https://go-review.googlesource.com/c/go/+/545655
Reviewed-by: Keith Randall <khr@google.com>
Reviewed-by: Ludi Rehak <ludi317@gmail.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Keith Randall <khr@golang.org>

17 months agoruntime: docfix countAlloc
Peter Feichtinger [Fri, 24 Nov 2023 12:15:54 +0000 (12:15 +0000)]
runtime: docfix countAlloc

fix typo in `countAlloc` doc

Change-Id: I9f0752412b7a7dfae4915870edeab4ac52e38b2d
GitHub-Last-Rev: 6080d3c03ba6cacb1874af9724cfeb7cae27b78f
GitHub-Pull-Request: golang/go#64357
Reviewed-on: https://go-review.googlesource.com/c/go/+/544755
Reviewed-by: Michael Pratt <mpratt@google.com>
Reviewed-by: Hiro Hamada <laciferin@gmail.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
Auto-Submit: Michael Knyszek <mknyszek@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

17 months agocmd/cgo/internal/test: only test specific frames in testCallbackCallersSEH
qmuntal [Wed, 29 Nov 2023 10:15:54 +0000 (11:15 +0100)]
cmd/cgo/internal/test: only test specific frames in testCallbackCallersSEH

testCallbackCallersSEH will break if anything in this call chain is
refactored to have a different number of function calls.

This change makes the test more robust by only testing the frames
that are actually relevant to the test.

Change-Id: Idb51514d7079f55da6e6ddc52ad43b1ffe32c8c8
Reviewed-on: https://go-review.googlesource.com/c/go/+/545755
Run-TryBot: Quim Muntal <quimmuntal@gmail.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Than McIntosh <thanm@google.com>
Auto-Submit: Quim Muntal <quimmuntal@gmail.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>

17 months agocmd/internal/moddeps: walk GOROOT when it is a symlink
qiulaidongfeng [Wed, 29 Nov 2023 12:45:01 +0000 (12:45 +0000)]
cmd/internal/moddeps: walk GOROOT when it is a symlink

Fixes #64375

Change-Id: I24ce67ef254db447cdf37a3fda5b5ab5fc782a36
GitHub-Last-Rev: 05590b9e20b31413d455a6e87bc38843e33ff116
GitHub-Pull-Request: golang/go#64376
Reviewed-on: https://go-review.googlesource.com/c/go/+/544757
Auto-Submit: Bryan Mills <bcmills@google.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Bryan Mills <bcmills@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
17 months agogo/types: include AST in instantiatedIdent panic message
Alan Donovan [Tue, 28 Nov 2023 21:07:01 +0000 (16:07 -0500)]
go/types: include AST in instantiatedIdent panic message

Updates golang/go#63933

Change-Id: I3ff71e92d124c850eb8b4d136e163bf5786897a2
Reviewed-on: https://go-review.googlesource.com/c/go/+/545615
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Auto-Submit: Alan Donovan <adonovan@google.com>

17 months agointernal/trace/v2: tolerate having a P in GoCreateSyscall
Michael Anthony Knyszek [Tue, 28 Nov 2023 17:10:46 +0000 (17:10 +0000)]
internal/trace/v2: tolerate having a P in GoCreateSyscall

On non-pthread platforms, it's totally possible for the same M to
GoCreateSyscall/GoDestroySyscall on the same thread multiple times. That
same thread may hold onto its P through all those calls.

For #64060.

Change-Id: Ib968bfd439ecd5bc24fc98d78c06145b0d4b7802
Reviewed-on: https://go-review.googlesource.com/c/go/+/545515
Reviewed-by: Michael Pratt <mpratt@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

17 months agoruntime: put ReadMemStats debug assertions behind a double-check mode
Michael Anthony Knyszek [Mon, 27 Nov 2023 22:27:32 +0000 (22:27 +0000)]
runtime: put ReadMemStats debug assertions behind a double-check mode

ReadMemStats has a few assertions it makes about the consistency of the
stats it's about to produce. Specifically, how those stats line up with
runtime-internal stats. These checks are generally useful, but crashing
just because some stats are wrong is a heavy price to pay.

For a long time this wasn't a problem, but very recently it became a
real problem. It turns out that there's real benign skew that can happen
wherein sysmon (which doesn't synchronize with a STW) generates a trace
event when tracing is enabled, and may mutate some stats while
ReadMemStats is running its checks.

Fix this by synchronizing with both sysmon and the tracer. This is a bit
heavy-handed, but better that than false positives.

Also, put the checks behind a debug mode. We want to reduce the risk of
backporting this change, and again, it's not great to crash just because
user-facing stats are off. Still, enable this debug mode during the
runtime tests so we don't lose quite as much coverage from disabling
these checks by default.

Fixes #64401.

Change-Id: I9adb3e5c7161d207648d07373a11da8a5f0fda9a
Reviewed-on: https://go-review.googlesource.com/c/go/+/545277
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
Reviewed-by: Felix Geisendörfer <felix.geisendoerfer@datadoghq.com>
17 months agodoc: document PCALIGN directive
Mauri de Souza Meneguzzo [Tue, 28 Nov 2023 18:50:01 +0000 (18:50 +0000)]
doc: document PCALIGN directive

Fixes #64283

Change-Id: Ie0167410fc5557143f0b2e17af56ad9fe34605cf
GitHub-Last-Rev: 77144c76d38ca261b6728ba12ff128d22553efa5
GitHub-Pull-Request: golang/go#64296
Reviewed-on: https://go-review.googlesource.com/c/go/+/544115
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Keith Randall <khr@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Auto-Submit: Keith Randall <khr@golang.org>
Reviewed-by: Keith Randall <khr@google.com>
17 months agosort: add available godoc link
cui fliter [Fri, 3 Nov 2023 11:54:41 +0000 (19:54 +0800)]
sort: add available godoc link

Change-Id: I64645fef0ffd1cea7c7710ec974520f85e0f7496
Reviewed-on: https://go-review.googlesource.com/c/go/+/539579
Run-TryBot: shuang cui <imcusg@gmail.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
Reviewed-by: qiulaidongfeng <2645477756@qq.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
17 months agoruntime: test for contention in both semaphore paths
Rhys Hiltner [Tue, 21 Nov 2023 23:31:26 +0000 (15:31 -0800)]
runtime: test for contention in both semaphore paths

Most contention on the runtime locks inside semaphores is observed in
runtime.semrelease1, but it can also appear in runtime.semacquire1. When
examining contention profiles in TestRuntimeLockMetricsAndProfile, allow
call stacks that include either.

For #64253

Change-Id: Id4f16af5e9a28615ab5032a3197e8df90f7e382f
Reviewed-on: https://go-review.googlesource.com/c/go/+/544375
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Rhys Hiltner <rhys@justin.tv>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
17 months agocmd/compile: use correct type for slice pointer
Keith Randall [Tue, 28 Nov 2023 01:21:08 +0000 (17:21 -0800)]
cmd/compile: use correct type for slice pointer

The type of the data pointer field of a slice should be a pointer
to the element type, not a *uint8.

This ensures that the SSA value representing the slice's data pointer
can be spilled to the stack slot for the corresponding argument.
Before this change the types didn't match so we ended up spilling the
argument to an autotmp instead of to the dedicated argument slot.

Fixes #64414

Change-Id: I09ee39e93f05aee07e3eceb14e39736d7fd70a33
Reviewed-on: https://go-review.googlesource.com/c/go/+/545357
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Keith Randall <khr@google.com>
Reviewed-by: David Chase <drchase@google.com>
17 months agosyscall: remove ptrace1 on darwin
Cherry Mui [Mon, 27 Nov 2023 20:17:11 +0000 (15:17 -0500)]
syscall: remove ptrace1 on darwin

On Darwin, the ptrace syscall is called in ptrace1, which then be
called in ptrace. This allows ptrace1 be disabled on iOS (by
implementing ptrace differently). But we can also achieve this by
adding a conditional directly in ptrace. This reduces stack usage
with -N -l, while keeping ptrace disabled on iOS.

For #64113.

Change-Id: I89d8e317e77352fffdbb5a25ba21ee9cdf2e1e20
Reviewed-on: https://go-review.googlesource.com/c/go/+/545276
Reviewed-by: Michael Knyszek <mknyszek@google.com>
Run-TryBot: Cherry Mui <cherryyz@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>

17 months agodoc: update release notes from the relnote tool
Michael Anthony Knyszek [Wed, 22 Nov 2023 20:37:13 +0000 (20:37 +0000)]
doc: update release notes from the relnote tool

For #61422.

Change-Id: I0e091c30a445dbc55054c31837c6f051fea3c07d
Reviewed-on: https://go-review.googlesource.com/c/go/+/544537
Auto-Submit: Michael Knyszek <mknyszek@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

17 months agotext/template: more godoc links
Olivier Mengué [Thu, 23 Nov 2023 08:17:56 +0000 (09:17 +0100)]
text/template: more godoc links

Change-Id: I4f3684ea6cbdcc09985c93fef5107d3bd006699e
Reviewed-on: https://go-review.googlesource.com/c/go/+/544735
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Rob Pike <r@golang.org>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>

17 months ago.github: clean up issue forms
Sean Liao [Thu, 23 Nov 2023 18:43:12 +0000 (18:43 +0000)]
.github: clean up issue forms

bugs:
* drop unused labels
* drop the reproduce checkbox:
  it's not a strong signal and introduces clutter in github as a task list
* link go.dev/play

govuln:
* use correct label
* ask for version of the tool
* link go.dev/play

telemetry:
* align title with purpose

Change-Id: Id7dd876e518c75dc22e9aec43d9af6e18af088fc
Reviewed-on: https://go-review.googlesource.com/c/go/+/544775
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
TryBot-Bypass: Dmitri Shuralyov <dmitshur@golang.org>

17 months agoRevert "testing: simplify concurrency and cleanup logic"
Lasse Folger [Fri, 24 Nov 2023 15:30:07 +0000 (15:30 +0000)]
Revert "testing: simplify concurrency and cleanup logic"

reverts CL 506755

Reason for revert: This leads to deadlocks of tests in Google internal testing

For #64402

Change-Id: I78329fc9dcc2377e7e880b264ac1d18d577ef99a
Reviewed-on: https://go-review.googlesource.com/c/go/+/544895
Auto-Submit: Bryan Mills <bcmills@google.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Bryan Mills <bcmills@google.com>

17 months agoio/fs, os: unify PathError.Path for dirFS.{ReadFile, ReadDir}
Jes Cok [Fri, 24 Nov 2023 10:02:34 +0000 (10:02 +0000)]
io/fs, os: unify PathError.Path for dirFS.{ReadFile, ReadDir}

Fixes #64366

Change-Id: Ie78ab2cb9e11c0766665cd37fd7a26d36a1c24fa
GitHub-Last-Rev: 3cb3bb84a0ccccc88ba30258f52f699ac1364a7b
GitHub-Pull-Request: golang/go#64372
Reviewed-on: https://go-review.googlesource.com/c/go/+/544835
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Run-TryBot: Jes Cok <xigua67damn@gmail.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
17 months agoall: fix field names
cui fliter [Sat, 18 Nov 2023 05:51:35 +0000 (13:51 +0800)]
all: fix field names

Change-Id: I3ad7a50707486ebdbbd676b3581df6e3ed0fd3a1
Reviewed-on: https://go-review.googlesource.com/c/go/+/543476
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@google.com>
Reviewed-by: Keith Randall <khr@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Run-TryBot: shuang cui <imcusg@gmail.com>

17 months agopath: add available godoc link
cui fliter [Fri, 3 Nov 2023 11:18:00 +0000 (19:18 +0800)]
path: add available godoc link

Change-Id: I6d40a59cde4c3f1d5094f5126fdbc1195285195f
Reviewed-on: https://go-review.googlesource.com/c/go/+/539577
Reviewed-by: Cherry Mui <cherryyz@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Rob Pike <r@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>

17 months ago.github: try "import/path: issue title" instead of "affected/package: "
Dmitri Shuralyov [Wed, 22 Nov 2023 19:31:37 +0000 (14:31 -0500)]
.github: try "import/path: issue title" instead of "affected/package: "

The multiple issue templates pre-populate the issue title with a prefix
that Go issues customarily have. The "affected/package" phrase is short
for "the import path of the affected package". Let's try simplifying it
to just "import/path", and also include "issue title" to make the title
a more representative template of what the final title should look like.

Updates #29839.

Change-Id: I9736d24cf3d0a51536ac13dd07dd189fb51da021
Reviewed-on: https://go-review.googlesource.com/c/go/+/544556
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Sean Liao <sean@liao.dev>
17 months ago.github: delete archived issue templates
Dmitri Shuralyov [Wed, 22 Nov 2023 20:09:57 +0000 (15:09 -0500)]
.github: delete archived issue templates

We can get the previous templates back from version control if needed.
Delete these unused copies to avoid confusion.

For #63970.

Change-Id: I44e54da06e259239745f4a493de1dae94cca3755
Reviewed-on: https://go-review.googlesource.com/c/go/+/544536
Reviewed-by: Heschi Kreinick <heschi@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

17 months agoruntime: throw from the systemstack in wirep
Michael Anthony Knyszek [Wed, 22 Nov 2023 16:36:23 +0000 (16:36 +0000)]
runtime: throw from the systemstack in wirep

The exitsyscall path, since the introduction of the new execution
tracer, stores a just little bit more data in the exitsyscall stack
frame, causing a build failure from exceeding the nosplit limit with
'-N -l' set on all packages (like Delve does).

One of the paths through which this fails is "throw" from wirep, called
by a callee of exitsyscall. By switching to the systemstack on this
path, we can avoid hitting the nosplit limit, fixing the build. It's
also not totally unreasonable to switch to the systemstack for the
throws in this function, since the function has to be nosplit anyway. It
gives the throw path a bit more wiggle room to dump information than it
otherwise would have.

Fixes #64113.

Change-Id: I56e94e40614a202b8ac2fdc8b8b731493b74e5d0
Reviewed-on: https://go-review.googlesource.com/c/go/+/544535
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
17 months agoapi: promote next to go1.22
Michael Anthony Knyszek [Wed, 22 Nov 2023 16:54:29 +0000 (16:54 +0000)]
api: promote next to go1.22

Change-Id: Ib9d1eabfe6199575ea57183ccb21f62c40e11d84
Reviewed-on: https://go-review.googlesource.com/c/go/+/544555
Reviewed-by: Heschi Kreinick <heschi@google.com>
Auto-Submit: Michael Knyszek <mknyszek@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

17 months agogithub: use forms for issue templates
Hiro [Wed, 22 Nov 2023 17:20:35 +0000 (17:20 +0000)]
github: use forms for issue templates

Migrate from markdown github issue templates to more user friendly yaml templates.

Fixes #63970

Change-Id: I1baf8989f72c1b98bf0027bbb794c0324ea21e32
GitHub-Last-Rev: 725110bf87fa879c2787158a68d01612665ad4e6
GitHub-Pull-Request: golang/go#63655
Reviewed-on: https://go-review.googlesource.com/c/go/+/536776
Reviewed-by: Heschi Kreinick <heschi@google.com>
Run-TryBot: t hepudds <thepudds1460@gmail.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Heschi Kreinick <heschi@google.com>

17 months agoruntime: hold sched.lock over traceThreadDestroy in dropm
Michael Anthony Knyszek [Tue, 21 Nov 2023 23:57:36 +0000 (23:57 +0000)]
runtime: hold sched.lock over traceThreadDestroy in dropm

This is required by traceThreadDestroy, though it's not strictly
necessary in this case. The requirement to hold sched.lock comes from
the assumption that traceThreadDestroy is getting called when the thread
leaves the tracer's view, but in this case the extra m that dropm is
dropping never leaves the allm list. Nevertheless, traceThreadDestroy
requires it just as a safety measure, and that's reasonable. dropm is
generally rare on pthread platforms, so the extra lock acquire over this
short critical section (and only when tracing is enabled) is fine.

Change-Id: Ib631820963c74f2f087d14a0067d0441d75d6785
Reviewed-on: https://go-review.googlesource.com/c/go/+/544396
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
Run-TryBot: Michael Knyszek <mknyszek@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>

17 months agoruntime: move the wakeableSleep lock under sched in the lock rank
Michael Anthony Knyszek [Tue, 21 Nov 2023 23:46:48 +0000 (23:46 +0000)]
runtime: move the wakeableSleep lock under sched in the lock rank

Currently the wakeableSleep lock is placed just after timers in the
ranking, but it turns out the timers lock can never be held over a timer
func, so that's wrong. Meanwhile, wakeableSleep can acquire sched.lock.

wakeableSleep, as it turns out, doesn't have any dependencies -- it's
always acquired in a (mostly) regular goroutine context.

Change-Id: Icc8ea76a8b309fbaf0f02215f16e5f706d49cd95
Reviewed-on: https://go-review.googlesource.com/c/go/+/544395
TryBot-Bypass: Michael Knyszek <mknyszek@google.com>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
17 months agoruntime: don't hold trace.lock over semrelease in readTrace0
Michael Anthony Knyszek [Tue, 21 Nov 2023 23:24:59 +0000 (23:24 +0000)]
runtime: don't hold trace.lock over semrelease in readTrace0

semrelease may unblock a goroutine, but the act of unblocking a
goroutine may emit an event, which in turn may try to acquire trace.lock
again.

It's safe to release trace.lock in readTrace0 for this because all of
the state (one variable) it uses under the lock will be recomputed when
it reacquires the lock. There's also no other synchronization
requirement to hold trace.lock. This is just a mistake.

Change-Id: Iff6c6b02efa298ebed8e60cdf6539ec161d5ec48
Reviewed-on: https://go-review.googlesource.com/c/go/+/544178
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
17 months agoruntime: don't hold the table lock in (*traceStackTable).dump
Michael Anthony Knyszek [Tue, 21 Nov 2023 22:42:35 +0000 (22:42 +0000)]
runtime: don't hold the table lock in (*traceStackTable).dump

There's a conceptual cycle between traceStackTable.lock and
allocation-related locks, but it can't happen in practice because the
caller guarantees that there are no more writers to the table at the
point that dump is called.

But if that's true, then the lock isn't necessary at all. It would be
difficult to model this quiesence in the lockrank mode, so just don't
hold the lock and expand the documentation of the dump method.

Change-Id: Id4db61363f075b7574135529915e8bd4f4f4c082
Reviewed-on: https://go-review.googlesource.com/c/go/+/544177
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
17 months agocmd/link/internal/loadpe: fix .xdata unwind info parsing
Davis Goodin [Wed, 22 Nov 2023 01:12:48 +0000 (17:12 -0800)]
cmd/link/internal/loadpe: fix .xdata unwind info parsing

Unwind info in .xdata was being parsed incorrectly, causing targetOff to
be incorrect and miss finding data in .xdata that it should have found.
This causes a linker issue when using the MinGW MSVCRT compiler.

Contains several fixes based on the exception handling docs: the offset
used to get the number of unwind codes, the calculation of the target
offset based on the dynamic size of the unwind data, and the
UNW_FLAG_CHAININFO flag's value.

Fixes #64200

Change-Id: I6483d921b2bf8a2512a95223bf3c8ce8bc63dc4a
Reviewed-on: https://go-review.googlesource.com/c/go/+/544415
TryBot-Result: Gopher Robot <gobot@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Quim Muntal <quimmuntal@gmail.com>
Run-TryBot: Quim Muntal <quimmuntal@gmail.com>
Reviewed-by: Than McIntosh <thanm@google.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
17 months agocmd/link/internal/ld: fix DWARF type DIE "go kind" bug for synthetic ptr types
Than McIntosh [Tue, 21 Nov 2023 19:44:41 +0000 (14:44 -0500)]
cmd/link/internal/ld: fix DWARF type DIE "go kind" bug for synthetic ptr types

The code path in linker DWARF type generation that synthesizes pointer
type DIEs needed for other synthesized types wasn't properly setting
the DW_AT_go_kind attribute for the new pointer types.

Fixes #64231.

Change-Id: I70c338d2b33ae3b93a4c6f201e5836d91d368086
Reviewed-on: https://go-review.googlesource.com/c/go/+/544315
Reviewed-by: Cherry Mui <cherryyz@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

17 months agoruntime/internal/atomic: deduplicate And/Or code on wasm
Mauri de Souza Meneguzzo [Tue, 21 Nov 2023 19:34:36 +0000 (19:34 +0000)]
runtime/internal/atomic: deduplicate And/Or code on wasm

When I initially added the wasm code for these ops I did not saw that
wasm actually has the Cas operations implemented, although they are
merely pointer assignments since wasm is single threaded.

Now with a generic implementation for And/Or we can add wasm to the
build tags.

For #61395

Change-Id: I997dc90477c772882d6703df1b795dfc0d90a699
GitHub-Last-Rev: 92736a6e34104a9d234c791673fe0bb79fc97b0b
GitHub-Pull-Request: golang/go#64300
Reviewed-on: https://go-review.googlesource.com/c/go/+/544116
Run-TryBot: Mauri de Souza Meneguzzo <mauri870@gmail.com>
Reviewed-by: Keith Randall <khr@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Keith Randall <khr@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>

17 months agointernal/abi, runtime, reflect, cmd: merge maxZero const into internal/abi
qiulaidongfeng [Tue, 21 Nov 2023 22:57:59 +0000 (22:57 +0000)]
internal/abi, runtime,  reflect, cmd: merge maxZero const into internal/abi

For #59670

Change-Id: If38a74ad067a3ea3ff551c0c25c8ef41abec114b
GitHub-Last-Rev: fb1f2f3c9f320017627bc3342b061e1e7f6f7fad
GitHub-Pull-Request: golang/go#64268
Reviewed-on: https://go-review.googlesource.com/c/go/+/543655
Run-TryBot: qiulaidongfeng <2645477756@qq.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
Reviewed-by: Keith Randall <khr@google.com>
Reviewed-by: Keith Randall <khr@golang.org>
17 months agocmd/asm: fix the KMCTR instruction encoding and argument passing
Srinivas Pokala [Mon, 16 Oct 2023 09:18:50 +0000 (11:18 +0200)]
cmd/asm: fix the KMCTR instruction encoding and argument passing

KMCTR encoding arguments incorrect way, which leading illegal instruction wherver we call KMCTR instruction.IBM z13 machine test's TestAESGCM test using gcmASM implementation, which uses KMCTR instruction to encrypt using AES in counter mode and the KIMD instruction for GHASH. z14+ machines onwards uses gcmKMA implementation for the same.

Fixes #63387

Change-Id: I86aeb99573c3f636a71908c99e06a9530655aa5d
Reviewed-on: https://go-review.googlesource.com/c/go/+/535675
Reviewed-by: Vishwanatha HD <vishwanatha.hd@ibm.com>
Reviewed-by: Keith Randall <khr@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Keith Randall <khr@golang.org>
17 months agoruntime: change the name of variables in asan-related assembly functions
fanzha02 [Tue, 21 Nov 2023 06:52:54 +0000 (06:52 +0000)]
runtime: change the name of variables in asan-related assembly functions

Variables in functions implemented in assembly should have the
same names as when they were defined. The names of some variables
in asan-related assembly functions do not follow the above rule,
which will cause the runtime test to fail. This CL fixes this issue.

Updates #64257

Change-Id: I261f4db807d25e460513ef1c92cd1b707cdd1a16
Reviewed-on: https://go-review.googlesource.com/c/go/+/543837
Run-TryBot: Fannie Zhang <Fannie.Zhang@arm.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
17 months agointernal/bytealg: optimize Count with PCALIGN in riscv64
Meng Zhuo [Sun, 12 Nov 2023 09:13:29 +0000 (17:13 +0800)]
internal/bytealg: optimize Count with PCALIGN in riscv64

For #63678

Benchmark on Milk-V Mars CM eMMC (Starfive/JH7110 SoC)

goos: linux
goarch: riscv64
pkg: bytes
                │ /root/bytes.old.bench │        /root/bytes.pc16.bench         │
                │        sec/op         │   sec/op     vs base                  │
Count/10                    223.9n ± 1%   220.8n ± 1%   -1.36% (p=0.001 n=10)
Count/32                    571.6n ± 0%   571.3n ± 0%        ~ (p=0.054 n=10)
Count/4K                    38.56µ ± 0%   38.55µ ± 0%   -0.01% (p=0.010 n=10)
Count/4M                    40.13m ± 0%   39.21m ± 0%   -2.28% (p=0.000 n=10)
Count/64M                   627.5m ± 0%   627.4m ± 0%   -0.01% (p=0.019 n=10)
CountEasy/10                101.3n ± 0%   101.3n ± 0%        ~ (p=1.000 n=10) ¹
CountEasy/32                139.3n ± 0%   139.3n ± 0%        ~ (p=1.000 n=10) ¹
CountEasy/4K                5.565µ ± 0%   5.564µ ± 0%   -0.02% (p=0.001 n=10)
CountEasy/4M                5.619m ± 0%   5.619m ± 0%        ~ (p=0.190 n=10)
CountEasy/64M               89.94m ± 0%   89.93m ± 0%        ~ (p=0.436 n=10)
CountSingle/10              53.80n ± 0%   46.06n ± 0%  -14.39% (p=0.000 n=10)
CountSingle/32             104.30n ± 0%   79.64n ± 0%  -23.64% (p=0.000 n=10)
CountSingle/4K             10.413µ ± 0%   7.247µ ± 0%  -30.40% (p=0.000 n=10)
CountSingle/4M             11.603m ± 0%   8.388m ± 0%  -27.71% (p=0.000 n=10)
CountSingle/64M             230.9m ± 0%   172.3m ± 0%  -25.40% (p=0.000 n=10)
CountHard1                  9.981m ± 0%   9.981m ± 0%        ~ (p=0.810 n=10)
CountHard2                  9.981m ± 0%   9.981m ± 0%        ~ (p=0.315 n=10)
CountHard3                  9.981m ± 0%   9.981m ± 0%        ~ (p=0.159 n=10)
geomean                     144.6µ        133.5µ        -7.70%
¹ all samples are equal

                │ /root/bytes.old.bench │        /root/bytes.pc16.bench         │
                │          B/s          │      B/s       vs base                │
Count/10                   42.60Mi ± 1%    43.19Mi ± 1%   +1.39% (p=0.001 n=10)
Count/32                   53.38Mi ± 0%    53.42Mi ± 0%   +0.06% (p=0.049 n=10)
Count/4K                   101.3Mi ± 0%    101.3Mi ± 0%        ~ (p=0.077 n=10)
Count/4M                   99.68Mi ± 0%   102.01Mi ± 0%   +2.34% (p=0.000 n=10)
Count/64M                  102.0Mi ± 0%    102.0Mi ± 0%        ~ (p=0.076 n=10)
CountEasy/10               94.18Mi ± 0%    94.18Mi ± 0%        ~ (p=0.054 n=10)
CountEasy/32               219.1Mi ± 0%    219.1Mi ± 0%   +0.01% (p=0.016 n=10)
CountEasy/4K               702.0Mi ± 0%    702.0Mi ± 0%   +0.00% (p=0.000 n=10)
CountEasy/4M               711.9Mi ± 0%    711.9Mi ± 0%        ~ (p=0.133 n=10)
CountEasy/64M              711.6Mi ± 0%    711.7Mi ± 0%        ~ (p=0.447 n=10)
CountSingle/10             177.2Mi ± 0%    207.0Mi ± 0%  +16.81% (p=0.000 n=10)
CountSingle/32             292.7Mi ± 0%    383.2Mi ± 0%  +30.91% (p=0.000 n=10)
CountSingle/4K             375.1Mi ± 0%    539.0Mi ± 0%  +43.70% (p=0.000 n=10)
CountSingle/4M             344.7Mi ± 0%    476.9Mi ± 0%  +38.33% (p=0.000 n=10)
CountSingle/64M            277.2Mi ± 0%    371.5Mi ± 0%  +34.05% (p=0.000 n=10)
geomean                    199.7Mi         219.8Mi       +10.10%

Change-Id: I1abf6b220b9802028f8ad5eebc8d3b7cfa3e89ea
Reviewed-on: https://go-review.googlesource.com/c/go/+/541756
Reviewed-by: David Chase <drchase@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Joel Sing <joel@sing.id.au>
Run-TryBot: M Zhuo <mzh@golangcn.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Wang Yaduo <wangyaduo@linux.alibaba.com>
Reviewed-by: Mark Ryan <markdryan@rivosinc.com>
17 months agocmd/internal/obj/riscv: add support of PCALIGN directive
Meng Zhuo [Sun, 12 Nov 2023 09:05:57 +0000 (17:05 +0800)]
cmd/internal/obj/riscv: add support of PCALIGN directive

Add support for PCALIGN directive on riscv.
This directive can be used within Go asm to align instruction
by padding NOP directives.

This patch also adds a test to verify the correctness of the PCALIGN
directive.

Original credit by Cooper Qu (Alibaba)
https://gitee.com/xuantie_riscv/xuantie-patch

Change-Id: I8b6524a2bf81a1baf7c9d04b7da2db6c1a7b428f
Reviewed-on: https://go-review.googlesource.com/c/go/+/541740
Run-TryBot: M Zhuo <mzh@golangcn.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Wang Yaduo <wangyaduo@linux.alibaba.com>
Reviewed-by: David Chase <drchase@google.com>
Reviewed-by: Mark Ryan <markdryan@rivosinc.com>
TryBot-Result: Gopher Robot <gobot@golang.org>

17 months agocrypto/tls: mark RSA KEX cipher suites insecure
Filippo Valsorda [Tue, 21 Nov 2023 22:25:51 +0000 (23:25 +0100)]
crypto/tls: mark RSA KEX cipher suites insecure

Updates #63413

Change-Id: I31fc2f9728582524cac5d101d0011093dbd05ed3
Reviewed-on: https://go-review.googlesource.com/c/go/+/544336
Auto-Submit: Filippo Valsorda <filippo@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: David Chase <drchase@google.com>
Reviewed-by: Roland Shoemaker <roland@golang.org>
17 months agocrypto/tls: check and record godebugs more granularly
Filippo Valsorda [Tue, 21 Nov 2023 22:16:56 +0000 (23:16 +0100)]
crypto/tls: check and record godebugs more granularly

We should call Value as late as possible to allow programs to set
GODEBUG with os.Setenv, and IncNonDefault only when (and every time) the
GODEBUG has an effect on a connection (that we'd have regularly
rejected).

Change-Id: If7a1446de407db7ca2d904d41dda13558b684dda
Reviewed-on: https://go-review.googlesource.com/c/go/+/544335
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Roland Shoemaker <roland@golang.org>
Reviewed-by: David Chase <drchase@google.com>
Auto-Submit: Filippo Valsorda <filippo@golang.org>

17 months agointernal/trace/v2: dump text trace on failure only if it fits in the log
Michael Anthony Knyszek [Tue, 21 Nov 2023 16:51:57 +0000 (16:51 +0000)]
internal/trace/v2: dump text trace on failure only if it fits in the log

Currently we dump text traces to the build log on failure
unconditionally, but this may cause the old infrastructure's builds'
logs to get truncated. Avoid that by setting a threshold on the maximum
size of the text trace we're willing to dump.

We don't need this workaround on the new infrastructure -- logs don't
get truncated there.

Change-Id: I0f50f50bb4b90f87250b673fbe56f48235325610
Reviewed-on: https://go-review.googlesource.com/c/go/+/544216
Auto-Submit: Michael Knyszek <mknyszek@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
17 months agotesting: simplify concurrency and cleanup logic
Bryan C. Mills [Wed, 28 Jun 2023 13:37:47 +0000 (09:37 -0400)]
testing: simplify concurrency and cleanup logic

While investigating #60083, I found a couple of bugs (notably #61034)
that had slipped through code review in part because the concurrency
patterns used in the testing package were too complex for me to fully
reason about. This change adjusts those patterns to be more in line
with current idioms, and to reduce the number of special cases that
depend on details that should be orthogonal. (For example: the details
of how we invoke the Cleanup functions should not depend on whether
the test happened to run any parallel subtests.)

In the process, this change fixes a handful of bugs:

- Concurrent calls to Run (explicitly allowed by TestParallelSub)
  could previously drive the testcontext.running count negative,
  causing the number of running parallel tests to exceed the -parallel
  flag.

- The -failfast flag now takes effect immediately on failure. It no
  longer delays until the test finishes, and no longer misses failures
  during cleanup (fixing #61034).

- If a Cleanup function calls runtime.Goexit (typically via t.FailNow)
  during a panic, Cleanup functions from its parent tests are no
  longer skipped and buffered logs from its parent tests are now
  flushed.

- The time reported for a test with subtests now includes the time spent
  running those subtests, regardless of whether they are parallel.
  (Previously, non-parallel subtests were included but parallel subtests
  were not.)

- Calls to (*B).Run in iterations after the first are now diagnosed
  with a panic. (This diagnoses badly-behaved benchmarks: if Run is
  called during the first iteration, no subsequent iterations are
  supposed to occur.)

Fixes #61034.

Change-Id: I3797f6ef5210a3d2d5d6c2710d3f35c0219b02ea
Cq-Include-Trybots: luci.golang.try:gotip-linux-amd64-longtest,gotip-linux-amd64-longtest-race,gotip-windows-amd64-longtest
Reviewed-on: https://go-review.googlesource.com/c/go/+/506755
Auto-Submit: Bryan Mills <bcmills@google.com>
Reviewed-by: Alan Donovan <adonovan@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

17 months agocrypto/tls: check client's supported versions when using QUIC
Marten Seemann [Wed, 25 Oct 2023 04:04:24 +0000 (11:04 +0700)]
crypto/tls: check client's supported versions when using QUIC

According to RFC 9001 Section 4.2, the client MUST NOT offer any TLS version
older than 1.3.

Fixes #63723.

Change-Id: Ia92f98274ca784e2bc151faf236380af51f699c1
Reviewed-on: https://go-review.googlesource.com/c/go/+/537576
Reviewed-by: Filippo Valsorda <filippo@golang.org>
Auto-Submit: Filippo Valsorda <filippo@golang.org>
Reviewed-by: Roland Shoemaker <roland@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
17 months agoruntime: disable trace v2 optimistic deadlock debugger
Michael Anthony Knyszek [Tue, 21 Nov 2023 16:59:44 +0000 (16:59 +0000)]
runtime: disable trace v2 optimistic deadlock debugger

The v2 execution tracer has a rudimentary deadlock detector, but it's
based on an arbitrary threshold that an actually get hit even if there's
no deadlock. This ends up breaking tests sometimes, and it would be bad
if this just appeared in production logs.

Put this 'deadlock detector' behind a flag.

For #55317.

Change-Id: I286f0c05b3ac9600f4f2f9696065cac8bbd25f00
Reviewed-on: https://go-review.googlesource.com/c/go/+/544235
Reviewed-by: Michael Pratt <mpratt@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

17 months agoruntime: emit a ProcSteal from entersyscall_gcwait
Michael Anthony Knyszek [Tue, 21 Nov 2023 03:23:05 +0000 (03:23 +0000)]
runtime: emit a ProcSteal from entersyscall_gcwait

Currently entersyscall_gcwait always emits a ProcStop event. Most of the
time, this is correct, since the thread that just put the P into
_Psyscall is the same one that is putting it into _Pgcstop. However it's
possible for another thread to steal the P, start running a goroutine,
and then enter another syscall, putting the P back into _Psyscall. In
this case ProcStop is incorrect; the P is getting stolen. This leads to
broken traces.

Fix this by always emitting a ProcSteal event from entersyscall_gcwait.
This means that most of the time a thread will be 'stealing' the proc
from itself when it enters this function, but that's theoretically fine.
A ProcSteal is really just a fancy ProcStop.

Well, it would be if the parser correctly handled a self-steal. This is
a minor bug that just never came up before, but it's an update order
error (the mState is looked up and modified, but then it's modified
again at the end of the function to match newCtx). There's really no
reason a self-steal shouldn't be allowed, so fix that up and add a test.

Change-Id: Iec3d7639d331e3f2d127f92ce50c2c4a7818fcd3
Reviewed-on: https://go-review.googlesource.com/c/go/+/544215
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
17 months agointernal/syscall/unix: add PidFDSendSignal for Linux
Kir Kolyshkin [Wed, 13 Sep 2023 07:58:20 +0000 (00:58 -0700)]
internal/syscall/unix: add PidFDSendSignal for Linux

CL 520266 added pidfd_send_signal linux syscall numbers to the
syscall package for the sake of a unit test.

As pidfd_send_signal will be used from the os package, let's revert the
changes to syscall package, add the pidfd_send_signal syscall numbers
and the implementation to internal/syscall/unix, and change the above
test to use it.

Updates #51246.
For #62654.

Change-Id: I862174c3c1a64baf1080792bdb3a1c1d1b417bb4
Reviewed-on: https://go-review.googlesource.com/c/go/+/528436
Run-TryBot: Kirill Kolyshkin <kolyshkin@gmail.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Michael Pratt <mpratt@google.com>
17 months agoruntime: add lock partial order edge between wakeableSleep and hchan
Michael Anthony Knyszek [Tue, 21 Nov 2023 21:47:57 +0000 (21:47 +0000)]
runtime: add lock partial order edge between wakeableSleep and hchan

This is totally valid and always was, but the staticlockranking builder
started failing when the new execution tracer was enabled by default.

Change-Id: I011e7d86bd968b1251bcc4d74395633036753b00
Reviewed-on: https://go-review.googlesource.com/c/go/+/544319
Reviewed-by: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Run-TryBot: Michael Knyszek <mknyszek@google.com>

17 months agoerrors: clarify references to Unwrap in doc
Olivier Mengué [Thu, 26 Oct 2023 20:18:33 +0000 (22:18 +0200)]
errors: clarify references to Unwrap in doc

CL 535080 incorrectly links the unclear mention of Unwrap to the func
Unwrap in doc for errors.Is and errors.As

Instead we clarify that "Unwrap" is a reference
to the "Unwrap() error" or "Unwrap() []error" methods, not to the
"Unwrap(error) error" function which is also available in the package.

Change-Id: I8314993932e1e7a2dc77400f74d81f3a8aa891de
Reviewed-on: https://go-review.googlesource.com/c/go/+/538155
Reviewed-by: Cherry Mui <cherryyz@google.com>
Run-TryBot: qiulaidongfeng <2645477756@qq.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: David Chase <drchase@google.com>
17 months agosyscall: check SyscallIsNotSupported in TestPidFDWithUserNS
Michael Pratt [Tue, 21 Nov 2023 20:37:49 +0000 (15:37 -0500)]
syscall: check SyscallIsNotSupported in TestPidFDWithUserNS

For #51246.

Change-Id: Ief2e2e14f039123a6580cb60be7ee74f4a20a649
Reviewed-on: https://go-review.googlesource.com/c/go/+/544318
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Michael Pratt <mpratt@google.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

17 months agoruntime: enable the exectracer2 experiment by default
Michael Anthony Knyszek [Wed, 8 Nov 2023 19:05:11 +0000 (19:05 +0000)]
runtime: enable the exectracer2 experiment by default

Fixes #60773.

Cq-Include-Trybots: luci.golang.try:gotip-linux-amd64-longtest,gotip-linux-amd64-longtest-race
Change-Id: Ice3fd4577d58ce593611144242f2cba99c9d2ecb
Reviewed-on: https://go-review.googlesource.com/c/go/+/540778
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
Auto-Submit: Michael Knyszek <mknyszek@google.com>

17 months agocmd/trace/v2: emit regions in the goroutine-oriented task view
Michael Anthony Knyszek [Mon, 20 Nov 2023 21:19:35 +0000 (21:19 +0000)]
cmd/trace/v2: emit regions in the goroutine-oriented task view

This change emits regions in the goroutine-oriented task view (the
/trace endpoint with the taskid query variable set) in the same way the
old cmd/trace does.

For #60773.
Fixes #63960.

Change-Id: If6c3e7072c694c84a7d2d6c34df668f48d3acc2a
Reviewed-on: https://go-review.googlesource.com/c/go/+/543995
Reviewed-by: Michael Pratt <mpratt@google.com>
Auto-Submit: Michael Knyszek <mknyszek@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

17 months agocmd/trace/v2: add support for the goroutine-oriented task view
Michael Anthony Knyszek [Mon, 20 Nov 2023 19:29:42 +0000 (19:29 +0000)]
cmd/trace/v2: add support for the goroutine-oriented task view

This change adds support for a goroutine-oriented task view via the
/trace?taskid=<taskid> endpoint. This works but it's missing regions.
That will be implemented in a follow-up CL.

For #60773.
For #63960.

Change-Id: I086694143e5c71596ac22fc551416868f0b80923
Reviewed-on: https://go-review.googlesource.com/c/go/+/543937
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Michael Knyszek <mknyszek@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
17 months agocmd/trace/v2: add thread-oriented mode for v2 traces
Michael Anthony Knyszek [Mon, 20 Nov 2023 07:28:37 +0000 (07:28 +0000)]
cmd/trace/v2: add thread-oriented mode for v2 traces

This is a nice-to-have that's now straightforward to do with the new
trace format. This change adds a new query variable passed to the
/trace endpoint called "view," which indicates the type of view to
use. It is orthogonal with task-related views.

Unfortunately a goroutine-based view isn't included because it's too
likely to cause the browser tab to crash.

For #60773.
For #63960.

Change-Id: Ifbcb8f2d58ffd425819bdb09c586819cb786478d
Reviewed-on: https://go-review.googlesource.com/c/go/+/543695
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
Auto-Submit: Michael Knyszek <mknyszek@google.com>

17 months agocmd/trace/v2: add support for a task-oriented procs-based view
Michael Anthony Knyszek [Mon, 20 Nov 2023 05:37:36 +0000 (05:37 +0000)]
cmd/trace/v2: add support for a task-oriented procs-based view

This change implements support for the trace?focustask=<taskid> endpoint
in the trace tool for v2 traces.

Note: the one missing feature in v2 vs. v1 is that the "irrelevant" (but
still rendered) events are not grayed out. This basically includes
events that overlapped with events that overlapped with other events
that were in the task time period, but aren't themselves directly
associated. This is probably fine -- the UI already puts a very obvious
focus on the period of time the selected task was running.

For #60773.
For #63960.

Change-Id: I5c78a220ae816e331b74cb67c01c5cd98be40dd4
Reviewed-on: https://go-review.googlesource.com/c/go/+/543596
Auto-Submit: Michael Knyszek <mknyszek@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
17 months agocmd/trace/v2: add support for goroutine filtering
Michael Anthony Knyszek [Sat, 18 Nov 2023 05:50:50 +0000 (05:50 +0000)]
cmd/trace/v2: add support for goroutine filtering

This change adds support for the trace?goid=<goid> endpoint to the trace
tool for v2 traces.

In effect, this change actually implements a per-goroutine view. I tried
to add a link to the main page to enable a "view by goroutines" view
without filtering, but the web trace viewer broke the browser tab when
there were a few hundred goroutines. The risk of a browser hang probably
isn't worth the cases where this is nice, especially since filtering by
goroutine already works. Unfortunate, but c'est l'vie. Might be worth
revisiting if we change out the web viewer in the future.

For #60773.
For #63960.

Change-Id: I8e29f4ab8346af6708fd8824505c30f2c43db796
Reviewed-on: https://go-review.googlesource.com/c/go/+/543595
TryBot-Bypass: Michael Knyszek <mknyszek@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
Auto-Submit: Michael Knyszek <mknyszek@google.com>

17 months agocmd/trace/v2: add support for task and region endpoints
Michael Anthony Knyszek [Tue, 14 Nov 2023 03:38:04 +0000 (03:38 +0000)]
cmd/trace/v2: add support for task and region endpoints

This change fills out the last of cmd/trace's subpages for v2 traces by
adding support for task and region endpoints.

For #60773.
For #63960.

Change-Id: Ifc4c660514b3904788785a1b20e3abc3bb9e55f1
Reviewed-on: https://go-review.googlesource.com/c/go/+/542077
Reviewed-by: Michael Pratt <mpratt@google.com>
Auto-Submit: Michael Knyszek <mknyszek@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

17 months agocmd/trace: factor out durationHistogram
Michael Anthony Knyszek [Tue, 14 Nov 2023 06:19:07 +0000 (06:19 +0000)]
cmd/trace: factor out durationHistogram

This code will be useful for the new tracer, and there's no need to
duplicate it. This change copies it to internal/trace/traceviewer, adds
some comments, and renames it to TimeHistogram.

While we're here, let's get rid of the unused String method which has a
comment talking about how awful the rendering is.

Also, let's get rid of uses of niceDuration. We'd have to bring it
with us in the move and I don't think it's worth it. The difference
between the default time.Duration rendering and the niceDuration
rendering is usually a few extra digits of precision. Yes, it's noisier,
but AFAICT it's not substantially worse. It doesn't seem worth the new
API, even if it's just internal. We can also always bring it back later.

For #60773.
For #63960.

Change-Id: I795f58f579f1d503c540c3a40bed12e52bce38e2
Reviewed-on: https://go-review.googlesource.com/c/go/+/542001
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Michael Knyszek <mknyszek@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
17 months agointernal/trace: add task analysis for v2 traces
Michael Anthony Knyszek [Mon, 13 Nov 2023 21:06:40 +0000 (21:06 +0000)]
internal/trace: add task analysis for v2 traces

For v1 traces, cmd/trace contains code for analyzing tasks separately
from the goroutine analysis code present in internal/trace. As I started
to look into porting that code to v2 traces, I noticed that it wouldn't
be too hard to just generalize the existing v2 goroutine summary code to
generate exactly the same information.

This change does exactly that.

For #60773.
For #63960.

Change-Id: I0cdd9bf9ba11fb292a9ffc37dbf18c2a6a2483b8
Reviewed-on: https://go-review.googlesource.com/c/go/+/542076
Auto-Submit: Michael Knyszek <mknyszek@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
17 months agointernal/trace/v2: redefine NoTask and add BackgroundTask
Michael Anthony Knyszek [Fri, 17 Nov 2023 17:56:27 +0000 (17:56 +0000)]
internal/trace/v2: redefine NoTask and add BackgroundTask

The v2 trace parser currently handles task inheritance and region task
association incorrectly. It assumes that a TaskID of 0 means that there
is no task. However, this is only true for task events. A TaskID of 0
means that a region gets assigned to the "background task." The parser
currently has no concept of a "background task."

Fix this by defining the background task as task ID 0 and redefining
NoTask to ^uint64(0). This aligns the TaskID values more closely with
other IDs in the parser and also enables disambiguating these two cases.

For #60773.
For #63960.

Change-Id: I09c8217b33b87c8f8f8ea3b0203ed83fd3b61e11
Reviewed-on: https://go-review.googlesource.com/c/go/+/543019
Reviewed-by: Michael Pratt <mpratt@google.com>
Auto-Submit: Michael Knyszek <mknyszek@google.com>
TryBot-Bypass: Michael Knyszek <mknyszek@google.com>

17 months agocmd/trace/v2: add support for pprof endpoints
Michael Anthony Knyszek [Mon, 13 Nov 2023 18:57:06 +0000 (18:57 +0000)]
cmd/trace/v2: add support for pprof endpoints

This change adds support for the pprof endpoints to cmd/trace/v2.

In the process, I realized we need to pass the goroutine summaries to
more places, and previous CLs had already done the goroutine analysis
during cmd/trace startup. This change thus refactors the goroutine
analysis API once again to operate in a streaming manner, and to run
at the same time as the initial trace parsing. Now we can include it in
the parsedTrace type and pass that around as the de-facto global trace
context.

Note: for simplicity, this change redefines "syscall" profiles to
capture *all* syscalls, not just syscalls that block. IIUC, this choice
was partly the result of a limitation in the previous trace format that
syscalls don't all have complete durations and many short syscalls are
treated as instant. To this end, this change modifies the text on the
main trace webpage to reflect this change.

For #60773.
For #63960.

Change-Id: I601d9250ab0849a0bfaef233fd9b1e81aca9a22a
Reviewed-on: https://go-review.googlesource.com/c/go/+/541999
Auto-Submit: Michael Knyszek <mknyszek@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

17 months agocmd/trace: refactor pprof HTTP SVG serving into traceviewer
Michael Anthony Knyszek [Mon, 13 Nov 2023 05:00:41 +0000 (05:00 +0000)]
cmd/trace: refactor pprof HTTP SVG serving into traceviewer

For #60773.
For #63960.

Change-Id: Id97380f19267ec765b25a703ea3e2f284396ad75
Reviewed-on: https://go-review.googlesource.com/c/go/+/541998
Auto-Submit: Michael Knyszek <mknyszek@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
17 months agointernal/trace/traceviewer: make the mmu handler more self-contained
Michael Anthony Knyszek [Sun, 12 Nov 2023 23:27:31 +0000 (23:27 +0000)]
internal/trace/traceviewer: make the mmu handler more self-contained

The last change made the MMU rendering code common and introduced a new
API, but it was kind of messy. Part of the problem was that some of the
Javascript in the template for the main page referred to specific
endpoints on the server.

Fix this by having the Javascript access the same endpoint but with a
different query variable. Now the Javascript code doesn't depend on
specific endpoints, just on query variables for the current endpoint.

For #60773.
For #63960.

Change-Id: I1c559d9859c3a0d62e2094c9d4ab117890b63b31
Reviewed-on: https://go-review.googlesource.com/c/go/+/541259
Auto-Submit: Michael Knyszek <mknyszek@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

17 months agocmd/trace: common up the mmu page and add it to cmd/trace/v2
Michael Anthony Knyszek [Sun, 12 Nov 2023 23:10:04 +0000 (23:10 +0000)]
cmd/trace: common up the mmu page and add it to cmd/trace/v2

This change moves the MMU HTTP handlers and functionality into the
traceviewer package, since unlike the goroutine pages the vast majority
of that functionality is identical between v1 and v2. This change
involves some refactoring so that callers can plug in their own mutator
utilization computation functions (which is the only point of difference
between v1 and v2). The new interface isn't especially nice, but part of
the problem is the MMU handlers depend on specific endpoints to exist. A
follow-up CL will clean this up a bit.

Like the previous CL did for goroutine analysis, modify the v2 mutator
utilization API to accept a slice of trace events. Again, we might as
well reuse what was already parsed and will be needed for other
purposes. It also simplifies the API slightly.

For #60773.
For #63960.

Change-Id: I6c21ec8d1bf7e95eff5363d0e0005c9217fa00e7
Reviewed-on: https://go-review.googlesource.com/c/go/+/541258
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
Auto-Submit: Michael Knyszek <mknyszek@google.com>

17 months agocmd/trace/v2: add goroutine analysis pages
Michael Anthony Knyszek [Sun, 12 Nov 2023 21:48:31 +0000 (21:48 +0000)]
cmd/trace/v2: add goroutine analysis pages

This is a complete fork and most of a rewrite of the goroutine analysis
pages for v2 traces. It fixes an issue with the old page where GC time
didn't really make any sense, generalizes the page and breaks things
down further, and adds clarifying text.

This change also modifies the SummarizeGoroutines API to not stream the
trace. This is unfortunate, but we're already reading and holding the
entire trace in memory for the trace viewer. We can revisit this
decision in the future. Also, we want to do this now because the
GoroutineSummary holds on to pointers to events, and these events will
be used by the user region and user task analyses. While tracev2 events
are values and they should be equivalent no matter how many times we
parse a trace, this lets us reference the event in the slice directly.

For #60773.
For #63960.
Fixes #62443.

Change-Id: I1c5ab68141869378843f4f2826686038e4533090
Reviewed-on: https://go-review.googlesource.com/c/go/+/541257
Reviewed-by: Michael Pratt <mpratt@google.com>
Auto-Submit: Michael Knyszek <mknyszek@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

17 months agointernal/trace: use the correct stack for goroutine naming in v2 traces
Michael Anthony Knyszek [Tue, 21 Nov 2023 17:33:20 +0000 (17:33 +0000)]
internal/trace: use the correct stack for goroutine naming in v2 traces

Currently goroutine names are determined (for v2 traces) by
internal/tracev/2.Event.Stack, but this is wrong in general. For
example, if we end up seeing a transition from GoNotExist->GoRunnable
(goroutine creation) then we're taking the stack from the creator, not
the created goroutine (which is what we're naming at that point).

Use the StateTransition.Stack instead. This is always the correct one to
use because we're always naming the goroutine that the state transition
is for.

Change-Id: I3fc7c8e4f85dfee3802d666c0c091b6953c7d6cf
Reviewed-on: https://go-review.googlesource.com/c/go/+/544317
Auto-Submit: Michael Knyszek <mknyszek@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
17 months agoruntime: profile contended lock calls
Rhys Hiltner [Tue, 21 Nov 2023 16:03:54 +0000 (16:03 +0000)]
runtime: profile contended lock calls

Add runtime-internal locks to the mutex contention profile.

Store up to one call stack responsible for lock contention on the M,
until it's safe to contribute its value to the mprof table. Try to use
that limited local storage space for a relatively large source of
contention, and attribute any contention in stacks we're not able to
store to a sentinel _LostContendedLock function.

Avoid ballooning lock contention while manipulating the mprof table by
attributing to that sentinel function any lock contention experienced
while reporting lock contention.

Guard collecting real call stacks with GODEBUG=profileruntimelocks=1,
since the available data has mixed semantics; we can easily capture an
M's own wait time, but we'd prefer for the profile entry of each
critical section to describe how long it made the other Ms wait. It's
too late in the Go 1.22 cycle to make the required changes to
futex-based locks. When not enabled, attribute the time to the sentinel
function instead.

Fixes #57071

This is a roll-forward of https://go.dev/cl/528657, which was reverted
in https://go.dev/cl/543660

Reason for revert: de-flakes tests (reduces dependence on fine-grained
timers, correctly identifies contention on big-endian futex locks,
attempts to measure contention in the semaphore implementation but only
uses that secondary measurement to finish the test early, skips tests on
single-processor systems)

Change-Id: I31389f24283d85e46ad9ba8d4f514cb9add8dfb0
Reviewed-on: https://go-review.googlesource.com/c/go/+/544195
Reviewed-by: Michael Pratt <mpratt@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
Auto-Submit: Rhys Hiltner <rhys@justin.tv>
Run-TryBot: Rhys Hiltner <rhys@justin.tv>

17 months agointernal/trace/v2: forward Event.Stack to StateTransition.Stack
Michael Anthony Knyszek [Tue, 21 Nov 2023 17:23:43 +0000 (17:23 +0000)]
internal/trace/v2: forward Event.Stack to StateTransition.Stack

Currently StateTransition.Stack is only set for the GoCreate case,
because there are two stacks and we need to distinguish them. But the
docs for StateTransition.Stack say that that stack always references the
resource that is transitioning. There are quite a few cases where
Event.Stack is actually the appropriate stack to for
StateTransition.Stack, but in these cases it's left empty, and the
caller just needs to understand which one to look at. This isn't great.
Forward Event.Stack to StateTransition.Stack whenever Event.Stack also
refers to the resource experiencing the state transition.

Change-Id: Ie43fc6036f2712c7982174d5739d95765312dfcc
Reviewed-on: https://go-review.googlesource.com/c/go/+/544316
Auto-Submit: Michael Knyszek <mknyszek@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>