runtime: make explicit nil check in (*spanInlineMarkBits).init

The hugo binary gets slower, potentially dramatically so, with
GOEXPERIMENT=greenteagc. The root cause is page mapping churn. The Green
Tea code introduced a new implicit nil check on value in a
freshly-allocated span to clear some new heap metadata. This nil check
would read the fresh memory, causing Linux to back that virtual address
space with an RO page. This would then be almost immediately written to,
causing Linux to possibly flush the TLB and find memory to replace that
read-only page (likely deduplicated as just the zero page).

This CL fixes the issue by replacing the implicit nil check, which is a
memory read expected to fault if it's truly nil, with an explicit one.
The explicit nil check is a branch, and thus makes no reads to memory.
The result is that the hugo binary no longer gets slower.

No regression test because it doesn't seem possible without access to OS
internals, like Linux tracepoints. We briefly experimented with RSS
metrics, but they're inconsistent. Some system RSS metrics count the
deduplicated zero page, while others (like those produced by
/proc/self/smaps) do not.

Instead, we'll add a new benchmark to our benchmark suite, separately.

For #73581.
Fixes #74375.

Change-Id: I708321c14749a94ccff55072663012eba18b3b91
Reviewed-on: https://go-review.googlesource.com/c/go/+/684015
Reviewed-by: Keith Randall <khr@golang.org>
Reviewed-by: Michael Pratt <mpratt@google.com>
Auto-Submit: Michael Knyszek <mknyszek@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Keith Randall <khr@google.com>

diff --git a/src/runtime/mgcmark_greenteagc.go b/src/runtime/mgcmark_greenteagc.go
index 75c347b9e93bcfc9a45a65ff0c6c72c285ca7181..ac2b1732f930265aac262ba3e3726c1a1b4e848c 100644 (file)
--- a/src/runtime/mgcmark_greenteagc.go
+++ b/src/runtime/mgcmark_greenteagc.go
@@ -111,6 +111,26 @@ func (o *spanScanOwnership) or(v spanScanOwnership) spanScanOwnership {
 }
 
 func (imb *spanInlineMarkBits) init(class spanClass) {
+       if imb == nil {
+               // This nil check and throw is almost pointless. Normally we would
+               // expect imb to never be nil. However, this is called on potentially
+               // freshly-allocated virtual memory. As of 2025, the compiler-inserted
+               // nil check is not a branch but a memory read that we expect to fault
+               // if the pointer really is nil.
+               //
+               // However, this causes a read of the page, and operating systems may
+               // take it as a hint to back the accessed memory with a read-only zero
+               // page. However, we immediately write to this memory, which can then
+               // force operating systems to have to update the page table and flush
+               // the TLB, causing a lot of churn for programs that are short-lived
+               // and monotonically grow in size.
+               //
+               // This nil check is thus an explicit branch instead of what the compiler
+               // would insert circa 2025, which is a memory read instruction.
+               //
+               // See go.dev/issue/74375 for details.
+               throw("runtime: span inline mark bits nil?")
+       }
        *imb = spanInlineMarkBits{}
        imb.class = class
 }