runtime: offset the heap arena index by 2^47 on amd64

On amd64, the virtual address space, when interpreted as signed
values, is [-2^47, 2^47). Currently, we only support heap addresses in
the "positive" half of this, [0, 2^47). This suffices for linux/amd64
and windows/amd64, but solaris/amd64 can map user addresses in the
negative part of this range. Specifically, addresses
0xFFFF8000'00000000 to 0xFFFFFD80'00000000 are part of user space.
This leads to "memory allocated by OS not in usable address space"
panic, since we don't map heap arena index space for these addresses.

Fix this by offsetting addresses when computing arena indexes so that
arena entry 0 corresponds to address -2^47 on amd64. We already map
enough arena space for 2^48 heap addresses on 64-bit (because arm64's
virtual address space is [0, 2^48)), so we don't need to grow any
structures to support this.

A different approach would be to simply mask out the top 16 bits.
However, there are two advantages to the offset approach: 1) invalid
heap addresses continue to naturally map to invalid arena indexes so
we don't need extra checks and 2) it perturbs the mapping of addresses
to arena indexes more, which helps check that we don't accidentally
compute incorrect arena indexes somewhere that happen to be right most
of the time.

Several comments and constant names are now somewhat misleading. We'll
fix that in the next CL. This CL is the core change the arena
indexing.

Fixes #23862.

Change-Id: Idb8e299fded04593a286b01a9582da6ddbac2f9a
Reviewed-on: https://go-review.googlesource.com/95497
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Rick Hudson <rlh@golang.org>

diff --git a/src/runtime/malloc.go b/src/runtime/malloc.go
index a44dcd8c9da4d70c384de63cc5b6def9666a30b1..b6e3337d67ed5e0dae80c3e35dc0bef4754bde72 100644 (file)
--- a/src/runtime/malloc.go
+++ b/src/runtime/malloc.go
@@ -214,6 +214,17 @@ const (
 
        pagesPerArena = heapArenaBytes / pageSize
 
+       // arenaBaseOffset is the pointer value that corresponds to
+       // index 0 in the heap arena index.
+       //
+       // On amd64, the address space is 48 bits, sign extended to 64
+       // bits. This offset lets us handle "negative" addresses (or
+       // high addresses if viewed as unsigned).
+       //
+       // On other platforms, the user address space is contiguous
+       // and starts at 0, so no offset is necessary.
+       arenaBaseOffset uintptr = sys.GoarchAmd64 * (1 << 47)
+
        // Max number of threads to run garbage collection.
        // 2, 3, and 4 are all plausible maximums depending
        // on the hardware details of the machine. The garbage

diff --git a/src/runtime/mheap.go b/src/runtime/mheap.go
index 88fcbdda6e8459cc44e34dd31b103a2938dc5110..78cf2c8df901bac334a0746a762e1336324c1969 100644 (file)
--- a/src/runtime/mheap.go
+++ b/src/runtime/mheap.go
@@ -96,8 +96,9 @@ type mheap struct {
        nlargefree  uint64                  // number of frees for large objects (>maxsmallsize)
        nsmallfree  [_NumSizeClasses]uint64 // number of frees for small objects (<=maxsmallsize)
 
-       // arenas is the heap arena index. arenas[va/heapArenaBytes]
-       // points to the metadata for the heap arena containing va.
+       // arenas is the heap arena index.
+       // arenas[(va+arenaBaseOffset)/heapArenaBytes] points to the
+       // metadata for the heap arena containing va.
        //
        // Use arenaIndex to compute indexes into this array.
        //
@@ -418,13 +419,13 @@ func (sc spanClass) noscan() bool {
 //
 //go:nosplit
 func arenaIndex(p uintptr) uint {
-       return uint(p / heapArenaBytes)
+       return uint((p + arenaBaseOffset) / heapArenaBytes)
 }
 
 // arenaBase returns the low address of the region covered by heap
 // arena i.
 func arenaBase(i uint) uintptr {
-       return uintptr(i) * heapArenaBytes
+       return uintptr(i)*heapArenaBytes - arenaBaseOffset
 }
 
 // inheap reports whether b is a pointer into a (potentially dead) heap object.