runtime: optimize storing new keys in mapassign_fastNN

Prior to this change, we use typedmemmove to write the key
value to its new location in mapassign_fast32 and mapassign_fast64.
(The use of typedmemmove was a last-minute fix in the 1.9 cycle;
see #21297 and CL 53414.)

This is significantly less inefficient than direct assignment or
calling writebarrierptr directly.

Fortunately, there aren't many cases to consider.

On systems with 32 bit pointers:

* A 32 bit AMEM value either is a single pointer or has no pointers.
* A 64 bit AMEM value may contain a pointer at the beginning,
  a pointer at 32 bits, or two pointers.

On systems with 64 bit pointers:

* A 32 bit AMEM value contains no pointers.
* A 64 bit AMEM value either is a single pointer or has no pointers.

All combinations except the 32 bit pointers / 64 bit AMEM value are
cheap and easy to handle, and the problematic case is likely rare.
The most popular map keys appear to be ints and pointers.

So we handle them exhaustively. The sys.PtrSize checks are constant branches
and are eliminated by the compiler.

An alternative fix would be to return a pointer to the key,
and have the calling code do the assignment, at which point the compiler
would have full type information.

Initial tests suggest that the performance difference between these
strategies is negligible, and this fix is considerably simpler,
and has much less impact on binary size.

Fixes #21321

Change-Id: Ib03200e89e2324dd3c76d041131447df66f22bfe
Reviewed-on: https://go-review.googlesource.com/59110
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
Reviewed-by: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>

diff --git a/src/runtime/hashmap_fast.go b/src/runtime/hashmap_fast.go
index befc4794fbc32f01bce21b7b86c8c1396a24aeba..0dfef324a2b3edded4185fbf15b877ec8016a4d6 100644 (file)
--- a/src/runtime/hashmap_fast.go
+++ b/src/runtime/hashmap_fast.go
@@ -419,8 +419,12 @@ again:
                val = add(insertk, bucketCnt*4)
        }
 
-       // store new key/value at insert position
-       typedmemmove(t.key, insertk, unsafe.Pointer(&key))
+       // store new key at insert position
+       if sys.PtrSize == 4 && t.key.kind&kindNoPointers == 0 && writeBarrier.enabled {
+               writebarrierptr((*uintptr)(insertk), uintptr(key))
+       } else {
+               *(*uint32)(insertk) = key
+       }
        *inserti = top
        h.count++
 
@@ -504,8 +508,19 @@ again:
                val = add(insertk, bucketCnt*8)
        }
 
-       // store new key/value at insert position
-       typedmemmove(t.key, insertk, unsafe.Pointer(&key))
+       // store new key at insert position
+       if t.key.kind&kindNoPointers == 0 && writeBarrier.enabled {
+               if sys.PtrSize == 8 {
+                       writebarrierptr((*uintptr)(insertk), uintptr(key))
+               } else {
+                       // There are three ways to squeeze at least one 32 bit pointer into 64 bits.
+                       // Give up and call typedmemmove.
+                       typedmemmove(t.key, insertk, unsafe.Pointer(&key))
+               }
+       } else {
+               *(*uint64)(insertk) = key
+       }
+
        *inserti = top
        h.count++
 
@@ -594,7 +609,7 @@ again:
                val = add(insertk, bucketCnt*2*sys.PtrSize)
        }
 
-       // store new key/value at insert position
+       // store new key at insert position
        *((*stringStruct)(insertk)) = *key
        *inserti = top
        h.count++