runtime: make sure heapBitsBulkBarrier cannot be preempted

Changes the torture test in #12068 from failing about 1/10 times
to not failing in almost 2,000 runs.

This was only happening in -race mode because functions are
bigger in -race mode, so a few of the helpers for heapBitsBulkBarrier
were not being inlined, and they were not marked nosplit,
so (only in -race mode) the write barrier was being preempted by GC,
causing missed pointer updates.

Filed issue #12069 for diagnosis of any other similar errors.

Fixes #12068.

Change-Id: Ic174d9b050ba278b18b08ab0d85a73c33bd5b175
Reviewed-on: https://go-review.googlesource.com/13364
Reviewed-by: Austin Clements <austin@google.com>

diff --git a/src/runtime/mbitmap.go b/src/runtime/mbitmap.go
index c439158f078080f50d43685838cd24e6df01677d..fd6b4b11a2f8e31eada566981e03cf9f1968c369 100644 (file)
--- a/src/runtime/mbitmap.go
+++ b/src/runtime/mbitmap.go
@@ -109,6 +109,9 @@ func add1(p *byte) *byte {
 
 // subtract1 returns the byte pointer p-1.
 //go:nowritebarrier
+//
+// nosplit because it is used during write barriers and must not be preempted.
+//go:nosplit
 func subtract1(p *byte) *byte {
        // Note: wrote out full expression instead of calling subtractb(p, 1)
        // to reduce the number of temporaries generated by the
@@ -245,6 +248,9 @@ func (h heapBits) prefetch() {
 // next returns the heapBits describing the next pointer-sized word in memory.
 // That is, if h describes address p, h.next() describes p+ptrSize.
 // Note that next does not modify h. The caller must record the result.
+//
+// nosplit because it is used during write barriers and must not be preempted.
+//go:nosplit
 func (h heapBits) next() heapBits {
        if h.shift < 3*heapBitsShift {
                return heapBits{h.bitp, h.shift + heapBitsShift}
@@ -293,6 +299,9 @@ func (h heapBits) setMarkedNonAtomic() {
 
 // isPointer reports whether the heap bits describe a pointer word.
 // h must describe the initial word of the object.
+//
+// nosplit because it is used during write barriers and must not be preempted.
+//go:nosplit
 func (h heapBits) isPointer() bool {
        return (*h.bitp>>h.shift)&bitPointer != 0
 }