runtime: make assists preemptible

Currently, assists are non-preemptible, which means a heavily
assisting G can block other Gs from running. At the beginning of a GC
cycle, it can also delay scang, which will spin until the assist is
done. Since scanning is currently done sequentially, this can
seriously extend the length of the scan phase.

Fix this by making assists preemptible. Since the assist holds work
buffers and runs on the system stack, this must be done cooperatively:
we make gcDrainN return on preemption, and make the assist return from
the system stack and voluntarily Gosched.

This is prerequisite to enlarging the work buffers. Without this
change, the delays and spinning in scang increase significantly.

This has no effect on the go1 benchmarks.

name              old time/op  new time/op  delta
XBenchGarbage-12  5.72ms ± 4%  5.37ms ± 5%  -6.11%  (p=0.000 n=20+20)

Change-Id: I829e732a0f23b126da633516a1a9ec1a508fdbf1
Reviewed-on: https://go-review.googlesource.com/15894
Reviewed-by: Rick Hudson <rlh@golang.org>
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>

diff --git a/src/runtime/mgcmark.go b/src/runtime/mgcmark.go
index ab1af218a55e3b8e5e0ea2eb633d9e58cb5e32e5..ed8633c30f78cde8a04c9eb19a5d55248238c7ae 100644 (file)
--- a/src/runtime/mgcmark.go
+++ b/src/runtime/mgcmark.go
@@ -440,6 +440,13 @@ retry:
                // do one of these before letting the mutator allocate
                // more to prevent over-allocation.
                //
+               // If this is because we were preempted, reschedule
+               // and try some more.
+               if gp.preempt {
+                       Gosched()
+                       goto retry
+               }
+
                // Add this G to an assist queue and park. When the GC
                // has more background credit, it will satisfy queued
                // assists before flushing to the global credit pool.
@@ -845,11 +852,11 @@ func gcDrain(gcw *gcWork, flags gcDrainFlags) {
 }
 
 // gcDrainN blackens grey objects until it has performed roughly
-// scanWork units of scan work. This is best-effort, so it may perform
-// less work if it fails to get a work buffer. Otherwise, it will
-// perform at least n units of work, but may perform more because
-// scanning is always done in whole object increments. It returns the
-// amount of scan work performed.
+// scanWork units of scan work or the G is preempted. This is
+// best-effort, so it may perform less work if it fails to get a work
+// buffer. Otherwise, it will perform at least n units of work, but
+// may perform more because scanning is always done in whole object
+// increments. It returns the amount of scan work performed.
 //go:nowritebarrier
 func gcDrainN(gcw *gcWork, scanWork int64) int64 {
        if !writeBarrierEnabled {
@@ -860,7 +867,8 @@ func gcDrainN(gcw *gcWork, scanWork int64) int64 {
        // want to claim was done by this call.
        workFlushed := -gcw.scanWork
 
-       for workFlushed+gcw.scanWork < scanWork {
+       gp := getg().m.curg
+       for !gp.preempt && workFlushed+gcw.scanWork < scanWork {
                // This might be a good place to add prefetch code...
                // if(wbuf.nobj > 4) {
                //         PREFETCH(wbuf->obj[wbuf.nobj - 3];