runtime: flush write barrier buffer to create work

Currently, if the gcWork runs out of work, we'll fall out of the GC
worker, even though flushing the write barrier buffer could produce
more work. While this is not a correctness issue, it can lead to
premature mark 2 or mark termination.

Fix this by flushing the write barrier buffer if the local gcWork runs
out of work and then checking the local gcWork again.

This reduces the number of premature mark terminations during all.bash
by about a factor of 10.

Updates #26903. This is preparation for eliminating mark 2.

Change-Id: I48577220b90c86bfd28d498e8603bc379a8cd617
Reviewed-on: https://go-review.googlesource.com/c/134315
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Rick Hudson <rlh@golang.org>

diff --git a/src/runtime/mgcmark.go b/src/runtime/mgcmark.go
index f713841cfa856b1213bb79f5e45ee97ea8ccc821..bf69172f6a3b75f7d9fbf3ec5b066ca31fde683d 100644 (file)
--- a/src/runtime/mgcmark.go
+++ b/src/runtime/mgcmark.go
@@ -882,6 +882,13 @@ func gcDrain(gcw *gcWork, flags gcDrainFlags) {
                        b = gcw.tryGetFast()
                        if b == 0 {
                                b = gcw.tryGet()
+                               if b == 0 {
+                                       // Flush the write barrier
+                                       // buffer; this may create
+                                       // more work.
+                                       wbBufFlush(nil, 0)
+                                       b = gcw.tryGet()
+                               }
                        }
                }
                if b == 0 {
@@ -963,6 +970,12 @@ func gcDrainN(gcw *gcWork, scanWork int64) int64 {
                b := gcw.tryGetFast()
                if b == 0 {
                        b = gcw.tryGet()
+                       if b == 0 {
+                               // Flush the write barrier buffer;
+                               // this may create more work.
+                               wbBufFlush(nil, 0)
+                               b = gcw.tryGet()
+                       }
                }
 
                if b == 0 {