runtime: close window that hides GC work from concurrent mark

Currently we enter mark 2 by first flushing all existing gcWork caches
and then setting gcBlackenPromptly, which disables further gcWork
caching. However, if a worker or assist pulls a work buffer in to its
gcWork cache after that cache has been flushed but before caching is
disabled, that work may remain in that cache until mark termination.
If that work represents a heap bottleneck (e.g., a single pointer that
is the only way to reach a large amount of the heap), this can force
mark termination to do a large amount of work, resulting in a long
STW.

Fix this by reversing the order of these steps: first disable caching,
then flush all existing caches.

Rick Hudson <rlh> did the hard work of tracking this down. This CL
combined with CL 12672 and CL 12646 distills the critical parts of his
fix from CL 12539.

Fixes #11694.

Change-Id: Ib10d0a21e3f6170a80727d0286f9990df049fed2
Reviewed-on: https://go-review.googlesource.com/12688
Reviewed-by: Rick Hudson <rlh@golang.org>

diff --git a/src/runtime/mgc.go b/src/runtime/mgc.go
index 7b224fedfa9392de3ae6b3735c9da15e9bed9c2a..620625754a5c8e6298defdd03a00857e0c89e9a0 100644 (file)
--- a/src/runtime/mgc.go
+++ b/src/runtime/mgc.go
@@ -1003,12 +1003,18 @@ func gc(mode int) {
                        // rescan global data and bss.
                        markroot(nil, _RootData)
                        markroot(nil, _RootBss)
+
+                       // Disallow caching workbufs.
+                       gcBlackenPromptly = true
+
+                       // Flush all currently cached workbufs. This
+                       // also forces any remaining background
+                       // workers out of their loop.
                        forEachP(func(_p_ *p) {
                                _p_.gcw.dispose()
                        })
                })
 
-               gcBlackenPromptly = true
                // Wait for this more aggressive background mark to complete.
                work.bgMark2.clear()
                work.bgMark2.wait()