runtime: move concurrent mark setup off system stack

For historical reasons we currently do a lot of the concurrent mark
setup on the system stack. In fact, at this point the one and only
thing that needs to happen on the system stack is the start-the-world.

Clean up this code by lifting everything other than the
start-the-world off the system stack.

The diff for this change looks large, but the only code change is to
narrow the systemstack call. Everything else is re-indentation.

Change-Id: I1e03b8afc759fad726f2397b05a17d183c2713ce
Reviewed-on: https://go-review.googlesource.com/16354
Reviewed-by: Rick Hudson <rlh@golang.org>
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>

diff --git a/src/runtime/mgc.go b/src/runtime/mgc.go
index 4a24a50a9f896c6ffb0964f4a4647fd91edc11a9..b70d91412502d8dc996ace4fa5c4a53504125128 100644 (file)
--- a/src/runtime/mgc.go
+++ b/src/runtime/mgc.go
@@ -1040,54 +1040,52 @@ func gc(mode gcMode) {
                gcController.startCycle()
                work.heapGoal = gcController.heapGoal
 
-               systemstack(func() {
-                       // Enter concurrent mark phase and enable
-                       // write barriers.
-                       //
-                       // Because the world is stopped, all Ps will
-                       // observe that write barriers are enabled by
-                       // the time we start the world and begin
-                       // scanning.
-                       //
-                       // It's necessary to enable write barriers
-                       // during the scan phase for several reasons:
-                       //
-                       // They must be enabled for writes to higher
-                       // stack frames before we scan stacks and
-                       // install stack barriers because this is how
-                       // we track writes to inactive stack frames.
-                       // (Alternatively, we could not install stack
-                       // barriers over frame boundaries with
-                       // up-pointers).
-                       //
-                       // They must be enabled before assists are
-                       // enabled because they must be enabled before
-                       // any non-leaf heap objects are marked. Since
-                       // allocations are blocked until assists can
-                       // happen, we want enable assists as early as
-                       // possible.
-                       setGCPhase(_GCmark)
-
-                       // markrootSpans uses work.spans, so make sure
-                       // it is up to date.
-                       gcCopySpans()
-
-                       gcBgMarkPrepare() // Must happen before assist enable.
-                       gcMarkRootPrepare()
-
-                       // At this point all Ps have enabled the write
-                       // barrier, thus maintaining the no white to
-                       // black invariant. Enable mutator assists to
-                       // put back-pressure on fast allocating
-                       // mutators.
-                       atomicstore(&gcBlackenEnabled, 1)
-
-                       // Concurrent mark.
-                       startTheWorldWithSema()
-                       now = nanotime()
-                       work.pauseNS += now - work.pauseStart
-                       gcController.assistStartTime = now
-               })
+               // Enter concurrent mark phase and enable
+               // write barriers.
+               //
+               // Because the world is stopped, all Ps will
+               // observe that write barriers are enabled by
+               // the time we start the world and begin
+               // scanning.
+               //
+               // It's necessary to enable write barriers
+               // during the scan phase for several reasons:
+               //
+               // They must be enabled for writes to higher
+               // stack frames before we scan stacks and
+               // install stack barriers because this is how
+               // we track writes to inactive stack frames.
+               // (Alternatively, we could not install stack
+               // barriers over frame boundaries with
+               // up-pointers).
+               //
+               // They must be enabled before assists are
+               // enabled because they must be enabled before
+               // any non-leaf heap objects are marked. Since
+               // allocations are blocked until assists can
+               // happen, we want enable assists as early as
+               // possible.
+               setGCPhase(_GCmark)
+
+               // markrootSpans uses work.spans, so make sure
+               // it is up to date.
+               gcCopySpans()
+
+               gcBgMarkPrepare() // Must happen before assist enable.
+               gcMarkRootPrepare()
+
+               // At this point all Ps have enabled the write
+               // barrier, thus maintaining the no white to
+               // black invariant. Enable mutator assists to
+               // put back-pressure on fast allocating
+               // mutators.
+               atomicstore(&gcBlackenEnabled, 1)
+
+               // Concurrent mark.
+               systemstack(startTheWorldWithSema)
+               now = nanotime()
+               work.pauseNS += now - work.pauseStart
+               gcController.assistStartTime = now
                work.tMark = now
 
                // Enable background mark workers and wait for