From: Austin Clements Date: Mon, 13 Aug 2018 20:30:54 +0000 (-0400) Subject: runtime: implement STW GC in terms of concurrent GC X-Git-Tag: go1.12beta1~941 X-Git-Url: http://www.git.cypherpunks.su/?a=commitdiff_plain;h=1678b2c580584fa1ea8de3c14df0d8d77b6f7387;p=gostls13.git runtime: implement STW GC in terms of concurrent GC Currently, STW GC works very differently from concurrent GC. The largest differences in that in concurrent GC, all marking work is done by background mark workers during the mark phase, while in STW GC, all marking work is done by gchelper during the mark termination phase. This is a consequence of the evolution of Go's GC from a STW GC by incrementally moving work from STW mark termination into concurrent mark. However, at this point, the STW code paths exist only as a debugging mode. Having separate code paths for this increases the maintenance burden and complexity of the garbage collector. At the same time, these code paths aren't tested nearly as well, making it far more likely that they will bit-rot. This CL reverses the relationship between STW GC, by re-implementing STW GC in terms of concurrent GC. This builds on the new scheduled support for disabling user goroutine scheduling. During sweep termination, it disables user scheduling, so when the GC starts the world again for concurrent mark, it's really only "concurrent" with itself. There are several code paths that were specific to STW GC that are now vestigial. We'll remove these in the follow-up CLs. Updates #26903. Change-Id: Ia3883d2fcf7ab1d89bdc9c8ee54bf9bffb32c096 Reviewed-on: https://go-review.googlesource.com/c/134780 Run-TryBot: Austin Clements TryBot-Result: Gobot Gobot Reviewed-by: Rick Hudson --- diff --git a/src/runtime/mgc.go b/src/runtime/mgc.go index b685415872..0bdff3d657 100644 --- a/src/runtime/mgc.go +++ b/src/runtime/mgc.go @@ -455,6 +455,12 @@ func (c *gcControllerState) startCycle() { c.fractionalUtilizationGoal = 0 } + // In STW mode, we just want dedicated workers. + if debug.gcstoptheworld > 0 { + c.dedicatedMarkWorkersNeeded = int64(gomaxprocs) + c.fractionalUtilizationGoal = 0 + } + // Clear per-P state for _, p := range allp { p.gcAssistTime = 0 @@ -1264,9 +1270,7 @@ func gcStart(trigger gcTrigger) { traceGCStart() } - if mode == gcBackgroundMode { - gcBgMarkStartWorkers() - } + gcBgMarkStartWorkers() gcResetMarkState() @@ -1296,65 +1300,65 @@ func gcStart(trigger gcTrigger) { clearpools() work.cycles++ - if mode == gcBackgroundMode { // Do as much work concurrently as possible - gcController.startCycle() - work.heapGoal = memstats.next_gc - // Enter concurrent mark phase and enable - // write barriers. - // - // Because the world is stopped, all Ps will - // observe that write barriers are enabled by - // the time we start the world and begin - // scanning. - // - // Write barriers must be enabled before assists are - // enabled because they must be enabled before - // any non-leaf heap objects are marked. Since - // allocations are blocked until assists can - // happen, we want enable assists as early as - // possible. - setGCPhase(_GCmark) - - gcBgMarkPrepare() // Must happen before assist enable. - gcMarkRootPrepare() - - // Mark all active tinyalloc blocks. Since we're - // allocating from these, they need to be black like - // other allocations. The alternative is to blacken - // the tiny block on every allocation from it, which - // would slow down the tiny allocator. - gcMarkTinyAllocs() - - // At this point all Ps have enabled the write - // barrier, thus maintaining the no white to - // black invariant. Enable mutator assists to - // put back-pressure on fast allocating - // mutators. - atomic.Store(&gcBlackenEnabled, 1) - - // Assists and workers can start the moment we start - // the world. - gcController.markStartTime = now - - // Concurrent mark. - systemstack(func() { - now = startTheWorldWithSema(trace.enabled) - }) + gcController.startCycle() + work.heapGoal = memstats.next_gc + + // In STW mode, disable scheduling of user Gs. This may also + // disable scheduling of this goroutine, so it may block as + // soon as we start the world again. + if mode != gcBackgroundMode { + schedEnableUser(false) + } + + // Enter concurrent mark phase and enable + // write barriers. + // + // Because the world is stopped, all Ps will + // observe that write barriers are enabled by + // the time we start the world and begin + // scanning. + // + // Write barriers must be enabled before assists are + // enabled because they must be enabled before + // any non-leaf heap objects are marked. Since + // allocations are blocked until assists can + // happen, we want enable assists as early as + // possible. + setGCPhase(_GCmark) + + gcBgMarkPrepare() // Must happen before assist enable. + gcMarkRootPrepare() + + // Mark all active tinyalloc blocks. Since we're + // allocating from these, they need to be black like + // other allocations. The alternative is to blacken + // the tiny block on every allocation from it, which + // would slow down the tiny allocator. + gcMarkTinyAllocs() + + // At this point all Ps have enabled the write + // barrier, thus maintaining the no white to + // black invariant. Enable mutator assists to + // put back-pressure on fast allocating + // mutators. + atomic.Store(&gcBlackenEnabled, 1) + + // Assists and workers can start the moment we start + // the world. + gcController.markStartTime = now + + // Concurrent mark. + systemstack(func() { + now = startTheWorldWithSema(trace.enabled) work.pauseNS += now - work.pauseStart work.tMark = now - } else { - if trace.enabled { - // Switch to mark termination STW. - traceGCSTWDone() - traceGCSTWStart(0) - } - t := nanotime() - work.tMark, work.tMarkTerm = t, t - work.heapGoal = work.heap0 - - // Perform mark termination. This will restart the world. - gcMarkTermination(memstats.triggerRatio) + }) + // In STW mode, we could block the instant systemstack + // returns, so don't do anything important here. Make sure we + // block rather than returning to user code. + if mode != gcBackgroundMode { + Gosched() } semrelease(&work.startSema) @@ -1468,6 +1472,10 @@ top: // world again. semrelease(&work.markDoneSema) + // In STW mode, re-enable user goroutines. These will be + // queued to run after we start the world. + schedEnableUser(true) + // endCycle depends on all gcWork cache stats being flushed. // The termination algorithm above ensured that up to // allocations since the ragged barrier.