From: Austin Clements Date: Sat, 26 Sep 2015 18:00:57 +0000 (-0400) Subject: runtime: remove sweep wait loop in finishsweep_m X-Git-Tag: go1.6beta1~925 X-Git-Url: http://www.git.cypherpunks.su/?a=commitdiff_plain;h=9a31d38f65354567719a31414ebcb31f0337b803;p=gostls13.git runtime: remove sweep wait loop in finishsweep_m In general, finishsweep_m must block until any spans that are concurrently being swept have been swept. It accomplishes this by looping over all spans, which, as in the previous commit, takes ~1ms/heap GB. Unfortunately, we do this during the STW sweep termination phase, so multi-gigabyte heaps can push our STW time past 10ms. However, there's no need to do this wait if the world is stopped because, in effect, stopping the world already had to wait for anything that was sweeping (and if it didn't, the wait in finishsweep_m would deadlock). Hence, we can simply skip this loop if the world is stopped, such as during sweep termination. In fact, currently all calls to finishsweep_m are STW, but this hasn't always been the case and may not be the case in the future, so we keep the logic around. For 24GB heaps, this reduces max pause time by 75% relative to tip and by 90% relative to Go 1.5. Notably, all pauses are now well under 10ms. Here are the results for the garbage benchmark: ------------- max pause ------------ Heap Procs after change before change 1.5.1 24GB 12 3.8ms 16ms 37ms 24GB 4 3.7ms 16ms 37ms 4GB 4 3.7ms 3ms 6.9ms In the 4GB/4P case, it seems the "before change" run got lucky: the max went up, but the 99%ile pause time went down from 3ms to 2.04ms. Change-Id: Ica22189559f231d408ef2815019c9dbb5f38bf31 Reviewed-on: https://go-review.googlesource.com/15071 Reviewed-by: Rick Hudson Run-TryBot: Austin Clements TryBot-Result: Gobot Gobot --- diff --git a/src/runtime/mgc.go b/src/runtime/mgc.go index efb8a32dfa..b95289aaa6 100644 --- a/src/runtime/mgc.go +++ b/src/runtime/mgc.go @@ -937,7 +937,10 @@ func gc(mode gcMode) { pauseStart = now systemstack(stopTheWorldWithSema) - systemstack(finishsweep_m) // finish sweep before we start concurrent scan. + // Finish sweep before we start concurrent scan. + systemstack(func() { + finishsweep_m(true) + }) // clearpools before we start the GC. If we wait they memory will not be // reclaimed until the next GC cycle. clearpools() @@ -1127,7 +1130,7 @@ func gc(mode gcMode) { // Reset these so that all stacks will be rescanned. gcResetGState() gcResetMarkState() - finishsweep_m() + finishsweep_m(true) // Still in STW but gcphase is _GCoff, reset to _GCmarktermination // At this point all objects will be found during the gcMark which diff --git a/src/runtime/mgcsweep.go b/src/runtime/mgcsweep.go index d9eb58fdf7..9468af941a 100644 --- a/src/runtime/mgcsweep.go +++ b/src/runtime/mgcsweep.go @@ -24,20 +24,28 @@ type sweepdata struct { } //go:nowritebarrier -func finishsweep_m() { - // The world is stopped so we should be able to complete the sweeps - // quickly. +func finishsweep_m(stw bool) { + // Sweeping must be complete before marking commences, so + // sweep any unswept spans. If this is a concurrent GC, there + // shouldn't be any spans left to sweep, so this should finish + // instantly. If GC was forced before the concurrent sweep + // finished, there may be spans to sweep. for sweepone() != ^uintptr(0) { sweep.npausesweep++ } // There may be some other spans being swept concurrently that // we need to wait for. If finishsweep_m is done with the world stopped - // this code is not required. - sg := mheap_.sweepgen - for _, s := range work.spans { - if s.sweepgen != sg && s.state == _MSpanInUse { - mSpan_EnsureSwept(s) + // this is not required because the STW must have waited for sweeps. + // + // TODO(austin): As of this writing, we always pass true for stw. + // Consider removing this code. + if !stw { + sg := mheap_.sweepgen + for _, s := range work.spans { + if s.sweepgen != sg && s.state == _MSpanInUse { + mSpan_EnsureSwept(s) + } } } }