runtime: fix over-aggressive proportional sweep

Currently, sweeping is performed before allocating a span by charging
for the entire size of the span requested, rather than the number of
bytes actually available for allocation from the returned span. That
is, if the returned span is 8K, but already has 6K in use, the mutator
is charged for 8K of heap allocation even though it can only allocate
2K more from the span. As a result, proportional sweep is
over-aggressive and tends to finish much earlier than it needs to.
This effect is more amplified by fragmented heaps.

Fix this by reimbursing the mutator for the used space in a span once
it has allocated that span. We still have to charge up-front for the
worst-case because we don't know which span the mutator will get, but
at least we can correct the over-charge once it has a span, which will
go toward later span allocations.

This has negligible effect on the throughput of the go1 benchmarks and
the garbage benchmark.

Fixes #12040.

Change-Id: I0e23e7a4ccf126cca000fed5067b20017028dd6b
Reviewed-on: https://go-review.googlesource.com/16515
Reviewed-by: Rick Hudson <rlh@golang.org>

diff --git a/src/runtime/mcentral.go b/src/runtime/mcentral.go
index 1e1f6fd13d7600a081e490df3f0772dee599ae9d..a09801a5490b8cb6075a6d0ec2cd06fa334a6fc6 100644 (file)
--- a/src/runtime/mcentral.go
+++ b/src/runtime/mcentral.go
@@ -101,6 +101,10 @@ havespan:
        if n == 0 {
                throw("empty span")
        }
+       usedBytes := uintptr(s.ref) * s.elemsize
+       if usedBytes > 0 {
+               reimburseSweepCredit(usedBytes)
+       }
        if s.freelist.ptr() == nil {
                throw("freelist empty")
        }

diff --git a/src/runtime/mgcsweep.go b/src/runtime/mgcsweep.go
index fc92301857632ec3d99fb8b52f2bbf1b85cd12a0..84b6181b6e840cc7c54f97d8ace8bb23ff13c54d 100644 (file)
--- a/src/runtime/mgcsweep.go
+++ b/src/runtime/mgcsweep.go
@@ -354,6 +354,11 @@ func mSpan_Sweep(s *mspan, preserve bool) bool {
 // also sweep pages (e.g., for a large allocation), it can pass a
 // non-zero callerSweepPages to leave that many pages unswept.
 //
+// deductSweepCredit makes a worst-case assumption that all spanBytes
+// bytes of the ultimately allocated span will be available for object
+// allocation. The caller should call reimburseSweepCredit if that
+// turns out not to be the case once the span is allocated.
+//
 // deductSweepCredit is the core of the "proportional sweep" system.
 // It uses statistics gathered by the garbage collector to perform
 // enough sweeping so that all pages are swept during the concurrent
@@ -379,6 +384,17 @@ func deductSweepCredit(spanBytes uintptr, callerSweepPages uintptr) {
        }
 }
 
+// reimburseSweepCredit records that unusableBytes bytes of a
+// just-allocated span are not available for object allocation. This
+// offsets the worst-case charge performed by deductSweepCredit.
+func reimburseSweepCredit(unusableBytes uintptr) {
+       if mheap_.sweepPagesPerByte == 0 {
+               // Nobody cares about the credit. Avoid the atomic.
+               return
+       }
+       xadd64(&mheap_.spanBytesAlloc, -int64(unusableBytes))
+}
+
 func dumpFreeList(s *mspan) {
        printlock()
        print("runtime: free list of span ", s, ":\n")