runtime: add credit system for scavenging

When scavenging small amounts it's possible we over-scavenge by a
significant margin since we choose to scavenge the largest spans first.
This over-scavenging is never accounted for.

With this change, we add a scavenge credit pool, similar to the reclaim
credit pool. Any time scavenging triggered by RSS growth starts up, it
checks if it can cash in some credit first. If after using all the
credit it still needs to scavenge, then any extra it does it adds back
into the credit pool.

This change mitigates the performance impact of golang.org/cl/159500 on
the Garbage benchmark. On Go1 it suggests some improvements, but most of
that is within the realm of noise (Revcomp seems very sensitive to
GC-related changes, both postively and negatively).

Garbage: https://perf.golang.org/search?q=upload:20190131.5
Go1:     https://perf.golang.org/search?q=upload:20190131.4

Performance change with both changes:

Garbage: https://perf.golang.org/search?q=upload:20190131.7
Go1:     https://perf.golang.org/search?q=upload:20190131.6

Change-Id: I87bd3c183e71656fdafef94714194b9fdbb77aa2
Reviewed-on: https://go-review.googlesource.com/c/160297
Reviewed-by: Austin Clements <austin@google.com>

diff --git a/src/runtime/mheap.go b/src/runtime/mheap.go
index 055dfeed99fc1985817075d1f69b1127abab41d8..47e3a333911411b8c3d060bd6a89a6a833e3d1db 100644 (file)
--- a/src/runtime/mheap.go
+++ b/src/runtime/mheap.go
@@ -107,6 +107,14 @@ type mheap struct {
        // This is accessed atomically.
        reclaimCredit uintptr
 
+       // scavengeCredit is spare credit for extra bytes scavenged.
+       // Since the scavenging mechanisms operate on spans, it may
+       // scavenge more than requested. Any spare pages released
+       // go to this credit pool.
+       //
+       // This is protected by the mheap lock.
+       scavengeCredit uintptr
+
        // Malloc stats.
        largealloc  uint64                  // bytes allocated for large objects
        nlargealloc uint64                  // number of large object allocations
@@ -165,7 +173,7 @@ type mheap struct {
        // simply blocking GC (by disabling preemption).
        sweepArenas []arenaIdx
 
-       _ uint32 // ensure 64-bit alignment of central
+       // _ uint32 // ensure 64-bit alignment of central
 
        // central free lists for small size classes.
        // the padding makes sure that the mcentrals are
@@ -1349,6 +1357,14 @@ func (h *mheap) freeSpanLocked(s *mspan, acctinuse, acctidle bool, unusedsince i
 // starting from the largest span and working down. It then takes those spans
 // and places them in scav. h must be locked.
 func (h *mheap) scavengeLargest(nbytes uintptr) {
+       // Use up scavenge credit if there's any available.
+       if nbytes > h.scavengeCredit {
+               nbytes -= h.scavengeCredit
+               h.scavengeCredit = 0
+       } else {
+               h.scavengeCredit -= nbytes
+               return
+       }
        // Iterate over the treap backwards (from largest to smallest) scavenging spans
        // until we've reached our quota of nbytes.
        released := uintptr(0)
@@ -1377,6 +1393,10 @@ func (h *mheap) scavengeLargest(nbytes uintptr) {
                h.scav.insert(s)
                released += r
        }
+       // If we over-scavenged, turn that extra amount into credit.
+       if released > nbytes {
+               h.scavengeCredit += released - nbytes
+       }
 }
 
 // scavengeAll visits each node in the unscav treap and scavenges the