runtime: steal timers from running P's

Previously we did not steal timers from running P's, because that P
should be responsible for running its own timers. However, if the P
is running a CPU-bound G, this can cause measurable delays in running
ready timers. Also, in CL 214185 we avoided taking the timer lock of a P
with no ready timers, which reduces the chances of timer lock contention.

So, if we can't find any ready timers on sleeping P's, try stealing
them from running P's.

Fixes #38860

Change-Id: I0bf1d5dc56258838bdacccbf89493524e23d7fed
Reviewed-on: https://go-review.googlesource.com/c/go/+/232199
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>

diff --git a/src/runtime/proc.go b/src/runtime/proc.go
index 766784c07edac57d0bbc1042302df23deb8104e7..2399f0a1d3d0628b92022dec930effea21c4f993 100644 (file)
--- a/src/runtime/proc.go
+++ b/src/runtime/proc.go
@@ -2231,11 +2231,14 @@ top:
                        // Consider stealing timers from p2.
                        // This call to checkTimers is the only place where
                        // we hold a lock on a different P's timers.
-                       // Lock contention can be a problem here, so avoid
-                       // grabbing the lock if p2 is running and not marked
-                       // for preemption. If p2 is running and not being
-                       // preempted we assume it will handle its own timers.
-                       if i > 2 && shouldStealTimers(p2) {
+                       // Lock contention can be a problem here, so
+                       // initially avoid grabbing the lock if p2 is running
+                       // and is not marked for preemption. If p2 is running
+                       // and not being preempted we assume it will handle its
+                       // own timers.
+                       // If we're still looking for work after checking all
+                       // the P's, then go ahead and steal from an active P.
+                       if i > 2 || (i > 1 && shouldStealTimers(p2)) {
                                tnow, w, ran := checkTimers(p2, now)
                                now = tnow
                                if w != 0 && (pollUntil == 0 || w < pollUntil) {