runtime: skip netpoll check if there are no waiters

If there are no netpoll waiters then calling netpoll will never find any
goroutines. The later blocking netpoll in findrunnable already has this
optimization.

With golang.org/cl/78538 also applied, this change has a small impact on
latency:

name                             old time/op  new time/op  delta
WakeupParallelSpinning/0s-12     13.6µs ± 1%  13.7µs ± 1%    ~     (p=0.873 n=19+20)
WakeupParallelSpinning/1µs-12    17.7µs ± 0%  17.6µs ± 0%  -0.31%  (p=0.000 n=20+20)
WakeupParallelSpinning/2µs-12    20.2µs ± 2%  19.9µs ± 1%  -1.59%  (p=0.000 n=20+19)
WakeupParallelSpinning/5µs-12    32.0µs ± 1%  32.1µs ± 1%    ~     (p=0.201 n=20+19)
WakeupParallelSpinning/10µs-12   51.7µs ± 0%  51.4µs ± 1%  -0.60%  (p=0.000 n=20+18)
WakeupParallelSpinning/20µs-12   92.2µs ± 0%  92.2µs ± 0%    ~     (p=0.474 n=19+19)
WakeupParallelSpinning/50µs-12    215µs ± 0%   215µs ± 0%    ~     (p=0.319 n=20+19)
WakeupParallelSpinning/100µs-12   330µs ± 2%   331µs ± 2%    ~     (p=0.296 n=20+19)
WakeupParallelSyscall/0s-12       127µs ± 0%   126µs ± 0%  -0.57%  (p=0.000 n=18+18)
WakeupParallelSyscall/1µs-12      129µs ± 0%   128µs ± 1%  -0.43%  (p=0.000 n=18+19)
WakeupParallelSyscall/2µs-12      131µs ± 1%   130µs ± 1%  -0.78%  (p=0.000 n=20+19)
WakeupParallelSyscall/5µs-12      137µs ± 1%   136µs ± 0%  -0.54%  (p=0.000 n=18+19)
WakeupParallelSyscall/10µs-12     147µs ± 1%   146µs ± 0%  -0.58%  (p=0.000 n=18+19)
WakeupParallelSyscall/20µs-12     168µs ± 0%   167µs ± 0%  -0.52%  (p=0.000 n=19+19)
WakeupParallelSyscall/50µs-12     228µs ± 0%   227µs ± 0%  -0.37%  (p=0.000 n=19+18)
WakeupParallelSyscall/100µs-12    329µs ± 0%   328µs ± 0%  -0.28%  (p=0.000 n=20+18)

There is a bigger improvement in CPU utilization. Before this CL, these
benchmarks spent 12% of cycles in netpoll, which are gone after this CL.

This also fixes the sched.lastpoll load, which should be atomic.

Change-Id: I600961460608bd5ba3eeddc599493d2be62064c6
Reviewed-on: https://go-review.googlesource.com/78915
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
Reviewed-by: Austin Clements <austin@google.com>

diff --git a/src/runtime/proc.go b/src/runtime/proc.go
index 2120d647457f23b28e18afdb65b4a8b2fb17a2db..0492a4139ddbad660fdb448d99c2c5daea7c4c1f 100644 (file)
--- a/src/runtime/proc.go
+++ b/src/runtime/proc.go
@@ -2241,11 +2241,12 @@ top:
 
        // Poll network.
        // This netpoll is only an optimization before we resort to stealing.
-       // We can safely skip it if there a thread blocked in netpoll already.
-       // If there is any kind of logical race with that blocked thread
-       // (e.g. it has already returned from netpoll, but does not set lastpoll yet),
-       // this thread will do blocking netpoll below anyway.
-       if netpollinited() && sched.lastpoll != 0 {
+       // We can safely skip it if there are no waiters or a thread is blocked
+       // in netpoll already. If there is any kind of logical race with that
+       // blocked thread (e.g. it has already returned from netpoll, but does
+       // not set lastpoll yet), this thread will do blocking netpoll below
+       // anyway.
+       if netpollinited() && atomic.Load(&netpollWaiters) > 0 && atomic.Load64(&sched.lastpoll) != 0 {
                if gp := netpoll(false); gp != nil { // non-blocking
                        // netpoll returns list of goroutines linked by schedlink.
                        injectglist(gp.schedlink.ptr())