cmd/api: limit concurrent 'go list' calls to GOMAXPROCS

Each invocation of 'go list' may consume a significant quantity of
system resources, including buffers for reading files and RAM for the
runtime's memory footprint.
Very small builders may even hit swap as a result of that load,
further exacerbating resource contention.

To avoid overloading small builders, restrict 'go list' calls to
runtime.GOMAXPROCS as it is set at the first call to loadImports.

This also somewhat improves running time even on larger machines: on
my workstation, this change reduces the wall time for 'go test
cmd/api' by around 100ms.

Updates #38537

Change-Id: I968e0f961a8f1d84c27e1ab8b621b9670dcfd448
Reviewed-on: https://go-review.googlesource.com/c/go/+/228998
Run-TryBot: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

diff --git a/src/cmd/api/goapi.go b/src/cmd/api/goapi.go
index 9874b29292ce72bbcae567acf3f7c55849cb6e22..01b17b8839209d3ab242364ae25d78e6011f53af 100644 (file)
--- a/src/cmd/api/goapi.go
+++ b/src/cmd/api/goapi.go
@@ -444,6 +444,11 @@ type listImports struct {
 
 var listCache sync.Map // map[string]listImports, keyed by contextName
 
+// listSem is a semaphore restricting concurrent invocations of 'go list'.
+var listSem = make(chan semToken, runtime.GOMAXPROCS(0))
+
+type semToken struct{}
+
 // loadImports populates w with information about the packages in the standard
 // library and the packages they themselves import in w's build context.
 //
@@ -468,6 +473,9 @@ func (w *Walker) loadImports() {
 
        imports, ok := listCache.Load(name)
        if !ok {
+               listSem <- semToken{}
+               defer func() { <-listSem }()
+
                cmd := exec.Command(goCmd(), "list", "-e", "-deps", "-json", "std")
                cmd.Env = listEnv(w.context)
                out, err := cmd.CombinedOutput()