From f8a5ba2a3880f4782d8250fd26dde2baa3990afa Mon Sep 17 00:00:00 2001 From: Russ Cox Date: Tue, 21 May 2019 09:03:26 -0400 Subject: [PATCH] cmd/go: default to GOPROXY=https://proxy.golang.org and GOSUMDB=sum.golang.org This CL changes the default module download and module verification mechanisms to use the Go module mirror and Go checksum database run by Google. See https://proxy.golang.org/privacy for the services' privacy policy. (Today, that URL is a redirect to Google's standard privacy policy, which covers these services as well. If we publish a more specific privacy policy just for these services, that URL will be updated to display or redirect to it.) See 'go help modules' and 'go help modules-auth' for details (added in this CL). To disable the mirror and checksum database for non-public modules: go env -w GONOPROXY=*.private.net,your.com/* go env -w GONOSUMDB=*.private.net,your.com/* (If you are using a private module proxy then you'd only do the second.) If you run into problems with the behavior of the go command when using the Go module mirror or the Go checksum database, please file issues at https://golang.org/issue/new, so that we can address them for the Go 1.13 release. For #25530. This CL also documents GONOPROXY. Fixes #32056. Change-Id: I2fde82e071742272b0842efd9580df1a56947fec Reviewed-on: https://go-review.googlesource.com/c/go/+/178179 Run-TryBot: Russ Cox TryBot-Result: Gobot Gobot Reviewed-by: Bryan C. Mills --- src/cmd/go/alldocs.go | 76 +++++++++++-------- src/cmd/go/internal/cfg/cfg.go | 14 ---- src/cmd/go/internal/modfetch/fetch.go | 18 +++-- src/cmd/go/internal/modfetch/proxy.go | 15 ---- src/cmd/go/internal/modload/help.go | 47 +++++++++--- .../go/testdata/script/mod_sumdb_golang.txt | 8 +- 6 files changed, 96 insertions(+), 82 deletions(-) diff --git a/src/cmd/go/alldocs.go b/src/cmd/go/alldocs.go index 1a7eff29a2..26fb337f86 100644 --- a/src/cmd/go/alldocs.go +++ b/src/cmd/go/alldocs.go @@ -2020,21 +2020,6 @@ // // Module proxy protocol // -// The go command by default downloads modules from version control systems -// directly, just as 'go get' always has. The GOPROXY environment variable allows -// further control over the download source. If GOPROXY is unset, is the empty string, -// or is the string "direct", downloads use the default direct connection to version -// control systems. Setting GOPROXY to "off" disallows downloading modules from -// any source. Otherwise, GOPROXY is expected to be a comma-separated list of -// the URLs of module proxies, in which case the go command will fetch modules -// from those proxies. For each request, the go command tries each proxy in sequence, -// only moving to the next if the current proxy returns a 404 or 410 HTTP response. -// The string "direct" may appear in the proxy list, to cause a direct connection to -// be attempted at that point in the search. -// -// No matter the source of the modules, downloaded modules must match existing -// entries in go.sum (see 'go help modules' for discussion of verification). -// // A Go module proxy is any web server that can respond to GET requests for // URLs of a specified form. The requests have no query parameters, so even // a site serving from a fixed file system (including a file:/// URL) @@ -2591,16 +2576,43 @@ // // Module downloading and verification // -// The go command checks downloads against known checksums, -// to detect unexpected changes in the content of any specific module -// version from one day to the next. See 'go help module-auth' for details. +// The go command can fetch modules from a proxy or connect to source control +// servers directly, according to the setting of the GOPROXY environment +// variable (see 'go help env'). The default setting for GOPROXY is +// "https://proxy.golang.org", the Go module mirror run by Google. +// See https://proxy.golang.org/privacy for the service's privacy policy. +// If GOPROXY is set to the string "direct", downloads use a direct connection +// to source control servers. Setting GOPROXY to "off" disallows downloading +// modules from any source. Otherwise, GOPROXY is expected to be a comma-separated +// list of the URLs of module proxies, in which case the go command will fetch +// modules from those proxies. For each request, the go command tries each proxy +// in sequence, only moving to the next if the current proxy returns a 404 or 410 +// HTTP response. The string "direct" may appear in the proxy list, +// to cause a direct connection to be attempted at that point in the search. +// Any proxies listed after "direct" are never consulted. +// +// The GONOPROXY environment variable is a comma-separated list of +// glob patterns (in the syntax of Go's path.Match) of module path prefixes +// that should always be fetched directly, ignoring the GOPROXY setting. +// For example, +// +// GONOPROXY=*.corp.example.com,rsc.io/private +// +// forces a direct connection to download modules with path prefixes matching +// either pattern, including "git.corp.example.com/xyzzy", "rsc.io/private", +// and "rsc.io/private/quux". // -// The go command can fetch modules from a proxy instead of connecting -// to source control systems directly, according to the setting of the GOPROXY -// environment variable. +// The 'go env -w' command (see 'go help env') can be used to set these variables +// for future go command invocations. // -// See 'go help goproxy' for details about the proxy and also the format of -// the cached downloaded packages. +// No matter the source of the modules, the go command checks downloads against +// known checksums, to detect unexpected changes in the content of any specific +// module version from one day to the next. This check first consults the current +// module's go.sum file but falls back to the Go checksum database. +// See 'go help module-auth' for details. +// +// See 'go help goproxy' for details about the proxy protocol and also +// the format of the cached downloaded packages. // // Modules and vendoring // @@ -2778,18 +2790,17 @@ // database requires giving the public key explicitly. The URL defaults to // "https://" followed by the database name. // -// GOSUMDB defaults to "sum.golang.org" when GOPROXY="https://proxy.golang.org" -// and otherwise defaults to "off". NOTE: The GOSUMDB will later default to -// "sum.golang.org" unconditionally. +// GOSUMDB defaults to "sum.golang.org", the Go checksum database run by Google. +// See https://sum.golang.org/privacy for the service's privacy policy. // // If GOSUMDB is set to "off", or if "go get" is invoked with the -insecure flag, -// the checksum database is never consulted, but at the cost of giving up the -// security guarantee of verified repeatable downloads for all modules. -// A better way to bypass the checksum database for specific modules is -// to use the GONOSUMDB environment variable. +// the checksum database is not consulted, and all unrecognized modules are +// accepted, at the cost of giving up the security guarantee of verified repeatable +// downloads for all modules. A better way to bypass the checksum database +// for specific modules is to use the GONOSUMDB environment variable. // // The GONOSUMDB environment variable is a comma-separated list of -// patterns (in the syntax of Go's path.Match) of module path prefixes +// glob patterns (in the syntax of Go's path.Match) of module path prefixes // that should not be compared against the checksum database. // For example, // @@ -2799,6 +2810,9 @@ // either pattern, including "git.corp.example.com/xyzzy", "rsc.io/private", // and "rsc.io/private/quux". // +// The 'go env -w' command (see 'go help env') can be used to set these variables +// for future go command invocations. +// // // Testing flags // diff --git a/src/cmd/go/internal/cfg/cfg.go b/src/cmd/go/internal/cfg/cfg.go index 2d17d104a6..77d8bab14f 100644 --- a/src/cmd/go/internal/cfg/cfg.go +++ b/src/cmd/go/internal/cfg/cfg.go @@ -303,13 +303,6 @@ func goproxy() string { return v } - // Proxy is off by default for now. - // TODO(rsc): Remove this condition, turning it on always. - // (But do NOT do this without approval from rsc.) - if true { - return "direct" - } - return "https://proxy.golang.org" } @@ -319,13 +312,6 @@ func gosumdb() string { return v } - // Checksum database is off by default except when GOPROXY is proxy.golang.org. - // TODO(rsc): Remove this condition, turning it on always. - // (But do NOT do this without approval from rsc.) - if !strings.HasPrefix(GOPROXY, "https://proxy.golang.org") { - return "off" - } - return "sum.golang.org" } diff --git a/src/cmd/go/internal/modfetch/fetch.go b/src/cmd/go/internal/modfetch/fetch.go index 817f7657e2..d40d2c6fac 100644 --- a/src/cmd/go/internal/modfetch/fetch.go +++ b/src/cmd/go/internal/modfetch/fetch.go @@ -702,18 +702,17 @@ The go command knows the public key of sum.golang.org; use of any other database requires giving the public key explicitly. The URL defaults to "https://" followed by the database name. -GOSUMDB defaults to "sum.golang.org" when GOPROXY="https://proxy.golang.org" -and otherwise defaults to "off". NOTE: The GOSUMDB will later default to -"sum.golang.org" unconditionally. +GOSUMDB defaults to "sum.golang.org", the Go checksum database run by Google. +See https://sum.golang.org/privacy for the service's privacy policy. If GOSUMDB is set to "off", or if "go get" is invoked with the -insecure flag, -the checksum database is never consulted, but at the cost of giving up the -security guarantee of verified repeatable downloads for all modules. -A better way to bypass the checksum database for specific modules is -to use the GONOSUMDB environment variable. +the checksum database is not consulted, and all unrecognized modules are +accepted, at the cost of giving up the security guarantee of verified repeatable +downloads for all modules. A better way to bypass the checksum database +for specific modules is to use the GONOSUMDB environment variable. The GONOSUMDB environment variable is a comma-separated list of -patterns (in the syntax of Go's path.Match) of module path prefixes +glob patterns (in the syntax of Go's path.Match) of module path prefixes that should not be compared against the checksum database. For example, @@ -722,5 +721,8 @@ For example, disables checksum database lookups for modules with path prefixes matching either pattern, including "git.corp.example.com/xyzzy", "rsc.io/private", and "rsc.io/private/quux". + +The 'go env -w' command (see 'go help env') can be used to set these variables +for future go command invocations. `, } diff --git a/src/cmd/go/internal/modfetch/proxy.go b/src/cmd/go/internal/modfetch/proxy.go index 605c72c0ab..5f0432ceed 100644 --- a/src/cmd/go/internal/modfetch/proxy.go +++ b/src/cmd/go/internal/modfetch/proxy.go @@ -32,21 +32,6 @@ var HelpGoproxy = &base.Command{ UsageLine: "goproxy", Short: "module proxy protocol", Long: ` -The go command by default downloads modules from version control systems -directly, just as 'go get' always has. The GOPROXY environment variable allows -further control over the download source. If GOPROXY is unset, is the empty string, -or is the string "direct", downloads use the default direct connection to version -control systems. Setting GOPROXY to "off" disallows downloading modules from -any source. Otherwise, GOPROXY is expected to be a comma-separated list of -the URLs of module proxies, in which case the go command will fetch modules -from those proxies. For each request, the go command tries each proxy in sequence, -only moving to the next if the current proxy returns a 404 or 410 HTTP response. -The string "direct" may appear in the proxy list, to cause a direct connection to -be attempted at that point in the search. - -No matter the source of the modules, downloaded modules must match existing -entries in go.sum (see 'go help modules' for discussion of verification). - A Go module proxy is any web server that can respond to GET requests for URLs of a specified form. The requests have no query parameters, so even a site serving from a fixed file system (including a file:/// URL) diff --git a/src/cmd/go/internal/modload/help.go b/src/cmd/go/internal/modload/help.go index c1685ff08e..96fec8451e 100644 --- a/src/cmd/go/internal/modload/help.go +++ b/src/cmd/go/internal/modload/help.go @@ -328,16 +328,43 @@ module file trees. Module downloading and verification -The go command checks downloads against known checksums, -to detect unexpected changes in the content of any specific module -version from one day to the next. See 'go help module-auth' for details. - -The go command can fetch modules from a proxy instead of connecting -to source control systems directly, according to the setting of the GOPROXY -environment variable. - -See 'go help goproxy' for details about the proxy and also the format of -the cached downloaded packages. +The go command can fetch modules from a proxy or connect to source control +servers directly, according to the setting of the GOPROXY environment +variable (see 'go help env'). The default setting for GOPROXY is +"https://proxy.golang.org", the Go module mirror run by Google. +See https://proxy.golang.org/privacy for the service's privacy policy. +If GOPROXY is set to the string "direct", downloads use a direct connection +to source control servers. Setting GOPROXY to "off" disallows downloading +modules from any source. Otherwise, GOPROXY is expected to be a comma-separated +list of the URLs of module proxies, in which case the go command will fetch +modules from those proxies. For each request, the go command tries each proxy +in sequence, only moving to the next if the current proxy returns a 404 or 410 +HTTP response. The string "direct" may appear in the proxy list, +to cause a direct connection to be attempted at that point in the search. +Any proxies listed after "direct" are never consulted. + +The GONOPROXY environment variable is a comma-separated list of +glob patterns (in the syntax of Go's path.Match) of module path prefixes +that should always be fetched directly, ignoring the GOPROXY setting. +For example, + + GONOPROXY=*.corp.example.com,rsc.io/private + +forces a direct connection to download modules with path prefixes matching +either pattern, including "git.corp.example.com/xyzzy", "rsc.io/private", +and "rsc.io/private/quux". + +The 'go env -w' command (see 'go help env') can be used to set these variables +for future go command invocations. + +No matter the source of the modules, the go command checks downloads against +known checksums, to detect unexpected changes in the content of any specific +module version from one day to the next. This check first consults the current +module's go.sum file but falls back to the Go checksum database. +See 'go help module-auth' for details. + +See 'go help goproxy' for details about the proxy protocol and also +the format of the cached downloaded packages. Modules and vendoring diff --git a/src/cmd/go/testdata/script/mod_sumdb_golang.txt b/src/cmd/go/testdata/script/mod_sumdb_golang.txt index ca040c5dcf..0eb0fc84a7 100644 --- a/src/cmd/go/testdata/script/mod_sumdb_golang.txt +++ b/src/cmd/go/testdata/script/mod_sumdb_golang.txt @@ -1,16 +1,16 @@ -[!net] skip - +# Test default GOPROXY and GOSUMDB env GOPROXY= env GOSUMDB= go env GOPROXY -stdout '^direct$' +stdout '^https://proxy.golang.org$' go env GOSUMDB -stdout '^off$' +stdout '^sum.golang.org$' env GOPROXY=https://proxy.golang.org go env GOSUMDB stdout '^sum.golang.org$' # download direct from github +[!net] skip env GOSUMDB=sum.golang.org env GOPROXY=direct go get -m rsc.io/quote -- 2.50.0