Cherry Zhang [Wed, 12 Aug 2020 21:37:18 +0000 (17:37 -0400)]
cmd: merge branch 'dev.link' into master
In the dev.link branch we have continued developing the new object
file format support and the linker improvements described in
https://golang.org/s/better-linker . Since the last merge (May 1st
2020), more progress has been made to improve the new linker, with
improvements on both linker speed and memory usage.
Michael Matloob [Wed, 17 Jun 2020 19:50:14 +0000 (15:50 -0400)]
cmd/go: propagate context into PackagesForBuild and Do for tracing
This change propagates context into PackagesForErrors and Do for
the purpose of tracing, and calls trace.StartSpan on PackagesForErrors
and Do, so that the trace now shows the broad outline of where
the "Loading" and "Execution" phases are in the build.
Updates #38714
Change-Id: Ib9a7cf7030210f68f76663d1c8a7461e0a226611
Reviewed-on: https://go-review.googlesource.com/c/go/+/238541
Run-TryBot: Michael Matloob <matloob@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Jay Conrod <jayconrod@google.com> Reviewed-by: Bryan C. Mills <bcmills@google.com>
Michael Matloob [Fri, 19 Jun 2020 21:59:16 +0000 (17:59 -0400)]
cmd/trace: move viewer data structs into cmd/internal/traceviewer
The ViewerEvent, ViewerData and ViewerFrame structs are moved into
cmd/internal/traceviewer, and renamed Event, Data, and Frame.
The structs are the same, except for the following: A definition
for the JSON "bp" field that's defined in the trace format, but
missing in the structs has been added. Also, the Tid and Pid fields
on Event have been renamed TID and PID to better match Go style.
Finally, the footer field on ViewerData, which hasn't been used
for a while, has been removed.
This CL is in preparation for the usage of these structs by cmd/go's
tracing functionality.
Updates #38714
Change-Id: I345f23617b96d4629b876ae717f89d56a67e05a3
Reviewed-on: https://go-review.googlesource.com/c/go/+/239098
Run-TryBot: Michael Matloob <matloob@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com> Reviewed-by: Jay Conrod <jayconrod@google.com>
Dmitri Shuralyov [Wed, 12 Aug 2020 14:34:15 +0000 (10:34 -0400)]
doc/go1.16: start draft release notes
This template is based on CL 220278 and previous ones like it.
Include Compiler and Linker sections proactively, they can be
removed if they don't end up being needed for Go 1.16.
Use two spaces of indentation for TODOs to set a better precedent
for the final text that will take its place.
'relnote -html' does not report any changes at this time.
For #40700.
Change-Id: I096b0ce0d33aaaa6fae9c91c0d2dfb89b9c5e94c
Reviewed-on: https://go-review.googlesource.com/c/go/+/248198 Reviewed-by: Carlos Amedee <carlos@golang.org>
Dmitri Shuralyov [Tue, 11 Aug 2020 22:37:44 +0000 (18:37 -0400)]
internal/goversion: update Version to 1.16
This is the start of the Go 1.16 development cycle, so update the
Version value accordingly. It represents the Go 1.x version that
will soon open up for development (and eventually become released).
Historically, we used to bump this at an arbitrary time throughout
the development cycle, but it's better to be more predictable about
updating it. The start of a development cycle should be the most
appropriate time: it clearly marks the boundary between 1.15 and
1.16 development, and doing it early can help catch issues in other
tooling. See issue #38704 for more background.
There is no longer a need to update the list of Go versions in
src/go/build/doc.go because it does not exist as of CL 232981.
For #40705.
Updates #38704.
Updates #37018.
Change-Id: Id8ee733b5e79c53b6cd03509c6560614d8743833
Reviewed-on: https://go-review.googlesource.com/c/go/+/248038 Reviewed-by: Carlos Amedee <carlos@golang.org>
Cherry Zhang [Wed, 5 Aug 2020 00:25:10 +0000 (20:25 -0400)]
runtime: make nanotime1 reentrant
Currently, nanotime1 (and walltime1) is not reentrant, in that it
sets m.vdsoSP at entry and clears it at exit. If a signal lands
in between, and nanotime1 is called from the signal handler, it
will clear m.vdsoSP while we are still in nanotime1. If (in the
unlikely event) it is signaled again, m.vdsoSP will be wrong,
which may cause the stack unwinding code to crash.
This CL makes it reentrant, by saving/restoring the previous
vdsoPC and vdsoSP, instead of setting it to 0 at exit.
TODO: have some way to test?
Change-Id: I9ee53b251f1d8a5a489c71d4b4c0df1dee70c3e5
Reviewed-on: https://go-review.googlesource.com/c/go/+/246763
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Jeremy Faller [Wed, 29 Jul 2020 15:06:02 +0000 (11:06 -0400)]
[dev.link] use per package filenames to build pclntab
In order to prevent renumbering of filenames in pclntab generation, use
the per-package file list (previously only used for DWARF generation) as
file-indices. This is the largest step to eliminate renumbering of
filenames in pclntab.
Note, this is probably not the final state of the file table within the
object file. In this form, the linker loads all filenames for all
objects. I'll move to storing the filenames as regular string
symbols,and defaulting all string symbols to using the larger hash value
to make generation of pcln simplest, and most memory friendly.
Than McIntosh [Wed, 8 Jul 2020 15:02:51 +0000 (11:02 -0400)]
[dev.link] cmd/link: add PPC64 section splitting test
Add a new PPC64-only linker test that does a build with the
-debugppc64textsize debugging option (selecting a lower the threshold
for text section splitting) to verify that no bugs have been
introduced in the linker code that manages this process.
Change-Id: Iea3f16a04c894d528eab2cb52f1ec1d75a2770cc
Reviewed-on: https://go-review.googlesource.com/c/go/+/241499
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Jeremy Faller <jeremy@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com>
Than McIntosh [Tue, 7 Jul 2020 16:54:46 +0000 (12:54 -0400)]
[dev.link] cmd/link: add PPC64 debugging option to encourage text section splits
Add a new debugging command line option (-debugppc64textsize=N) that
forces the start of a new text section after ".text" hits N bytes as
opposed to the architected limit of 2^26. This is intended to enable
testing of the linker code paths that handle multiple .text sections
on PPC64 without resorting to building giant applications.
Updates #20492.
Change-Id: I74ab7fd1e412e9124de5bd0d8d248c5e73225ae3
Reviewed-on: https://go-review.googlesource.com/c/go/+/241073
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Jeremy Faller <jeremy@golang.org>
Than McIntosh [Wed, 8 Jul 2020 14:43:42 +0000 (10:43 -0400)]
[dev.link] cmd/link: fix ppc64-specific bug in genelfsym
The code in the the linker's genelfsym() routine was not properly
including runtime.text.%d marker symbols that are emitted on PPC64
when a very large text section is split into chunks. This bug was
introduced in CL 233338 when portions of asmb2() were converted
from sym.Symbol to loader.Sym usage.
Change-Id: Idfed944c41e1805f78f35be67bcdd18bdefd7819
Reviewed-on: https://go-review.googlesource.com/c/go/+/241498
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com> Reviewed-by: Jeremy Faller <jeremy@golang.org>
Katie Hockman [Tue, 4 Aug 2020 15:45:32 +0000 (11:45 -0400)]
encoding/binary: read at most MaxVarintLen64 bytes in ReadUvarint
This CL ensures that ReadUvarint consumes only a limited
amount of input (instead of an unbounded amount).
On some inputs, ReadUvarint could read an arbitrary number
of bytes before deciding to return an overflow error.
After this CL, ReadUvarint returns that same overflow
error sooner, after reading at most MaxVarintLen64 bytes.
Fix authored by Robert Griesemer and Filippo Valsorda.
Thanks to Diederik Loerakker, Jonny Rhea, Raúl Kripalani,
and Preston Van Loon for reporting this.
Michael Matloob [Tue, 4 Aug 2020 17:24:37 +0000 (13:24 -0400)]
cmd/go: fix error stacks when there are scanner errors
After golang.org/cl/228784 setLoadPackageDataError tries to decide whether an
error is caused by an imported package or an importing package by examining the
error itself to decide. Ideally, the errors themselves would belong to a
specific interface or some other property to make it unambiguous that they
were import errors. Since they don't, setLoadPackageDataError just checked
for nogoerrors and classified all other errors as import errors. But
it missed scanner errors which are also "caused" by the imported
package.
Fixes #40544
Change-Id: I39159bfdc286bee73697decd07b8aa9451f2db06
Reviewed-on: https://go-review.googlesource.com/c/go/+/246717
Run-TryBot: Michael Matloob <matloob@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
Cherry Zhang [Mon, 3 Aug 2020 21:59:41 +0000 (17:59 -0400)]
[dev.link] cmd/link: fix pclntab symbol handling on AIX
On AIX, container symbols are handled in a weird way (unlike
other platforms): the outer symbol needs to have size (but still
no data), and the inner symbols must not be in the symbol table
(otherwise it overlaps with the outer symbol, which the system
linker doesn't like).
As of CL 241598, pclntab becomes a container symbol. We need to
follow the rule above for AIX.
[dev.link] cmd/compile, cmd/link: generate itablink at link time
Currently, at compile time, for each itab symbol, we create an
"itablink" symbol which holds solely the address of the itab
symbol. At link time, all the itablink symbols are grouped
together to form the itablinks slice.
This CL removes the itablink symbols, and directly generate the
itablinks slice in the linker. This removes a number of symbols,
which are dupOK and generally have long names. And also removes
a special handling of itablink symbols in the deadcode pass which
iterates through all symbols.
Change-Id: I475c3c8899e9fbeec9abc7647b1e4a69aa5c3c5a
Reviewed-on: https://go-review.googlesource.com/c/go/+/245901 Reviewed-by: Jeremy Faller <jeremy@golang.org>
[dev.link] cmd/compile, cmd/link: make itab symbols content-addressable
Extend the content-addressable symbol mechanism to itab symbols.
Itab symbols require global uniqueness (as at run time we compare
pointers), so it needs to be reliably deduplicated. Currently the
content hash depends on symbol name expansion, so we can only do
this when all Go packages are built with know package paths. Fall
back to checking names if any Go package is built with unknown
package path.
Change-Id: Icf5e8873755050c20e5fc6549f6de1c883254c89
Reviewed-on: https://go-review.googlesource.com/c/go/+/245719 Reviewed-by: Jeremy Faller <jeremy@golang.org>
[dev.link] cmd/compile, cmd/link: reference type symbol of defined type by index
The type descriptor symbol of a defined (named) type (and pointer
to it) is defined only in the package that defines the type. It
is not dupOK, unlike other type descriptors. So it can be
referenced by index. Currently it is referenced by name for
cross-package references, because the index is not exported and
so not known to the referencing package.
This CL passes the index through the export data, so the symbol
can be referenced by index, and does not need to be looked up by
name. This also makes such symbol references consistent: it is
referenced by index within the defining package and also cross-
package, which makes it easier for content hashing (in later CLs).
One complication is that we need to set flags on referenced
symbols (specifically, the UsedInIface flag). Before, they are
non-package refs, which naturally carry flags in the object file.
For indexed refs, we currently don't put their flags in the
object file. Introduce a new block for this.
Change-Id: I8126f8e318ac4e6609eb2ac136201fd6c264c256
Reviewed-on: https://go-review.googlesource.com/c/go/+/245718 Reviewed-by: Jeremy Faller <jeremy@golang.org>
Jeremy Faller [Tue, 21 Jul 2020 19:53:30 +0000 (15:53 -0400)]
[dev.link] move FuncID creation into the compiler/assembler
Leaving creation of the funcID till the linker requires the linker to
load the function and file names into memory. Moving these into the
compiler/assembler prevents this.
This work is a step towards moving all func metadata into the compiler.
Keith Randall [Sat, 1 Aug 2020 19:55:47 +0000 (12:55 -0700)]
runtime/race: fix ppc64le build
The .syso test also fails for ppc64le. Not sure why. For now, just
disable the test for that architecture. The test really only needs to
run on a single builder of any arch.
Change-Id: I346cdc01ada09d43c4c504fbc30be806f59d5422
Reviewed-on: https://go-review.googlesource.com/c/go/+/246358
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
Keith Randall [Fri, 31 Jul 2020 20:56:18 +0000 (13:56 -0700)]
runtime/race: rebuild some .syso files to remove getauxval dependency
We can't depend on getauxval because it only exists in glibc >= 2.16.
Tsan has been updated to avoid that dependency
(https://reviews.llvm.org/D84859). This CL rebuilds the affected
.syso files, and adds a test to make sure we don't regress.
[dev.link] cmd/link: make symbol attribute setting more reliable
For dupOK symbols, their attributes should be OR'd. Most of the
attributes are expected to be set consistently across multiple
definitions, but UsedInIface must be OR'd, and for alignment we
need to pick the largest one. Currently the attributes are not
always OR'd, depending on addSym returning true or false. This
doesn't cause any real problem, but it would be a problem if we
make type descriptor symbols content-addressable.
This CL removes the second result of addSym, and lets preloadSyms
always set the attributes. Also removes the alignment handling on
addSym, handles it in preloadSyms only.
Change-Id: I06b3f0adb733f6681956ea9ef54736baa86ae7bc
Reviewed-on: https://go-review.googlesource.com/c/go/+/245720 Reviewed-by: Jeremy Faller <jeremy@golang.org>
Notably, _Ctype_union_U was declared as a defined type, but uses were
being rewritten into uses of the underlying type, which matched how
_Cfunc_f was declared.
After CL 230037, cgo started consistently rewriting "C.foo" type
expressions as "_Ctype_foo", which caused it to start emitting:
type _Ctype_enum_E uint32
type _Ctype_union_U [8]byte
Of course, this fails to type-check because _Ctype_enum_E and
_Ctype_union_U are defined types.
This CL changes cgo to emit:
type _Ctype_enum_E = uint32
type _Ctype_union_U = [8]byte
// f unchanged since CL 230037
// _Cfunc_f still unchanged
It would probably be better to fix this in (*typeConv).loadType so
that cgo generated code uses the _Ctype_foo aliases too. But as it
wouldn't have any effect on actual compilation, it's not worth the
risk of touching it at this point in the release cycle.
Updates #39537.
Fixes #40494.
Change-Id: I88269660b40aeda80a9a9433777601a781b48ac0
Reviewed-on: https://go-review.googlesource.com/c/go/+/246057 Reviewed-by: Ian Lance Taylor <iant@golang.org>
Michael Anthony Knyszek [Mon, 13 Jul 2020 19:51:50 +0000 (19:51 +0000)]
runtime: validate candidate searchAddr in pageAlloc.find
Currently pageAlloc.find attempts to find a better estimate for the
first free page in the heap, even if the space its looking for isn't
necessarily going to be the first free page in the heap (e.g. if npages
>= 2). However, in doing so it has the potential to return a searchAddr
candidate that doesn't actually correspond to mapped memory, but this
candidate might still be adopted. As a result, pageAlloc.alloc's fast
path may look at unmapped summary memory and segfault. This case is rare
on most operating systems since the heap is kept fairly contiguous, so
the chance that the candidate searchAddr discovered is unmapped is
fairly low. Even so, this is totally possible and outside the user's
control when it happens (in fact, it's likely to happen consistently for
a given user on a given system).
Fix this problem by ensuring that our candidate always points to mapped
memory. We do this by looking at mheap's arenas structure first. If it
turns out our candidate doesn't correspond to mapped memory, then we
look at inUse to round up the searchAddr to the next mapped address.
While we're here, clean up some documentation related to searchAddr.
Fixes #40191.
Change-Id: I759efec78987e4a8fde466ae45aabbaa3d9d4214
Reviewed-on: https://go-review.googlesource.com/c/go/+/242680
Run-TryBot: Michael Knyszek <mknyszek@google.com> Reviewed-by: Austin Clements <austin@google.com> Reviewed-by: Michael Pratt <mpratt@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Jeremy Faller [Tue, 28 Jul 2020 17:49:13 +0000 (13:49 -0400)]
[dev.link] ensure package path is set when TEXT symbols are created
We're reworking pclntab generation in the linker, and with that we're
moving FuncID generation in to the compiler. Determining the FuncID is
done by a lookup on the package.function name; therefore, we need the
package whenever we make the TEXT symbols.
Jeremy Faller [Thu, 16 Jul 2020 20:18:49 +0000 (16:18 -0400)]
[dev.link] create runtime.funcnametab
Move the function names out of runtime.pclntab_old, creating
runtime.funcnametab. There is an unfortunate artifact in this change in
that calculating the funcID still requires loading the name. Future work
will likely pull this out and put it into the object file Funcs.
Jeremy Faller [Wed, 24 Jun 2020 18:30:16 +0000 (14:30 -0400)]
[dev.link] cmd/link: add runtime.pcheader
As of July 2020, a fair amount of the new linker's live memory, and
runtime is spent generating pclntab. In an effort to streamline that
code, this change starts breaking up the generation of runtime.pclntab
into smaller chunks that can run later in a link. These changes are
described in an (as yet not widely distributed) document that lays out
an improved format. Largely the work consists of breaking up
runtime.pclntab into smaller pieces, stopping much of the data
rewriting, and getting runtime.pclntab into a form where we can reason
about its size and look to shrink it. This change is the first part of
that work -- just pulling out the header, and demonstrating where a
majority of that work will be.
[dev.link] cmd/internal/obj: trim trailing zeros for content hashing
The symbol's data in the object file (sym.P) may already not
contain trailing zeros (e,g, for [10]int{1}), but sometimes it
does (e.g. for [10]int{1,0}). The linker can already handle this
case. We just always trim the trailing zeros for content hashing,
so it can deduplicate [10]int{1} and [10]int{1,0}.
Note: in theory we could just trim the zeros in the symbol data
as well. But currently the linker depends on reading symbol data
for certain symbols (e.g. type symbol decoding), and trimming
will complicates things in the linker.
Cholerae Hu [Fri, 24 Jul 2020 03:00:36 +0000 (11:00 +0800)]
cmd/compile: don't addLocalInductiveFacts if there is no direct edge from if block to phi block
Currently in addLocalInductiveFacts, we only check whether
direct edge from if block to phi block exists. If not, the
following logic will treat the phi block as the first successor,
which is wrong.
This patch makes prove pass more conservative, so we disable
some cases in test/prove.go. We will do some optimization in
the following CL and enable these cases then.
Fixes #40367.
Change-Id: I27cf0248f3a82312a6f7dabe11c79a1a34cf5412
Reviewed-on: https://go-review.googlesource.com/c/go/+/244579 Reviewed-by: Zach Jones <zachj1@gmail.com> Reviewed-by: Keith Randall <khr@golang.org>
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
We have Reloc and Reloc2. Reloc2 is the better approach and most
code uses Reloc2. There are still uses of Reloc. This CL migrates
them to Reloc2, and removes Reloc.
Change-Id: Id5f6a6019e1e044add682d05e70ebb1548ec58d9
Reviewed-on: https://go-review.googlesource.com/c/go/+/245577 Reviewed-by: Jeremy Faller <jeremy@golang.org>
[dev.link] cmd/link: refactor ExtReloc data structures
We used to generate all external relocations in memory, then emit
the relocation records at a later pass. The data structures were
chosen so that it takes as little memory as possible. Now we just
stream out external relocations, and ExtReloc is just a local
variable. Change the data structure to avoid repeated read of
some fields. Also get rid of ExtRelocView, as it is no longer
necessary.
This reverts CL 202578 and CL 230677 which added an optimization
to use KDSA when available on s390x.
Inconsistencies have been found between the two implementations
in their handling of certain edge cases. Since the Go 1.15 release
is extremely soon it seems prudent to remove this optimization
for now and revisit it in a future release.
Fixes #40475.
Change-Id: Ifb2ed9b9e573784df57383671f1c29d8abae90d4
Reviewed-on: https://go-review.googlesource.com/c/go/+/245497
Run-TryBot: Michael Munday <mike.munday@ibm.com>
TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ruixin(Peter) Bao <ruixin.bao@ibm.com> Reviewed-by: Filippo Valsorda <filippo@golang.org>
Michael Pratt [Mon, 27 Jul 2020 19:04:17 +0000 (15:04 -0400)]
runtime: ensure startm new M is consistently visible to checkdead
If no M is available, startm first grabs an idle P, then drops
sched.lock and calls newm to start a new M to run than P.
Unfortunately, that leaves a window in which a G (e.g., returning from a
syscall) may find no idle P, add to the global runq, and then in stopm
discover that there are no running M's, a condition that should be
impossible with runnable G's.
To avoid this condition, we pre-allocate the new M ID in startm before
dropping sched.lock. This ensures that checkdead will see the M as
running, and since that new M must eventually run the scheduler, it will
handle any pending work as necessary.
Outside of startm, most other calls to newm/allocm don't have a P at
all. The only exception is startTheWorldWithSema, which always has an M
if there is 1 P (i.e., the currently running M), and if there is >1 P
the findrunnable spinning dance ensures the problem never occurs.
This has been tested with strategically placed sleeps in the runtime to
help induce the correct race ordering, but the timing on this is too
narrow for a test that can be checked in.
cmd/link: don't mark shared library symbols reachable unconditionally
During the transitioning period, we mark symbols from Go shared
libraries reachable unconditionally. That might be useful when
there was still a large portion of the linker using sym.Symbols,
and only reachable symbols were converted to sym.Symbols. Marking
them reachable brings them to the dynamic symbol table, even if
they are not needed, increased the binary size unexpectedly.
That time has passed. Now we largely operate on loader symbols,
and it is not needed to mark them reachable anymore.
OutData was used for a symbol to point to its data in the output
buffer, in order to apply relocations. Now we fold relocation
application to Asmb next to symbol data writing. We can just pass
the output data as a local variable.
Linking cmd/compile,
name old time/op new time/op delta
Asmb_GC 19.0ms ±10% 16.6ms ± 9% -12.50% (p=0.032 n=5+5)
name old alloc/op new alloc/op delta
Asmb_GC 3.78MB ± 0% 0.14MB ± 1% -96.41% (p=0.008 n=5+5)
name old live-B new live-B delta
Asmb_GC 27.5M ± 0% 23.9M ± 0% -13.24% (p=0.008 n=5+5)
Change-Id: Id870a10dce2a0a7447a05029c6d0ab39b47d0a12
Reviewed-on: https://go-review.googlesource.com/c/go/+/244017 Reviewed-by: Jeremy Faller <jeremy@golang.org>
[dev.link] cmd/link: stream external relocations on ARM64 and on Darwin
Support streaming external relocations on ARM64. Support
architecture-specific relocations.
Also support streaming external relocations on Darwin. Do it in
the same CL so ARM64's archreloc doesn't need to support both
streaming and non-streaming.
Change-Id: Ia7fee9957892f98c065022c69a51f47402f4d6e2
Reviewed-on: https://go-review.googlesource.com/c/go/+/243644 Reviewed-by: Jeremy Faller <jeremy@golang.org>
For content-addressable symbols, we build its content hash based
on the symbol data and relocations. When the compiler builds the
symbol data, it may not always include the trailing zeros, e.g.
the data of [10]int64{1,2,3} is only the first 24 bytes.
Therefore, we may end up with symbols with the same contents
(thus same hash) but different sizes. This is not actually a hash
collision. In this case, we can deduplicate them and keep the one
with the larger size.
[dev.link] cmd/internal/obj: handle content-addressable symbols with relocations
For content-addressable symbols with relocations, we build a
content hash based on its content and relocations. Depending on
the category of the referenced symbol, we choose different hash
algorithms such that the hash is globally consistent.
For now, we only support content-addressable symbols with
relocations when the current package's import path is known, so
that the symbol names are fully expanded. Otherwise, if the
referenced symbol is a named symbol whose name is not fully
expanded, the hash won't be globally consistent, and can cause
erroneous collisions. This is fine for now, as the deduplication
is just an optimization, not a requirement for correctness (until
we get to type descriptors).
[dev.link] cmd/compile: make read-only static temps content-addressable
For now, we only do this for symbols without relocations.
Mark static temps "local", as they are not referenced across DSO
boundaries. And deduplicating a local symbol and a non-local
symbol can be problematic.
Dmitri Shuralyov [Wed, 1 Jul 2020 16:49:43 +0000 (12:49 -0400)]
go/printer: remove exported StdFormat flag
The StdFormat flag was added as part of CL 231461, where the primary aim
was to fix the bug #37476. It's expected that the existing printer modes
only adjust spacing but do not change any of the code text itself. A new
printing flag served as a way for cmd/gofmt and go/format to delegate
a part of formatting work to the printer—where it's more more convenient
and efficient to perform—while maintaining current low-level printing
behavior of go/printer unmodified.
We already have cmd/gofmt and the go/format API that implement standard
formatting of Go source code, so there isn't a need to expose StdFormat
flag to the world, as it can only cause confusion.
Consider that to format source in canonical gofmt style completely it
may require tasks A, B, C to be done. In one version of Go, the printer
may do both A and B, while cmd/gofmt and go/format will do the remaining
task C. In another version, the printer may take on doing just A, while
cmd/gofmt and go/format will perform B and C. This makes it hard to add
a gofmt-like mode to the printer without compromising on above fluidity.
This change prefers to shift back some complexity to the implementation
of the standard library, allowing us to avoid creating the new exported
printing flag just for the internal needs of gofmt and go/format today.
We may still want to re-think the API and consider if something better
should be added, but unfortunately there isn't time for Go 1.15. We are
not adding new APIs now, so we can defer this decision until Go 1.16 or
later, when there is more time.
compress/flate: fix another deflate Reset inconsistency
While investigating #34121, fixed by CL 193605,
I discovered another case where Reset was not quite
resetting enough.
This specific case is not a problem in Reset itself but
rather that the Huffman bit writer in one code path
is using uninitialized memory left over from a previous
block, making the compression not choose the optimal
compression method.
Klaus Post [Thu, 7 May 2020 12:50:00 +0000 (12:50 +0000)]
compress/flate: fix deflate Reset consistency
Modify the overflow detection logic to shuffle the contents
of the table to a lower offset to avoid leaking the effects
of a previous use of compress.Writer past Reset calls.
Fixes #34121
Change-Id: I9963eadfa5482881e7b7adbad4c2cae146b669ab
GitHub-Last-Rev: 8b35798cdd4d5a901d6422647b12984d7e500ba3
GitHub-Pull-Request: golang/go#34128
Reviewed-on: https://go-review.googlesource.com/c/go/+/193605 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
[dev.link] cmd/link: stream out external relocations on AMD64 ELF
Currently, when external linking, in relocsym (in asmb pass), we
convert Go relocations to an in-memory representation of external
relocations, and then in asmb2 pass we write them out to the
output file. This is not memory efficient.
This CL makes it not do the conversion but directly stream out
the external relocations based on Go relocations. Currently only
do this on AMD64 ELF systems.
This reduces memory usage, but makes the asmb2 pass a little
slower.
name old alloc/op new alloc/op delta
Asmb_GC 26.0MB ± 0% 4.1MB ± 0% -84.15% (p=0.008 n=5+5)
Asmb2_GC 8.19MB ± 0% 8.18MB ± 0% ~ (p=0.222 n=5+5)
name old live-B new live-B delta
Asmb_GC 49.2M ± 0% 27.4M ± 0% -44.38% (p=0.008 n=5+5)
Asmb2_GC 51.5M ± 0% 29.7M ± 0% -42.33% (p=0.008 n=5+5)
TODO: figure out what is slow. Possible improvements:
- Remove redundant work in relocsym.
- Maybe there is a better representation for external relocations
now.
- Fine-grained parallelism in emitting external relocations.
- The old elfrelocsect only iterates over external relocations,
now we iterate over all relocations. Is it too many?
[dev.link] cmd/internal/goobj2, cmd/link: use short hash function for short symbols
For symbols of size 8 bytes or below, we can map them to 64-bit
hash values using the identity function. There is no need to use
longer and more expensive hash functions.
For them, we introduce another pseudo-package, PkgIdxHashed64. It
is like PkgIdxHashed except that the hash function is different.
Note that the hash value is not affected with trailing zeros,
e.g. "A" and "A\0\0\0" have the same hash value. This allows
deduplicating a few more symbols. When deduplicating them, we
need to keep the longer one.
Change-Id: Iad0c2e9e569b6a59ca6a121fb8c8f0c018c6da03
Reviewed-on: https://go-review.googlesource.com/c/go/+/242362 Reviewed-by: Jeremy Faller <jeremy@golang.org>
[dev.link] cmd/internal/obj: make integer/float constant symbols content-addressable
Fill in the data at compile time, and get rid of the preprocess
function in the linker.
We need to be careful with symbol alignment: data symbols are
generally naturally aligned, except for string symbols which are
not aligned. When deduplicating two symbols with same content but
different alignments, we need to keep the biggest alignment.
This CL introduces content-addressable symbols (a.k.a. hashed
symbols) to object files. Content-addressable symbols are
identified and referenced by their content hashes, instead of by
names.
In the object file, a new pseudo-package index PkgIdxHashed is
introduced, for content-addressable symbols, and a new block is
added to store their hashes. The hashes are used by the linker to
identify and deduplicate the symbols.
For now, we only support content-addressable symbols that are
always locally defined (i.e. no cross-package references).
As a proof of concept, make string constant symbols content-
addressable.
net/http: synchronize "100 Continue" write and Handler writes
The expectContinueReader writes to the connection on the first
Request.Body read. Since a Handler might be doing a read in parallel or
before a write, expectContinueReader needs to synchronize with the
ResponseWriter, and abort if a response already went out.
Filippo Valsorda [Fri, 19 Jun 2020 02:45:52 +0000 (22:45 -0400)]
crypto/x509: respect VerifyOptions.KeyUsages on Windows
When using the platform verifier on Windows (because Roots is nil) we
were always enforcing server auth EKUs if DNSName was set, and none
otherwise. If an application was setting KeyUsages, they were not being
respected.
Started correctly surfacing IncompatibleUsage errors from the system
verifier, as those are the ones applications will see if they are
affected by this change.
Also refactored verify_test.go to make it easier to add tests for this,
and replaced the EKULeaf chain with a new one that doesn't have a SHA-1
signature.
Copy and adapt tests from text/template, to exercise more of html/template's copy.
Various differences in behavior are flagged with NOTE comments or t.Skip
and documented in #40075. Many of them are probably bugs.
One clarifying test case added to both text/template and html/template.