Nigel Tao [Tue, 4 Oct 2011 00:09:03 +0000 (11:09 +1100)]
image: spin off a new color package out of the image package.
The spin-off renames some types. The new names are simply better:
image.Color -> color.Color
image.ColorModel -> color.Model
image.ColorModelFunc -> color.ModelFunc
image.PalettedColorModel -> color.Palette
image.RGBAColor -> color.RGBA
image.RGBAColorModel -> color.RGBAModel
image.RGBA64Color -> color.RGBA64
image.RGBA64ColorModel -> color.RGBA64Model
(similarly for NRGBAColor, GrayColorModel, etc)
The image.ColorImage type stays in the image package, but is renamed:
image.ColorImage -> image.Uniform
The image.Image implementations (image.RGBA, image.RGBA64, image.NRGBA,
image.Alpha, etc) do not change their name, and gain a nice symmetry:
an image.RGBA is an image of color.RGBA, etc.
The image.Black, image.Opaque uniform images remain unchanged (although
their type is renamed from image.ColorImage to image.Uniform). The
corresponding color types (color.Black, color.Opaque, etc) are new.
Nothing in the image/ycbcr is renamed yet. The ycbcr.YCbCrColor and
ycbcr.YCbCrImage types will eventually migrate to color.YCbCr and
image.YCbCr, but that will be a separate CL.
Russ Cox [Sat, 1 Oct 2011 17:00:53 +0000 (13:00 -0400)]
runtime: fix map memory leak
The map implementation was using the C idiom of using
a pointer just past the end of its table as a limit pointer.
Unfortunately, the garbage collector sees that pointer as
pointing at the block adjacent to the map table, pinning
in memory a block that would otherwise be freed.
Fix by making limit pointer point at last valid entry, not
just past it.
Reviewed by Mike Burrows.
R=golang-dev, bradfitz, lvd, r
CC=golang-dev
https://golang.org/cl/5158045
runtime: parallelize garbage collector mark + sweep
Running test/garbage/parser.out.
On a 4-core Lenovo X201s (Linux):
31.12u 0.60s 31.74r 1 cpu, no atomics
32.27u 0.58s 32.86r 1 cpu, atomic instructions
33.04u 0.83s 27.47r 2 cpu
On a 16-core Xeon (Linux):
33.08u 0.65s 33.80r 1 cpu, no atomics
34.87u 1.12s 29.60r 2 cpu
36.00u 1.87s 28.43r 3 cpu
36.46u 2.34s 27.10r 4 cpu
38.28u 3.85s 26.92r 5 cpu
37.72u 5.25s 26.73r 6 cpu
39.63u 7.11s 26.95r 7 cpu
39.67u 8.10s 26.68r 8 cpu
On a 2-core MacBook Pro Core 2 Duo 2.26 (circa 2009, MacBookPro5,5):
39.43u 1.45s 41.27r 1 cpu, no atomics
43.98u 2.95s 38.69r 2 cpu
On a 2-core Mac Mini Core 2 Duo 1.83 (circa 2008; Macmini2,1):
48.81u 2.12s 51.76r 1 cpu, no atomics
57.15u 4.72s 51.54r 2 cpu
The handoff algorithm is really only good for two cores.
Beyond that we will need to so something more sophisticated,
like have each core hand off to the next one, around a circle.
Even so, the code is a good checkpoint; for now we'll limit the
number of gc procs to at most 2.
This is a possible optimization. I'm not sure the complexity is worth it.
The new benchmark in escape_test is 46us without and 35us with the optimization.
Mike Samuel [Fri, 30 Sep 2011 01:09:11 +0000 (18:09 -0700)]
exp/template/html: simplify URL filtering
This removes a few cases from escapeAction and clarifies the
responsibilities of urlFilter which no longer does any
escaping or normalization. It is now solely a filter.
Mike Samuel [Wed, 28 Sep 2011 21:07:48 +0000 (14:07 -0700)]
exp/template/html: handle custom attrs and HTML5 embedded elements.
HTML5 allows embedded SVG and MathML.
Code searches show SVG is used for graphing.
This changes transition to deal with constructs like
<svg xmlns:xlink="http://www.w3.org/1999/xlink">
It changes attr and clients to call a single function that combines
the name lookup and "on" prefix check to determine an attribute
value type given an attribute name.
That function uses heuristics to recognize that
xlink:href and svg:href
have URL content, and that data-url is likely contains URL content,
since "javascript:" injection is such a problem.
I did a code search over a closure templates codebase to determine
patterns of custom attribute usage. I did something like
which seem to match all the ones that are likely URL content.
There are some short words that match that heuristic, but I still think it decent since
any custom attribute that has a numeric or enumerated keyword value will be unaffected by
the URL assumption.
Counterexamples from /usr/share/dict:
during, hourly, maturity, nourish, purloin, security, surly
*** This is a design review, not a code review. ***
Feel free to reply to the mail instead of picking out
individual lines to comment on in Rietveld.
This command, go, will replace both gomake/make and goinstall.
Make will stick around only for building our C commands
and perhaps package runtime.
In normal use while developing you'd run commands like
go compile
go test
go clean
go install
which apply to the package in the current directory.
To operate on code written by others, you add an explicit
package path:
go get gopath.googlecode.com/hg/oauth
go test gopath.googlecode.com/hg/oauth
The script.txt file is a script showing the output of
the various help commands that the command has.
(Right now, all the command can do is print help messages.)
This is just a new API to do many replacements at once.
While the point of this API is to be faster than doing replacements one
at a time, the implementation in this CL has the optimizations removed
and may actually be slower.
Robert Griesemer [Tue, 27 Sep 2011 23:21:28 +0000 (16:21 -0700)]
index/suffixarray: revert change from int -> int32
CL 5040041 (https://golang.org/cl/5040041)
changed the use of []int to []int32 internally so
that encoding/binary could be used. This is no
longer needed (gobs can encode ints), and using
[]int is more in sync w/ the semantics of the data
structure (the index elements are indices which are
ints). Changing it back.
The 512 MB array causes load delays on some systems.
Now that we have recover, we can do all the tests in
one binary, so that the delay is incurred just once.
R=golang-dev, r
CC=golang-dev
https://golang.org/cl/5142044
Mike Samuel [Mon, 26 Sep 2011 09:10:43 +0000 (02:10 -0700)]
exp/template/html: make sure marshalled JSON can be parsed as JS.
This makes sure that all JS newlines are encoded in JSON.
It also moots a TODO about possibly escaping supplemental codepoints.
I served:
Content-Type: text/javascript;charset=UTF-8
var s = "%s";
document.write("<p>", s, "</p><ol>");
for (var i = 0; i < s.length; i++) {
document.write("<li>", s.charCodeAt(i).toString(16), "</li>");
}
document.write("</l>");
where %s was replaced with bytes "\xf0\x9d\x84\x9e" to test
straight UTF-8 instead of encoding surrogates separately.
Recent Firefox, Chrome, and Safari all decoded it properly.
I have yet to try it on IE or older versions.
Mike Samuel [Mon, 26 Sep 2011 07:42:38 +0000 (00:42 -0700)]
exp/template/html: error out on ambiguous unquoted attributes
HTML parsers may differ on whether
<input id= onchange=f( ends in id's or onchange's value,
<a class=`foo ends inside a value,
<input style=font:'Arial' needs open-quote fixup.
Per
http://www.w3.org/TR/html5/tokenization.html#attribute-value-unquoted-state
this treats the error cases in 8.2.4.40 Attribute value (unquoted) state
as fatal errors.
\> U+0022 QUOTATION MARK (")
\> U+0027 APOSTROPHE (')
\> U+003C LESS-THAN SIGN (<)
\> U+003D EQUALS SIGN (=)
\> U+0060 GRAVE ACCENT (`)
Parse error. Treat it as per the "anything else" entry below.
Ian Lance Taylor [Sat, 24 Sep 2011 04:23:40 +0000 (21:23 -0700)]
test: match gccgo error messages
bug340.go:14:7: error: expected type
bug340.go:15:4: error: reference to undefined field or method ‘x’
bug350.go:12:1: error: redefinition of ‘m’
bug350.go:11:1: note: previous definition of ‘m’ was here
bug350.go:15:1: error: redefinition of ‘p’
bug350.go:14:1: note: previous definition of ‘p’ was here
bug351.go:12:6: error: non-name on left side of ‘:=’
runtime: gdb support: gracefully handle not being able to find types
The Dwarf info has the full typenames, the go *struct runtime.commonType
has the short name. A more permanent fix would link the two together
but this way the user gets useable stack traces for now.
websocket: add mutex to make websocket full-duplex
One benefit of websocket is that it is full-duplex so that it could
send and receive at the same time.
This CL makes websocket goroutine safe, so user could use websocket
both on goroutine for read and on goroutine for write.
Mike Samuel [Thu, 22 Sep 2011 04:38:40 +0000 (21:38 -0700)]
exp/template/html: elide comments in template source.
When templates are stored in external files, developers often embed
comments to explain&|disable code.
<!-- Oblique reference to project code name here -->
{{if .C}}...{{else}}<!-- commented out default -->{{end}}
This unnecessarily increases the size of shipped HTML and can leak
information.
This change elides all comments of the following types:
1. <!-- ... --> comments found in source.
2. /*...*/ and // comments found in <script> elements.
3. /*...*/ and // comments found in <style> elements.
It does not elide /*...*/ or // comments found in HTML attributes:
4. <button onclick="/*...*/">
5. <div style="/*...*/">
I can find no examples of comments in attributes in Closure Templates
code and doing so would require keeping track of character positions
post decode in
<button onclick="/*...*/">
To prevent token joining, /*comments*/ are JS and CSS comments are
replaced with a whitespace char.
HTML comments are not, but to prevent token joining we could try to
detect cases like
<<!---->b>
</<!---->b>
which has a well defined meaning in HTML but will cause a validator
to barf. This is difficult, and this is a very minor case.
I have punted for now, but if we need to address this case, the best
way would be to normalize '<' in stateText to '<' consistently.
The whitespace to replace a JS /*comment*/ with depends on whether
there is an embedded line terminator since
break/*
*/foo
...
is equivalent to
break;
foo
...
while
break/**/foo
...
is equivalent to
break foo;
...
Comment eliding can interfere with IE conditional comments.
http://en.wikipedia.org/wiki/Conditional_comment
<!--[if IE 6]>
<p>You are using Internet Explorer 6.</p>
<![endif]-->
/*@cc_on
document.write("You are using IE4 or higher");
@*/
I have not encountered these in production template code, and
the typed content change in CL 4962067 provides an escape-hatch
if conditional comments are needed.
Mike Samuel [Thu, 22 Sep 2011 02:04:41 +0000 (19:04 -0700)]
exp/template/html: simplify transition functions
This simplifies transition functions to make it easier to reliably
elide comments in a later CL.
Before:
- transition functions are responsible for detecting special end tags.
After:
- the code to detect special end tags is done in one place.
We were relying on end tags being skipped which meant we were
not noticing comments inside script/style elements that contain no
substitutions.
This change means we will notice all such comments where necessary,
but stripTags will notice none since it does not need to. This speeds
up stripTags.
Ian Lance Taylor [Thu, 22 Sep 2011 00:25:48 +0000 (17:25 -0700)]
test: match gccgo error messages
bug363.go:13:12: error: invalid context-determined non-integer type for shift operand
bug363.go:16:12: error: invalid context-determined non-integer type for shift operand
pointer.go:34:6: error: incompatible type in initialization (pointer to interface type has no methods)
pointer.go:36:6: error: incompatible type in initialization
method2.go:15:1: error: invalid pointer or interface receiver type
method2.go:16:1: error: invalid pointer or interface receiver type
method2.go:21:1: error: invalid pointer or interface receiver type
method2.go:22:1: error: invalid pointer or interface receiver type
method2.go:28:15: error: type ‘*Val’ has no method ‘val’
method2.go:33:11: error: reference to undefined field or method ‘val’
shift1.go:19:16: error: invalid context-determined non-integer type for shift operand
shift1.go:24:19: error: invalid context-determined non-integer type for shift operand
shift1.go:25:17: error: invalid context-determined non-integer type for shift operand
shift1.go:18:18: error: shift of non-integer operand
shift1.go:26:13: error: floating point constant truncated to integer
shift1.go:33:15: error: integer constant overflow
shift1.go:34:15: error: integer constant overflow
shift1.go:35:17: error: integer constant overflow
R=golang-dev, r
CC=golang-dev
https://golang.org/cl/5081051
The documentation for bytes.Replace says it copies
the slice but it won't necessarily copy them. Since
the data is mutable, breaking the contract is an issue.
We either have to fix this by making the copy at all
times, as suggested in this CL, or we should change the
documentation and perhaps make better use of the fact
it's fine to mutate the slice in place otherwise.
Ian Lance Taylor [Tue, 20 Sep 2011 23:47:17 +0000 (16:47 -0700)]
test: match gccgo error messages
Added a return to bug357.go to avoid an error which gccgo
reports but 6g does not.
bug353.go:16:14: error: reference to undefined identifer ‘io.ReadWriterCloser’
bug357.go:18:2: error: value computed is not used
bug358.go:14:11: error: imported and not used: ioutil
bug358.go:19:9: error: invalid use of type
bug359.go:25:14: error: redefinition of ‘a’
bug359.go:25:6: note: previous definition of ‘a’ was here
bug359.go:19:6: error: incompatible type in initialization (implicit assignment of ‘list.List’ hidden field ‘front’)
bug362.go:13:6: error: iota is only defined in const declarations
bug362.go:14:6: error: iota is only defined in const declarations
bug362.go:15:6: error: iota is only defined in const declarations
bug363.go:13:12: error: shift of non-integer operand
bug363.go:16:12: error: shift of non-integer operand
Robert Griesemer [Tue, 20 Sep 2011 21:36:19 +0000 (14:36 -0700)]
suffixarray: improved serialization code
Use gobs to serialize indexes instead of encoding/binary.
Even with gobs, serialize data in slices instead of
applying gob to the entire data structure at once,
to reduce the amount of extra buffer memory needed
inside gob.
7x faster Write/Read for new BenchmarkSaveRestore
compared to old code; possibly because encoding/binary
is more expensive for int32 slice elements (interface
call to get little/big endian encoding), while gob's
encoding is fixed (unconfirmed).
new (using gobs):
suffixarray.BenchmarkSaveRestore 1 2153604000 ns/op
old (using encoding/binary):
suffixarray.BenchmarkSaveRestore 1 15118322000 ns/op
The actual serialized data is slightly larger then using
the old code for very large indices because full 32bit indices
require 5bytes using gobs instead of 4bytes (encoding/binary)
in serialized form.