]> Cypherpunks repositories - gostls13.git/commit
os, syscall: support ill-formed UTF-16 strings on Windows
authorqmuntal <quimmuntal@gmail.com>
Fri, 5 May 2023 16:17:18 +0000 (18:17 +0200)
committerQuim Muntal <quimmuntal@gmail.com>
Mon, 15 May 2023 09:26:16 +0000 (09:26 +0000)
commit974236bda9b9aad87b4b10ec9af2cc01b14e382f
tree321cb16cc9fc2eec78fc81060ff64db06115a544
parent91b8cc0dfaae12af1a89e2b7ad3da10728883ee1
os, syscall: support ill-formed UTF-16 strings on Windows

Windows UTF-16 strings can contain unpaired surrogates, which can't be
decoded into a valid UTF-8 string. This file defines a set of functions
that can be used to encode and decode potentially ill-formed UTF-16
strings by using the
[the WTF-8 encoding](https://simonsapin.github.io/wtf-8/).

WTF-8 is a strict superset of UTF-8, i.e. any string that is
well-formed in UTF-8 is also well-formed in WTF-8 and the content
is unchanged. Also, the conversion never fails and is lossless.

The benefit of using WTF-8 instead of UTF-8 when decoding a UTF-16
string is that the conversion is lossless even for ill-formed
UTF-16 strings. This property allows to read an ill-formed UTF-16
string, convert it to a Go string, and convert it back to the same
original UTF-16 string.

Fixes #59971

Change-Id: Id6007f6e537844913402b233e73d698688cd5ba6
Reviewed-on: https://go-review.googlesource.com/c/go/+/493036
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
Run-TryBot: Quim Muntal <quimmuntal@gmail.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Paul Hampson <Paul.Hampson@Pobox.com>
12 files changed:
src/internal/syscall/execenv/execenv_windows.go
src/internal/syscall/windows/registry/value.go
src/internal/syscall/windows/syscall_windows.go
src/os/dir_windows.go
src/os/exec/lp_windows_test.go
src/os/file_windows.go
src/os/os_windows_test.go
src/syscall/env_windows.go
src/syscall/export_windows_test.go
src/syscall/syscall_windows.go
src/syscall/wtf8_windows.go [new file with mode: 0644]
src/syscall/wtf8_windows_test.go [new file with mode: 0644]