Most operations need an upper bound on the physical page size, which
is what sys.PhysPageSize is for (this is checked at runtime init on
Linux). However, a few operations need a *lower* bound on the physical
page size. Introduce a "minPhysPageSize" constant to act as this lower
bound and use it where it makes sense:
1) In addrspace_free, we have to query each page in the given range.
Currently we increment by the upper bound on the physical page
size, which means we may skip over pages if the true size is
smaller. Worse, we currently pass a result buffer that only has
enough room for one page. If there are actually multiple pages in
the range passed to mincore, the kernel will overflow this buffer.
Fix these problems by incrementing by the lower-bound on the
physical page size and by passing "1" for the length, which the
kernel will round up to the true physical page size.
2) In the write barrier, the bad pointer check tests for pointers to
the first physical page, which are presumably small integers
masquerading as pointers. However, if physical pages are smaller
than we think, we may have legitimate pointers below
sys.PhysPageSize. Hence, use minPhysPageSize for this test since
pointers should never fall below that.
In particular, this applies to ARM64 and MIPS. The runtime is
configured to use 64kB pages on ARM64, but by default Linux uses 4kB
pages. Similarly, the runtime assumes 16kB pages on MIPS, but both 4kB
and 16kB kernel configurations are common. This also applies to ARM on
systems where the runtime is recompiled to deal with a larger page
size. It is also a step toward making the runtime use only a
dynamically-queried page size.
Change-Id: I1fdfd18f6e7cbca170cc100354b9faa22fde8a69
Reviewed-on: https://go-review.googlesource.com/25020
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Run-TryBot: Austin Clements <austin@google.com>
if !writeBarrier.needed {
return
}
- if src != 0 && src < sys.PhysPageSize {
+ if src != 0 && src < minPhysPageSize {
systemstack(func() {
print("runtime: writebarrierptr *", dst, " = ", hex(src), "\n")
throw("bad pointer in write barrier")
if !writeBarrier.needed {
return
}
- if src != 0 && src < sys.PhysPageSize {
+ if src != 0 && src < minPhysPageSize {
systemstack(func() { throw("bad pointer in write barrier") })
}
writebarrierptr_nostore1(dst, src)
)
const (
- _PAGE_SIZE = sys.PhysPageSize
- _EACCES = 13
+ _EACCES = 13
+ _EINVAL = 22
)
// NOTE: vec must be just 1 byte long here.
var addrspace_vec [1]byte
func addrspace_free(v unsafe.Pointer, n uintptr) bool {
- var chunk uintptr
- for off := uintptr(0); off < n; off += chunk {
- chunk = _PAGE_SIZE * uintptr(len(addrspace_vec))
- if chunk > (n - off) {
- chunk = n - off
+ // Step by the minimum possible physical page size. This is
+ // safe even if we have the wrong physical page size; mincore
+ // will just return EINVAL for unaligned addresses.
+ for off := uintptr(0); off < n; off += minPhysPageSize {
+ // Use a length of 1 byte, which the kernel will round
+ // up to one physical page regardless of the true
+ // physical page size.
+ errval := mincore(unsafe.Pointer(uintptr(v)+off), 1, &addrspace_vec[0])
+ if errval == -_EINVAL {
+ // Address is not a multiple of the physical
+ // page size. That's fine.
+ continue
}
- errval := mincore(unsafe.Pointer(uintptr(v)+off), chunk, &addrspace_vec[0])
// ENOMEM means unmapped, which is what we want.
// Anything else we assume means the pages are mapped.
if errval != -_ENOMEM {
"unsafe"
)
+// minPhysPageSize is a lower-bound on the physical page size. The
+// true physical page size may be larger than this. In contrast,
+// sys.PhysPageSize is an upper-bound on the physical page size.
+const minPhysPageSize = 4096
+
// Main malloc heap.
// The heap itself is the "free[]" and "large" arrays,
// but all the other global data is here too.