close
Skip to content

runtime: linux/mips time64 nanosecond layout causes busy waits and empty CPU profiles #82147

Description

@cheruvian

Go version

Reproduced with upstream Go 1.26.5 and a Tailscale Go 1.27.1 toolchain. The same timespec definition is present in upstream Go 1.27.1 and current master.

Operating system and architecture

Linux 6.12, GOOS=linux, GOARCH=mips (32-bit big-endian), GOMIPS=softfloat, CGO_ENABLED=0.

Problem

The runtime's Linux/MIPS time64 structure declares tv_sec int64, tv_nsec int32, and four bytes of trailing padding. However, futex_time64 and timer_settime64 receive the userspace ABI structure __kernel_timespec, which declares both fields as 64-bit values.

On big-endian MIPS, the current layout places the nanosecond value in the high 32 bits of the ABI field and zeros in the low 32 bits. Linux converts this to its internal 32-bit long nanoseconds field, discarding the high bits. Sub-second waits therefore effectively become zero-length waits. This causes repeated immediate timeouts, substantial idle CPU, and CPU profiling timers with no samples.

The distinction between the kernel's internal timespec64 and the userspace ABI __kernel_timespec is essential here. This appears related to #77934 / #77935, but the observed current layout remains incompatible with the syscall boundary tested below.

References:

Minimal syscall reproducer

This program targets Linux big-endian MIPS o32 specifically. It requests a 100 ms futex timeout with each layout; the return value should be ETIMEDOUT in both cases, but the elapsed time differs.

package main

import (
 "fmt"
 "syscall"
 "time"
 "unsafe"
)

// Linux MIPS o32 futex_time64 syscall number.
const futexTime64 = 4422

func probe(label string, timeout *[4]uint32) {
 var word uint32
 for i := 0; i < 3; i++ {
  start := time.Now()
  _, _, errno := syscall.Syscall6(futexTime64, uintptr(unsafe.Pointer(&word)), 128, 0, uintptr(unsafe.Pointer(timeout)), 0, 0)
  elapsed := time.Since(start)
  fmt.Printf("%s attempt=%d elapsed_ms=%.3f errno=%d\n", label, i+1, float64(elapsed)/float64(time.Millisecond), errno)
 }
}

func main() {
 // Big-endian words: 64-bit seconds, then nanoseconds and padding.
 runtimeLayout := [4]uint32{0, 0, 100000000, 0}
 // Big-endian words: 64-bit seconds, then 64-bit nanoseconds.
 kernelLayout := [4]uint32{0, 0, 0, 100000000}
 probe("current-runtime-layout", &runtimeLayout)
 probe("64-bit-nanoseconds-layout", &kernelLayout)
}

Build with:

GOOS=linux GOARCH=mips GOMIPS=softfloat CGO_ENABLED=0 go build -o timer-probe main.go

Run on a matching Linux/MIPS target.

Observed elapsed times in milliseconds:

Layout Attempt 1 Attempt 2 Attempt 3
Current runtime (int32 nanoseconds + trailing padding) 0.098 0.043 0.244
ABI layout (int64 nanoseconds) 100.066 100.217 100.078

Independent runtime reproduction

A separate program containing only repeated short sleeps also reproduces substantial CPU consumption:

package main

import "time"

func main() {
    for deadline := time.Now().Add(45 * time.Second); time.Now().Before(deadline); {
        time.Sleep(20 * time.Millisecond)
    }
}

With upstream Go 1.26.5, a 15-second process CPU-counter measurement gave 264/1504 aggregate CPU ticks (17.55%). Changing only the runtime timer layout reduced the same program to 31/1503 ticks (2.06%). A single long time.Sleep(45*time.Second) measured 0/2003 ticks over a 20-second interval.

Full-daemon validation

Two Tailscale 1.104.1 builds from the same source and pinned Go 1.27.1 toolchain were compared using identical settings. Only the runtime timer correction below differed. Both were unauthenticated userspace-networking instances with separate state and sockets, measured one at a time after 20 seconds settling:

  • Control: 493/3005 CPU ticks over 30 seconds (16.41%).
  • Corrected: 1/3005 CPU ticks over 30 seconds (0.03%).
  • Control 20-second CPU profile: zero samples.
  • Corrected 20-second CPU profile: 40 ms of samples.

A short authenticated test with the corrected daemon measured 0.40% idle CPU, and direct tunnel ping, SSH, and a checksum-verified transfer succeeded. This is bounded validation, not a broad compatibility or throughput claim.

Experimental correction

The following correction was tested using the pinned Go 1.27.1 toolchain. The explicit downcast preserves the legacy timespec32 path's field type. A suitable upstream regression test and wider architecture validation are still needed.

--- runtime/defs_linux_mipsx.go
+++ runtime/defs_linux_mipsx.go
@@ -109,14 +109,13 @@
 
 type timespec struct {
 	tv_sec  int64
-	tv_nsec int32
-	_       [4]byte // the C ABI aligns int64 to 8 bytes
+	tv_nsec int64
 }
 
 //go:nosplit
 func (ts *timespec) setNsec(ns int64) {
 	ts.tv_sec = ns / 1e9
-	ts.tv_nsec = int32(ns % 1e9)
+	ts.tv_nsec = ns % 1e9
 }
 
 type timeval struct {
--- runtime/os_linux32.go
+++ runtime/os_linux32.go
@@ -106,6 +106,6 @@
 	if ts.tv_sec != int64(r.tv_sec) {
 		throw("timespec64 value cannot be represented in timespec32")
 	}
-	r.tv_nsec = ts.tv_nsec
+	r.tv_nsec = int32(ts.tv_nsec)
 	return
 }

Expected behavior

Sub-second futex waits should use the requested timeout without busy retrying, and CPU profiling timers should produce samples when the process consumes CPU.

Downstream tracking

Tailscale report: tailscale/tailscale#21757

Activity

  1. Jorropo commented on Oct 12, 2026

    @Jorropo
    Member

    Ah I see what happened, I've unfixed it in #77934 because I've red linux's internal API layout not the UAPI one. 🤦

  2. Jorropo commented on Oct 12, 2026

    @Jorropo
    Member

    @gopherbot please open backport issues for go1.26 and go1.27

  3. gopherbot commented on Oct 12, 2026

    @gopherbot
    Contributor

    Backport issue(s) opened: #82149 (for 1.26), #82150 (for 1.27).

    Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

  4. added
    BugReportIssues describing a possible bug in the Go implementation.
    FixPendingIssues that have a fix which has not yet been reviewed or submitted.
    on Oct 12, 2026
  5. gopherbot commented on Oct 12, 2026

    @gopherbot
    Contributor

    Change https://go.dev/cl/848347 mentions this issue: runtime: fix 64bits timespec layout for 32bits arches

  6. gopherbot commented on Oct 12, 2026

    @gopherbot
    Contributor

    Change https://go.dev/cl/848206 mentions this issue: [release-branch.go1.26] runtime: fix 64bits timespec layout for 32bits arches

  7. gopherbot commented on Oct 12, 2026

    @gopherbot
    Contributor

    Change https://go.dev/cl/848207 mentions this issue: [release-branch.go1.27] runtime: fix 64bits timespec layout for 32bits arches

  8. self-assigned this
    on Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

BugReportIssues describing a possible bug in the Go implementation.FixPendingIssues that have a fix which has not yet been reviewed or submitted.compiler/runtimeIssues related to the Go compiler and/or runtime.

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions