runtime: goroutine on a Cortex-A520 core stalls for up to 9.5 s during GC; lengthening suspendG's 5µs signal resend interval removes it #82146

Go version

go version go1.27.2 linux/arm64. This is the latest release. The stall also appears on 14 older releases, go1.14.15 through go1.27.0 (table below), and on tip: go1.28-devel_2c3c1caad2 (golang/go 2c3c1ca, 2026-10-10) built with make.bash, 6 of 6 A520 runs stalled (median 582.7 ms, max 4893.6 ms), against 6 of 6 for go1.27.2 in the same run (median 401.8 ms, max 5124.0 ms), and 0 of 6 stalled with asyncpreemptoff=1, 0 of 6 on the A720 loops and 0 of 18 on x86-64. The resend timing in suspendG (nextPreemptM = now + yieldDelay/2) is unchanged on that commit (preempt.go:225-226). Against go1.27.2 the only functional difference in suspendG/preemptM is 12c76d4 ("runtime: synchronize Windows preemption with G status"), which moves the casfrom_Gscanstatus(gp, _Gscanrunning, _Grunning) call into preemptM(gp); the signal is still sent after that call and with the same pacing.

Output of go env in your module/workspace:

AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='0'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE='on'
GOARCH='arm64'
GOARM64='v8.0'
GOAUTH='netrc'
GOBIN='/tmp/cstress/upfacts-1036731699/gopath/bin'
GOCACHE='/tmp/cstress/upfacts-1036731699/gocache'
GOCACHEPROG=''
GODEBUG=''
GOENV=''
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS='-modcacherw'
GOGCCFLAGS='-fPIC -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/cstress/upfacts-1036731699/go-build2427250262=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='arm64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/dev/null'
GOMODCACHE='/tmp/cstress/upfacts-1036731699/gomodcache'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPACKAGESDRIVER=''
GOPATH='/tmp/cstress/upfacts-1036731699/gopath'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/tmp/cstress/upfacts-1036731699/gomodcache/golang.org/toolchain@v0.0.1-go1.27.2.linux-arm64'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/tmp/cstress/upfacts-1036731699/xdg/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='go1.27.2'
GOTOOLDIR='/tmp/cstress/upfacts-1036731699/gomodcache/golang.org/toolchain@v0.0.1-go1.27.2.linux-arm64/pkg/tool/linux_arm64'
GOVCS=''
GOVERSION='go1.27.2'
GOWORK='off'
PKG_CONFIG='pkg-config'

This is go env for the go1.27.2 toolchain as the test harness ran it. The paths are its scratch directories.

Hardware: bare metal, Ubuntu 26.04.1, kernel 7.0.0-31-generic #31-Ubuntu SMP PREEMPT_DYNAMIC, 12 cores, performance governor with every core at its maximum frequency:

• cpus 2-5: Cortex-A520 r0p1 (MIDR 0x410fd801), cpu_capacity 279, 1.8 GHz.
• cpus 0-1 and 6-11: Cortex-A720 r0p1 (MIDR 0x410fd811), cpu_capacity 866-1024, 2.2-2.6 GHz.

The x86-64 control is a 16-CPU machine running Ubuntu 26.04 on the same kernel version.

What did you do?

Check first whether your machine can reproduce this (30 s). sigcost.c (below) times a self-tgkill(SIGURG) round trip in plain C, with a handler installed the way Go installs its own. I have only seen the stall natively on cores where the self p50 is well above 5 µs. The p50 covers the whole round trip. If the hypothesis below is right, only the part after the handler acknowledges the signal matters, so a p50 under 5 µs rules a core out but a high p50 does not rule it in. Without an A520, see "Reproducing without Cortex-A520 hardware" below.

cc -O2 -pthread -o sigcost sigcost.c && ./sigcost <cpu>

Reproducer (main.go below; standard library only, Linux only). Four goroutines each call runtime.LockOSThread, pin their thread to one CPU with sched_setaffinity, and loop on epoll_wait (idle epoll fd, 1 ms timeout) followed by about 20 µs of pure-Go arithmetic. Four allocator goroutines produce 256 MiB/s of pointer-bearing garbage, which keeps GC running (about 46 cycles per 30 s). The program reports the longest gap between two epoll_wait returns on any pinned loop, and prints STALL if that gap exceeds 50 ms. The expected gap is about 8 ms.

go run main.go                              # loops on the 4 lowest-capacity CPUs (here the A520s)
go run main.go -class=big                   # loops on the other CPUs (A720s): control
GODEBUG=asyncpreemptoff=1 go run main.go    # control

Every run below lasts 30 s (the default -secs).

What did you see happen?

In the stalled runs, the longest gap had a median of 663.5 ms and a maximum of 9495.5 ms.

Every release I ran stalled, go1.14.15 through go1.27.2, A520 loops, default settings:

With asyncpreemptoff=1, every release had 0 stalled runs. I could not run anything older than go1.14.15, so I have no data from before async preemption existed.

Usually only the pinned goroutine stops. The program also runs an unpinned goroutine that sleeps 1 ms in a loop. In 146 of the 153 stalled runs, that goroutine's longest gap stayed under 50 ms (median 18.8 ms; 18.0-18.8 ms in arm64 runs without a stall). In the other 7 it was 62.9-3935.6 ms. GC kept completing cycles: 39-50 per 30 s in stalled runs, against 43-49 with asyncpreemptoff=1.

One signal costs about 12x more on an A520 than on an A720. Compute is only 2.9x slower.

The C column is the median of per-run p50s, each run 100k samples: 11 runs on arm64, 7 on x86. The Go columns come from one run, and each range spans the CPUs of that type. The cost shows up in plain C, so it lies in the kernel/hardware signal path on this core, not in Go's handler. I do not know its cause. The A520s are r0p1, which is inside the range covered by the kernel's ARM64_ERRATUM_2966298 workaround (r0p0-r0p1). I could not read dmesg to see whether that workaround is active, and I have not tested whether it matters.

Changing only the suspendG resend delay removes the stall. I rebuilt with go build -overlay replacing src/runtime/preempt.go, interleaving the arms run by run. Every run used the A520 loops, 6 runs of 30 s per arm. Each cell shows stalled runs, then the median / max of the longest gap in ms.

What each arm does:

• no-op overlay: the file unchanged. Its runtime.suspendG builds to the same instructions as the unmodified toolchain (hash of go tool objdump).
• backoff: the resend delay starts at yieldDelay/2 and doubles after each resend, capped at 1 ms. It is reset on each suspendG call.
• min100us: nextPreemptM = now + 100*1000.
• CL 842745's change only: patchset 1 of CL 842745, which assigns nextPreemptM = nanotime() + yieldDelay/2 after preemptM returns. I applied it to go1.27.2's preemptM(asyncM) line without its parent CL 842725. It shortened the gaps, but all 6 runs still exceeded 50 ms.

The backoff and min100us results match asyncpreemptoff=1 (0/6, 8.3 / 9.2 ms, go1.27.0, same day). On the A720 loops and on x86-64, the two patched arms and the unmodified toolchain all gave 0 stalls (36/36 runs, go1.27.0). These patches are diagnostics, not proposed fixes: they lengthen the resend interval everywhere, and I have not measured what that costs in suspend or STW latency on fast cores.

What did you expect to see?

On the A520, with async preemption on, the loop should keep its gap near 8 ms, the same as with asyncpreemptoff=1, on the A720s and on x86-64.

Hypothesis, not verified. I have not captured the PC where the goroutine is stuck.

doSigPreempt acknowledges every signal (preemptGen.Add(1) and signalPending.Store(0), signal_unix.go:353-354), including signals that arrive where isAsyncSafePoint refuses. suspendG sees preemptGen change and resends once 5 µs have passed since its previous send (preempt.go:213, 231-233). The handler runs with all signals blocked (os_linux.go:481). So whatever the target thread still has to do after the acknowledgement (the rest of the handler, sigreturn, the return to user mode) races the next send. If that post-acknowledgement path takes longer than it takes suspendG to see the acknowledgement and send again (at least 5 µs after its previous send), the next SIGURG is already pending when sigreturn unblocks signals, and a goroutine caught at an unsafe point never runs past it. The commit that added the rate limit (b89b462, CL 204340) names this hazard: "if the first attempt catches the G at a non-preemptible point, the busy loop in suspendG may hammer it so hard that it never makes it past that non-preemptible point."

This is not a signal flood: signalPending allows at most one SIGURG in flight per M (signal_unix.go:376). One thing I cannot explain: the 9.5 s run still completed 40 GC cycles (typical: 46), so the goroutine was not frozen for the whole gap.

On the A520 the whole round trip is about 22 µs. I have not measured how much of it comes after the acknowledgement. On other hardware, a delay inserted after the acknowledgement is enough to reproduce the stall, and the stall grows with the delay (next section). That is emulation, not proof that the A520 stalls for the same reason.

Reproducing without Cortex-A520 hardware

sigstop.c (below) is a 65-line ptrace tracer. It resumes every signal-delivery stop at once. With SIGSTOP_SIGRETURN=1 it also stops the tracee at every rt_sigreturn, using a seccomp RET_TRACE filter, and busy-waits SIGSTOP_DELAY_US µs there before resuming. That stop comes after doSigPreempt has cleared signalPending. I ran it on an Apple M5 Mac in Docker Desktop with 8 vCPUs (golang:1.27 image, go1.27.1 when I ran it, linux/arm64; I did not record the VM kernel), from a directory holding main.go and sigstop.c. It needs at least 8 CPUs, or taskset -c 4-7 fails. An arm64 Linux machine without Docker should also work (untested); the docker flags only matter in a container.

docker run -it --rm --cap-add SYS_PTRACE --security-opt seccomp=unconfined -v "$PWD":/w -w /w golang:1.27 bash
gcc -O2 -o sigstop sigstop.c && go build -o stallrepro main.go
./stallrepro -cpus 0-3
SIGSTOP_SIGRETURN=1 SIGSTOP_DELAY_US=150 taskset -c 4-7 ./sigstop ./stallrepro -cpus 0-3
GODEBUG=asyncpreemptoff=1 SIGSTOP_SIGRETURN=1 SIGSTOP_DELAY_US=150 taskset -c 4-7 ./sigstop ./stallrepro -cpus 0-3

taskset keeps the tracer off the loops' CPUs. The tracee inherits that mask, so traced runs have GOMAXPROCS=4 (untraced: 8) and need the explicit -cpus 0-3. The filter matches only arm64. On x86-64 it adds no stops, so check that sigreturn_stops in sigstop's stderr is above 0. Do not try D=400: those runs took 6-16 minutes each.

Arms interleaved, 30 s per run. Longest gap: median (max) over all runs of the arm. Self round trip: sigcost p50 under the same tracer settings.

At D=50 the tracer was busy for about 3% of the run (12,014 stops × 73.92 µs), and all 4 runs still stalled. From D=150 up, the single tracer was busy almost the whole run (161,832 × 174.62 µs ≈ 28 s of 30 s), so part of those long gaps may be threads queued behind the tracer. The asyncpreemptoff=1 row is a weak control: no SIGURG was sent, so nothing was delayed. Stops before the acknowledgement alone did not stall (0/12), but that control was not run in the same setup as the table (details below).

Not tested / limits:

• other A520 systems or other kernels;
• a process confined to little cores without per-thread LockOSThread + sched_setaffinity;
• the backoff/min100us changes on tip (tip was run unpatched only, at 2c3c1ca, one dispatch of 6 runs per case);
• CL 842725 + CL 842745 together;
• native macOS: one test of a darwin port of main.go on an Apple M5 did not reproduce it. With threads at QoS background, 1/6 runs crossed 50 ms with async preemption on and 1/6 with it off, so QoS throttling confounds the result. I did not check which cores the threads ran on.

Would a change to suspendG's resend pacing be acceptable? For example, it could back off when a signal was acknowledged but the goroutine did not stop, or scale the delay with the measured send-to-acknowledge time. I can run instrumented builds or candidate CLs on this machine.

Related:

• runtime: sometimes 100% CPU spin during init phase in Go 1.14 with preemptive scheduler #37741: darwin preemption-signal livelock, fixed by CL 223737 (one pending signal per M). That limit is in effect here.
• runtime: slowdown on arm64 in GC while scanning stacks #37779: arm64 suspendG slowdown. There, raising yieldDelay helped, and the reporter asked whether pacing should adapt to the platform. It was fixed only for the cleantimers case (CL 223122).
• CL 223240: a sender-side throttle, abandoned in favour of CL 223737. The review judged suspendG's existing backoff sufficient; that backoff is the 5 µs above.
• runtime: reducing preemption in suspendG when G is running large nosplit functions #40229 (open): suspendG livelock against long nosplit functions on arm64. The plan there is to make those functions preemptible. Here the same code stalls only on the A520 and never on the A720, so the unsafe window does not look long. I am happy to move this into runtime: reducing preemption in suspendG when G is running large nosplit functions #40229 if you prefer.
• CL 842725 (missed retries after a skipped attempt) and CL 842745 (Windows: start the delay after preemptM returns), both open. CL 842745's change alone reduces but does not remove the stall (table above).
• runtime/trace: emit events for failed async preemption attempts #82084 (open): trace events for failed async-preemption attempts. These would show the stuck PC here.
• runtime: stop-the-world never completes while goroutines are blocked in read(2) on a pipe (go1.26 regression) #81759 (open): STW never completes with goroutines in read(2), a go1.26 regression. Different problem: this one goes back to go1.14.15 and a suspendG-only change removes it.
• runtime, x/mobile: Severe performance regression since 1.18+ on iOS, when combined with Xcode #57651, runtime: stalls under Rosetta 2 #42700, runtime: async preemption support on ARM64 breaks ARM64 on QEMU #36981: slow signal delivery under LLDB, Rosetta 2 and qemu-user, answered with asyncpreemptoff=1. This report is about bare metal with a stock kernel, and that setting is process-wide.

// Command stallrepro: does GC stall a process whose LockOSThread'd goroutines are pinned to a
// slow core? Linux only, standard library only.
//
//	go run main.go                        # -class=little: CPUs with cpu_capacity <= half the largest (max 4)
//	go run main.go -class=big             # the other CPUs (max 8): control
//	go run main.go -cpus=2-5 -secs=30     # explicit CPU list
//	GODEBUG=asyncpreemptoff=1 go run main.go   # async preemption off: the control that cures it
//
// Each loop is LockOSThread()ed, pinned with sched_setaffinity to one CPU, and loops on
// epoll_wait(idle epfd, 1 ms) plus ~20 us of pure-Go work per wake-up (calibrated on the fastest
// core, so a slow core runs Go code longer). Four allocator goroutines churn pointer-bearing garbage
// (256 MiB/s) over a 32 MiB live heap so GC cycles keep running (GOGC=100 unless set). Wake gap =
// time between two epoll_wait returns that are not EINTR. Prints one RESULT line; verdict STALL
// (exit 1) if the longest wake gap exceeds -threshold-ms. On one core type "little"/"big" are the
// lowest/highest CPU ids (negative control). max_tick_gap_ms: longest gap of an unpinned 1 ms ticker goroutine.
package main

import (
	"flag"
	"fmt"
	"io/ioutil"
	"os"
	"runtime"
	"runtime/debug"
	"strings"
	"sync/atomic"
	"syscall"
	"time"
	"unsafe"
)

type node struct {
	next *node
	_    [56]byte
}

var (
	sink uint64
	stop int32
	t0   = time.Now()
)

//go:noinline
func spin(n int) (x uint64) {
	for x = 88172645463325252; n > 0; n-- {
		x ^= x << 13
		x ^= x >> 7
		x ^= x << 17
	}
	return x
}

// affinity pins the calling thread to cpu (cpu >= 0), or returns the CPUs it may run on.
func affinity(cpu int) (set []int) {
	var m [16]uint64
	call := uintptr(syscall.SYS_SCHED_GETAFFINITY)
	if cpu >= 0 {
		call, m[cpu/64] = syscall.SYS_SCHED_SETAFFINITY, 1<<(uint(cpu)%64)
	}
	if _, _, e := syscall.RawSyscall(call, 0, unsafe.Sizeof(m), uintptr(unsafe.Pointer(&m[0]))); e != 0 {
		panic(e)
	}
	for c := 0; c < 1024; c++ {
		if m[c/64]>>(uint(c)%64)&1 == 1 {
			set = append(set, c)
		}
	}
	return set
}

func capacity(cpu int) (n int) { // 0 when the kernel does not export cpu_capacity (x86)
	b, _ := ioutil.ReadFile(fmt.Sprintf("/sys/devices/system/cpu/cpu%d/cpu_capacity", cpu))
	fmt.Sscan(string(b), &n)
	return n
}

func gcCycles() int64 { // takes only the heap lock, no stop-the-world (unlike ReadMemStats)
	var st debug.GCStats
	debug.ReadGCStats(&st)
	return st.NumGC
}

func main() {
	secs := flag.Int("secs", 30, "measuring time, seconds")
	list := flag.String("cpus", "", "CPU list, one loop on each, e.g. 2-5 (default: by -class)")
	class := flag.String("class", "little", "little|big, from cpu_capacity")
	thr := flag.Float64("threshold-ms", 50, "STALL when the longest wake gap exceeds this")
	flag.Parse()
	all := affinity(-1)
	hi, fast := 0, all[0]
	for _, c := range all {
		if k := capacity(c); k > hi {
			hi, fast = k, c
		}
	}
	var small, large []int // little: cpu_capacity <= half the largest; big: the rest
	for _, c := range all {
		if hi > 0 && capacity(c)*2 <= hi {
			small = append(small, c)
		} else {
			large = append(large, c)
		}
	}
	hetero := len(small) > 0 && len(large) > 0
	if !hetero { // one core type: little = the 4 lowest ids, big = the 8 highest
		small, large = all, all
	}
	cpus := small
	if *class == "big" {
		cpus = large
	}
	if *class == "little" && len(cpus) > 4 {
		cpus = cpus[:4]
	} else if len(cpus) > 8 {
		cpus = cpus[len(cpus)-8:]
	}
	label := *class
	if *list != "" {
		cpus, label = nil, "cpus="+*list
		for _, p := range strings.Split(*list, ",") {
			a, b := 0, -1
			fmt.Sscanf(p, "%d-%d", &a, &b)
			for c := a; c <= a || c <= b; c++ {
				cpus = append(cpus, c)
			}
		}
	}
	calib := make(chan float64)
	go func() { // ns per spin iteration on the fastest core: best of 5
		runtime.LockOSThread()
		affinity(fast)
		best := 1e18
		for i := 0; i < 5; i++ {
			s := time.Now()
			atomic.AddUint64(&sink, spin(2000000))
			if d := float64(time.Since(s)) / 2e6; d < best {
				best = d
			}
		}
		calib <- best
	}()
	nsPerIter := <-calib
	workIters := int(20000 / nsPerIter)
	last, maxGap := make([]int64, len(cpus)), make([]int64, len(cpus))
	ready := make(chan bool, len(cpus))
	for i, c := range cpus {
		go func(i, c int) {
			runtime.LockOSThread() // never unlocked: the thread dies with the goroutine
			epfd, err := syscall.EpollCreate1(syscall.EPOLL_CLOEXEC)
			if err != nil {
				panic(err)
			}
			affinity(c)
			ready <- true
			evs := make([]syscall.EpollEvent, 8)
			atomic.StoreInt64(&last[i], int64(time.Since(t0)))
			for atomic.LoadInt32(&stop) == 0 {
				if _, err := syscall.EpollWait(epfd, evs, 1); err != nil { // EINTR: a signal, keep waiting
					continue
				}
				now := int64(time.Since(t0))
				if g := now - atomic.LoadInt64(&last[i]); g > atomic.LoadInt64(&maxGap[i]) {
					atomic.StoreInt64(&maxGap[i], g)
				}
				atomic.StoreInt64(&last[i], now)
				atomic.AddUint64(&sink, spin(workIters))
			}
		}(i, c)
	}
	for range cpus {
		<-ready
	}
	live := make([]*node, 32<<20/64)
	for i := range live {
		live[i] = new(node)
	}
	gc0 := gcCycles()
	for a := 0; a < 4; a++ { // 4 allocators, 64 MiB/s of garbage each
		go func(lo, hi int) {
			start, did := time.Now(), 0.0
			for j := 1; atomic.LoadInt32(&stop) == 0; j++ {
				allowed := 64 << 20 * time.Since(start).Seconds()
				if allowed-did > 8<<20 {
					did = allowed - 8<<20 // no catch-up burst after a stall
				}
				if did >= allowed {
					time.Sleep(200 * time.Microsecond)
					continue
				}
				var head *node
				for k := 0; k < 1024; k++ {
					head = &node{next: head}
				}
				did += 1024 * 64
				if j%16 == 0 { // churn the live set so the write barrier is busy during mark
					live[lo+(j/16)%(hi-lo)] = &node{next: head.next.next}
				}
			}
		}(a*len(live)/4, (a+1)*len(live)/4)
	}
	var tickMax int64
	go func() {
		for t := time.Now(); atomic.LoadInt32(&stop) == 0; t = time.Now() {
			time.Sleep(time.Millisecond)
			if g := int64(time.Since(t)); g > atomic.LoadInt64(&tickMax) {
				atomic.StoreInt64(&tickMax, g)
			}
		}
	}()
	time.Sleep(time.Duration(*secs) * time.Second)
	end := int64(time.Since(t0))
	atomic.StoreInt32(&stop, 1)
	gcN := gcCycles() - gc0
	var worst int64
	for i := range cpus {
		g := atomic.LoadInt64(&maxGap[i])
		if open := end - atomic.LoadInt64(&last[i]); open > g { // a gap still open at the end counts
			g = open
		}
		if g > worst {
			worst = g
		}
	}
	verdict := "PASS"
	if float64(worst)/1e6 > *thr {
		verdict = "STALL"
	}
	fmt.Printf("RESULT go=%s goarch=%s class=%s hetero=%v cpus=%s gomaxprocs=%d gogc=%q godebug=%q ns_per_iter=%.3f work_iters=%d gc_cycles=%d max_gap_ms=%.1f max_tick_gap_ms=%.1f threshold_ms=%g verdict=%s\n",
		runtime.Version(), runtime.GOARCH, label, hetero, strings.Trim(strings.ReplaceAll(fmt.Sprint(cpus), " ", ","), "[]"), runtime.GOMAXPROCS(0), os.Getenv("GOGC"), os.Getenv("GODEBUG"), nsPerIter, workIters,
		gcN, float64(worst)/1e6, float64(atomic.LoadInt64(&tickMax))/1e6, *thr, verdict)
	if verdict == "STALL" {
		os.Exit(1)
	}
}

/* sigcost: cost of a SIGURG-like signal on one pinned CPU, in plain C (no Go).
 *   cc -O2 -pthread -o sigcost sigcost.c
 *   ./sigcost <target_cpu> [sender_cpu] [n=100000]
 * The handler is installed like Go's: SA_SIGINFO|SA_ONSTACK|SA_RESTART with a sigaltstack.
 * self:    the target thread tgkill()s itself; time from before the syscall until it returns
 *          (the handler has run and sigreturn is done).
 * cross:   a sender thread pinned to sender_cpu tgkill()s the target thread, which spins in user
 *          code on target_cpu; time from before tgkill until the handler's first instruction
 *          stamps CLOCK_MONOTONIC (send->ack), then the sender waits for the ack.
 * Also prints ns per iteration of a fixed compute loop on target_cpu, cpu_capacity and
 * scaling_cur_freq, to separate "the core is slower" from "signals are slower on this core".
 * One line per path: SIGCOST path=.. cpu=.. n=.. min_us=.. p50_us=.. p99_us=.. max_us=..  */
#define _GNU_SOURCE
#include <errno.h>
#include <pthread.h>
#include <sched.h>
#include <signal.h>
#include <stdatomic.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/syscall.h>
#include <sys/utsname.h>
#include <time.h>
#include <unistd.h>

static _Atomic long ack_ns, target_tid;
static _Atomic int stop;

static long now_ns(void) { struct timespec t; clock_gettime(CLOCK_MONOTONIC, &t); return t.tv_sec * 1000000000L + t.tv_nsec; }
static void handler(int sig, siginfo_t *si, void *uc) { (void)sig; (void)si; (void)uc; atomic_store(&ack_ns, now_ns()); }
static void pin(int cpu) {
	cpu_set_t s; CPU_ZERO(&s); CPU_SET(cpu, &s);
	if (sched_setaffinity(0, sizeof s, &s)) { perror("sched_setaffinity"); exit(2); }
}
static int cmp(const void *a, const void *b) { long x = *(const long *)a, y = *(const long *)b; return x < y ? -1 : x > y; }
static void report(const char *path, int cpu, long *v, int n) {
	qsort(v, n, sizeof *v, cmp);
	printf("SIGCOST path=%s cpu=%d n=%d min_us=%.2f p50_us=%.2f p99_us=%.2f max_us=%.2f\n", path, cpu, n,
	       v[0] / 1e3, v[n / 2] / 1e3, v[n * 99 / 100] / 1e3, v[n - 1] / 1e3);
}
static long readnum(const char *fmt, int cpu) {
	char p[128], b[64]; snprintf(p, sizeof p, fmt, cpu);
	FILE *f = fopen(p, "r"); if (!f) return -1;
	long v = fgets(b, sizeof b, f) ? atol(b) : -1; fclose(f); return v;
}
static void altstack(void) {
	stack_t ss = { .ss_sp = malloc(SIGSTKSZ * 4), .ss_size = SIGSTKSZ * 4 };
	sigaltstack(&ss, NULL);
}
static void install(void) {
	struct sigaction sa; memset(&sa, 0, sizeof sa);
	sa.sa_sigaction = handler; sa.sa_flags = SA_SIGINFO | SA_ONSTACK | SA_RESTART; sigfillset(&sa.sa_mask);
	if (sigaction(SIGURG, &sa, NULL)) { perror("sigaction"); exit(2); }
}
static volatile unsigned long long sink;
static void *target(void *arg) { /* spins in user code until told to stop; signals interrupt it */
	pin((int)(long)arg); altstack();
	atomic_store(&target_tid, syscall(SYS_gettid));
	unsigned long long x = 88172645463325252ULL;
	while (!atomic_load(&stop)) { x ^= x << 13; x ^= x >> 7; x ^= x << 17; }
	sink = x; return NULL;
}

int main(int argc, char **argv) {
	if (argc < 2) { fprintf(stderr, "usage: %s target_cpu [sender_cpu] [n]\n", argv[0]); return 2; }
	int tcpu = atoi(argv[1]), scpu = argc > 2 ? atoi(argv[2]) : (tcpu == 0 ? 1 : 0), n = argc > 3 ? atoi(argv[3]) : 100000;
	struct utsname u; uname(&u);
	long *v = malloc(n * sizeof *v);
	install();
	pid_t pid = getpid();
	pin(tcpu); altstack();
	/* compute speed of this core: best of 5 x 10M iterations */
	double best = 1e18; unsigned long long x = 88172645463325252ULL;
	for (int r = 0; r < 5; r++) {
		long t = now_ns();
		for (int i = 0; i < 10000000; i++) { x ^= x << 13; x ^= x >> 7; x ^= x << 17; }
		double d = (now_ns() - t) / 1e7; if (d < best) best = d;
	}
	sink = x;
	printf("SIGCOST_FACTS kernel=%s machine=%s cpu=%d cpu_capacity=%ld scaling_cur_freq_khz=%ld compute_ns_per_iter=%.3f\n",
	       u.release, u.machine, tcpu, readnum("/sys/devices/system/cpu/cpu%d/cpu_capacity", tcpu),
	       readnum("/sys/devices/system/cpu/cpu%d/cpufreq/scaling_cur_freq", tcpu), best);
	pid_t me = (pid_t)syscall(SYS_gettid);
	for (int i = 0; i < n; i++) { /* self path */
		long t = now_ns();
		syscall(SYS_tgkill, pid, me, SIGURG);
		v[i] = now_ns() - t;
	}
	report("self", tcpu, v, n);
	pthread_t th; pthread_create(&th, NULL, target, (void *)(long)tcpu);
	while (!atomic_load(&target_tid)) usleep(1000);
	usleep(10000); pin(scpu);
	for (int i = 0; i < n; i++) {
		atomic_store(&ack_ns, 0);
		long t = now_ns();
		syscall(SYS_tgkill, pid, (pid_t)atomic_load(&target_tid), SIGURG);
		long a; while (!(a = atomic_load(&ack_ns))) ;
		v[i] = a - t;
	}
	atomic_store(&stop, 1); pthread_join(th, NULL);
	report("cross_spin", tcpu, v, n);
	return 0;
}

/* sigstop: minimal ptrace tracer for `sigstop CMD ARGS...` (PTRACE_SEIZE + TRACECLONE). It stops the tracee only on
 * signal-delivery stops and, with SIGSTOP_SIGRETURN=1, on rt_sigreturn (see below); no other syscall stops. On each stop
 * it does nothing but PTRACE_CONT, re-injecting the signal. Purpose: make each signal delivery cost a stop + tracer
 * wake + restart, with the least tracer-side variance. Prints stop counts to stderr at exit.
 *   cc -O2 -o sigstop sigstop.c */
#define _GNU_SOURCE
#include <errno.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/ptrace.h>
#include <sys/wait.h>
#include <unistd.h>
#include <linux/ptrace.h>
#include <linux/seccomp.h>
#include <linux/filter.h>
#include <linux/audit.h>
#include <sys/prctl.h>
#include <stddef.h>
#include <string.h>
#include <time.h>
/* SIGSTOP_DELAY_US=N: busy-wait N us in the tracer before resuming from each rt_sigreturn stop (extra post-ack delay). */
static void spin_us(long us) { struct timespec t; clock_gettime(CLOCK_MONOTONIC, &t); long e = t.tv_sec * 1000000000L + t.tv_nsec + us * 1000L;
	do { clock_gettime(CLOCK_MONOTONIC, &t); } while (t.tv_sec * 1000000000L + t.tv_nsec < e); }
/* SIGSTOP_SIGRETURN=1: also stop on every rt_sigreturn (arm64 nr 139) via a seccomp RET_TRACE filter, i.e. add a ptrace
 * stop AFTER the handler has run (after Go's signalPending was cleared); all other syscalls run untraced. */
static void install_sigreturn_filter(void) {
	struct sock_filter f[] = {
		BPF_STMT(BPF_LD | BPF_W | BPF_ABS, offsetof(struct seccomp_data, arch)),
		BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, AUDIT_ARCH_AARCH64, 1, 0),
		BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_ALLOW),
		BPF_STMT(BPF_LD | BPF_W | BPF_ABS, offsetof(struct seccomp_data, nr)),
		BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, 139, 0, 1),
		BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_TRACE),
		BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_ALLOW),
	};
	struct sock_fprog prog = { .len = sizeof f / sizeof f[0], .filter = f };
	if (prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0) || prctl(PR_SET_SECCOMP, SECCOMP_MODE_FILTER, &prog)) { perror("seccomp"); _exit(126); }
}
int main(int argc, char **argv) {
	if (argc < 2) { fprintf(stderr, "usage: %s cmd args...\n", argv[0]); return 2; }
	int p[2]; if (pipe(p)) { perror("pipe"); return 2; }
	pid_t child = fork();
	if (child == 0) { char c; close(p[1]); if (read(p[0], &c, 1) < 0) _exit(126); if (getenv("SIGSTOP_SIGRETURN")) install_sigreturn_filter(); execvp(argv[1], argv + 1); perror("exec"); _exit(127); }
	if (ptrace(PTRACE_SEIZE, child, 0, PTRACE_O_TRACECLONE | PTRACE_O_EXITKILL | (getenv("SIGSTOP_SIGRETURN") ? PTRACE_O_TRACESECCOMP : 0)) < 0) { perror("PTRACE_SEIZE"); kill(child, SIGKILL); return 2; }
	close(p[0]); if (write(p[1], "x", 1) != 1) return 2; close(p[1]);
	long delay_us = getenv("SIGSTOP_DELAY_US") ? atol(getenv("SIGSTOP_DELAY_US")) : 0;
	long nsig = 0, nurg = 0, nev = 0, nret = 0; int code = 0, st;
	for (;;) {
		pid_t t = waitpid(-1, &st, __WALL);
		if (t < 0) { if (errno == EINTR) continue; break; }
		if (WIFEXITED(st) || WIFSIGNALED(st)) {
			if (t == child) { code = WIFEXITED(st) ? WEXITSTATUS(st) : 128 + WTERMSIG(st); break; }
			continue;
		}
		if (!WIFSTOPPED(st)) continue;
		int sig = WSTOPSIG(st), ev = st >> 16, inj = 0;
		if (ev == PTRACE_EVENT_SECCOMP) { nret++; if (delay_us) spin_us(delay_us); }
		else if (ev) nev++;              /* PTRACE_EVENT_CLONE / PTRACE_EVENT_STOP (new thread, group stop): continue, no signal */
		else { inj = sig; nsig++; if (sig == SIGURG) nurg++; }   /* signal-delivery-stop: re-inject */
		ptrace(PTRACE_CONT, t, 0, (void *)(long)inj);
	}
	fprintf(stderr, "sigstop: signal_stops=%ld sigurg_stops=%ld event_stops=%ld sigreturn_stops=%ld\n", nsig, nurg, nev, nret);
	return code;
}

main.go and sigcost.c are also at https://github.com/goceleris/celeris/tree/c6d84c3e73d3607a0b456f2a0a07f801d94dcb9e/internal/stallprobe/testdata/upstream . The preempt.go arms are generated by https://github.com/goceleris/celeris/blob/c6d84c3e73d3607a0b456f2a0a07f801d94dcb9e/internal/stallprobe/t8_patches_linux_test.go .