NON-OS/nonos-micro-kernel

Watch

Fork 0

Secondary CPUs come up for the first time, and four defects that stopped them #9

Open

nonos-sync wants to merge 7 commits from nonos-sync/gh-479 into main AGit

nonos-sync commented

2026-09-11 11:22:35 -04:00

Member

The AP bring-up path had never executed on any machine: every QEMU lane pinned -smp 1 and no profile enabled nonos-smp. Turning it on found four defects, each fixed in its own commit.

The trampoline GDT lives in a page mapped read and execute only, and loading a segment register makes the processor write the Accessed bit into the descriptor. That write faulted with no IDT loaded, so the page fault became a double fault and a triple fault that reset the machine before the BSP's timeout fired. The descriptors are now emitted with the bit preset, which keeps W^X.

apic::id() returned one global cache, so every AP recorded the boot CPU's id and the TLB shootdown addressed all its IPIs there. cpus_online() is a population count and was being used as an index bound, which with any failed AP targets a dead slot and skips a live CPU. And the level-assert bit was missing from all eight runtime IPI sends; it is the only non-zero mode constant, so the command word looked complete and sent nothing.

A report_stuck dump prints every CPU's state before a shootdown timeout halts, lock_responsive lets a CPU spinning on the run-queue lock still answer shootdowns, and make qemu-smp is the lane that boots the desktop profile on four CPUs and reports [SMP-PROOF] cpu_count=4.

Shipped profiles stay single-CPU. The shootdown receive side still has an open timeout and AP startup is not yet stable across runs; this is the bring-up, not the switch-on.

Opened on GitHub by eKisNonos as pull request 479. Review and merge happen there while this repository is kept in step from GitHub; this copy follows it, and is marked merged or closed when it is.

The AP bring-up path had never executed on any machine: every QEMU lane pinned `-smp 1` and no profile enabled `nonos-smp`. Turning it on found four defects, each fixed in its own commit. The trampoline GDT lives in a page mapped read and execute only, and loading a segment register makes the processor write the Accessed bit into the descriptor. That write faulted with no IDT loaded, so the page fault became a double fault and a triple fault that reset the machine before the BSP's timeout fired. The descriptors are now emitted with the bit preset, which keeps W^X. `apic::id()` returned one global cache, so every AP recorded the boot CPU's id and the TLB shootdown addressed all its IPIs there. `cpus_online()` is a population count and was being used as an index bound, which with any failed AP targets a dead slot and skips a live CPU. And the level-assert bit was missing from all eight runtime IPI sends; it is the only non-zero mode constant, so the command word looked complete and sent nothing. A `report_stuck` dump prints every CPU's state before a shootdown timeout halts, `lock_responsive` lets a CPU spinning on the run-queue lock still answer shootdowns, and `make qemu-smp` is the lane that boots the desktop profile on four CPUs and reports `[SMP-PROOF] cpu_count=4`. Shipped profiles stay single-CPU. The shootdown receive side still has an open timeout and AP startup is not yet stable across runs; this is the bring-up, not the switch-on. --- Opened on GitHub by eKisNonos as [pull request 479](https://github.com/NON-OS/nonos-micro-kernel/pull/479). Review and merge happen there while this repository is kept in step from GitHub; this copy follows it, and is marked merged or closed when it is.

nonos-sync added 7 commits

2026-09-11 11:22:35 -04:00

smp: preset the accessed bit in the trampoline GDT descriptors 6ea1cbd817

Loading a segment register makes the processor set bit 40 of the
descriptor, and that is a write into the page that holds the GDT. The
trampoline GDT lives in the trampoline page, which is identity-mapped
read and execute only so the low mapping keeps W^X. The write faulted,
the AP had no IDT, and the page fault turned into a double fault and a
triple fault that reset the machine before the BSP's timeout fired.

Emitting every descriptor with the bit already set leaves the processor
nothing to write, which keeps the page free of writes without making it
writable. The long-mode stage's comment now says why removing the code's
own stores was necessary but not sufficient.

apic: read the calling CPU's id from its own APIC once initialised b0ad03567f

CACHED_ID is one global word written by whichever CPU brought the APIC
up, so every secondary asking for its own id was told the boot CPU's.
percpu::init_ap recorded apic_id 0 for every AP, and the TLB shootdown
then addressed its IPIs by that field: every round went to the boot CPU
and none to the CPUs that owed an acknowledgement.

Before the APIC is initialised there is one CPU and the cache holds its
id, and the MMIO base does not exist yet, so that path stays on the
cache. After it, the id comes from the hardware of the CPU asking.

apic: carry level-assert on every runtime IPI e6da329150

A fixed-delivery interrupt command without the level-assert bit is
dropped; the processor only honours a clear bit for an INIT de-assert.
Of the mode constants this file ORs together it is the only one that is
not zero, so the command word looked complete while sending nothing.
ipi_ap.rs passed the bit by hand, which is why AP startup worked while
every runtime IPI, the TLB shootdown included, went nowhere.

One ICR_SEND constant now carries it into all eight sends.

smp: enumerate CPUs by slot state, not by the online count ca591f4c4b

cpus_online is a population count. CPU numbers are handed out once per
AP attempted and are never reused when one fails to come up, so with a
single failed AP the live numbers are sparse and the largest of them is
above the count. Walking 0..count in the shootdown then targeted a slot
that never started, which can never acknowledge, and skipped a CPU that
was running, which never got the IPI. The wait always reached its
deadline and halted the machine.

Both shootdown loops now walk 0..MAX_CPUS and ask cpu_is_online, which
reads the descriptor's state rather than inferring it from a count.

smp: dump every CPU's state when a shootdown round times out e06a0c69b1

A timeout says only that an acknowledgement did not arrive. Which CPU
owed it, whether it was marked as a target, whether it sits in its idle
loop or inside an interrupt handler, and how deep its interrupt masking
goes are what separate an IPI that was never delivered from one that
was delivered to a CPU in no position to run it. The dump is printed
before the panic IPI so the answer survives the halt.

sched: take the run-queue lock through lock_responsive 29fd51822d

Every path that touches the run queue masks interrupts for the critical
section, so a timer tick cannot spin on a lock the interrupted code can
never release. On more than one CPU that masking has a cost: a CPU
spinning on the queue lock cannot answer a TLB shootdown, and the CPU
that sent it waits for exactly that answer before releasing what it
holds. lock_responsive keeps interrupts masked and services shootdowns
from inside the spin, which is what stops the two from waiting on each
other. The five queue sites use it.

build: an smp lane that boots the desktop profile on four CPUs 62e35375e4

Every QEMU lane pinned -smp 1 and no profile enabled nonos-smp, so the
AP bring-up path had never executed on any machine. nonos-mk-smp-prod
builds the desktop profile with the feature on, the same capsule set and
the same attestation, so a difference between this boot and the single
CPU one is the bring-up and nothing else. make qemu-smp runs it and the
[SMP-PROOF] line in the serial log is the kernel reporting how many
cores it brought online. CPUS=n overrides the default of four.

This pull request can be merged automatically.

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin +refs/pull/9/head:nonos-sync/gh-479

git switch nonos-sync/gh-479

No reviewers

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

NON-OS/nonos-micro-kernel!9

No description provided.

Rows
Columns