NON-OS/nonos-micro-kernel

Watch

Fork 0

git: object model, packfiles, clone and push over https #16

Open

nonos-sync wants to merge 24 commits from nonos-sync/gh-435 into main AGit

nonos-sync commented

2026-09-11 11:22:43 -04:00

Member

Git in the NONOS terminal, from the object model up to clone and push over HTTPS. The protocol layers are checked against real git rather than against themselves, and the paths that read remote bytes have been through a hardening pass. What has not happened is a run on a booted image, so this stays open until it has.

What is here

The object model, no_std, written from scratch:

SHA-1, streaming and one shot
the <type> <size>\0<content> framing objects are named by
zlib: a full inflate over stored, fixed and dynamic Huffman blocks, so it reads objects git itself wrote, and a stored-block writer
trees, including the sort rule where a directory compares as though its name ended in a slash, which is what decides the hash
commits, with parents in order and the author and committer lines
the loose object database, HEAD and the branch files
the binary DIRC version 2 index
enough of the config file to record a remote and read it back

Packfiles, read, written, and stored as packs. The reader resolves offset and reference deltas. The writer emits whole objects with a SHA-1 trailer. A fetched pack is kept whole with a version 2 index beside it, the way git does it, and the object store reads through that index: the first byte of an id picks a range out of the fanout, a binary search covers only that range, then the entry is inflated at its offset and its delta chain followed.

The wire protocol. pkt-line framing both ways, the ref advertisement, the want, deepen and done body, and the receive-pack command list.

Clone and push, over a Transport the crate does not implement.

Three crates the transport needed, none of which existed:

Crate	What
`nonos_http`	HTTP/1.1 client framing, response parsing including chunked, and url checking
`nonos_socket`	one client for `net.sockets`, a `TcpStream` that closes on drop
`nonos_tls`	TLS 1.3, moved out of the browser, plus a blocking session

In the terminal: git init, clone, add, status, commit -m, log, push, remote.

How it is checked

Nothing is graded by its own assertions where real git could grade it instead.

A repository built by this crate alone, with git never invoked to create any part of it, is then handed to git:

fsck --strict            clean
rev-parse HEAD           matches the commit we wrote
log                      our message and author
ls-tree -r HEAD          our mode, blob id and nested paths
cat-file blob            our bytes
ls-files --stage         the index we wrote, nested paths included
read-tree --reset -u     git checks the work tree out from our objects
status --porcelain       empty
remote get-url origin    the url our clone recorded

There is also a repository whose objects exist only inside a pack and an index this wrote. Git then resolves them: verify-pack accepts the index, cat-file returns the right type and bytes, fsck --strict is clean. Those answers have nowhere else to come from.

The pack reader was run against 3980 objects from rust-lang/log, 2222 of them deltas, every recomputed id matching git's verify-pack listing. Swapping one id made the test fail, so the pass is not vacuous.

Clone runs end to end on bytes github.com actually sent. Push runs against real git, driven through --advertise-refs and --stateless-rpc, which is what smart HTTP is a shell around. Chunked decoding is checked against a raw TLS payload captured with openssl s_client rather than through a client that would have decoded it first.

On a real depth-1 pack of this kernel, 33 MB and 18,264 objects: every id and offset matches git's own listing, object by object. A clone of it writes 17,456 files rather than 35,718, because the pack is stored rather than exploded. Peak memory is 157 MB, dominated by a single 33.8 MB object inside the pack rather than by how many objects there are.

95 tests. no_std builds, clippy -D warnings clean.

Bugs this found

Every one came from handing the result to git, or from asking what a hostile server could do, rather than from grading our own work.

Correctness:

The advertisement parser hardcoded the upload-pack banner, so every push failed at discovery. Push had only ever been driven through a hand-built request.
An empty repository answers with a capabilities^{} entry that names no object. It was read as a branch, so a first push to a fresh remote took garbage as the ref's current value.
Without an index, a clone leaves every file reading as staged-deleted and untracked at once.
Without .git/shallow, the parents a depth-1 fetch never sent are reported as broken links and fsck fails.
The root .gitignore excludes *.bin, so the recorded fixtures were never committed. CI on a clean checkout would have failed.

Security, from two hardening passes:

The pack trailer was never verified. PackError::Checksum existed and nothing could produce it, so a pack altered in transit was parsed anyway.
Inflate had no ceiling. Deflate expands about a thousand to one at its worst, so a few kilobytes could ask for gigabytes before anything was verified.
A delta's stated target size went straight into Vec::with_capacity, so a delta claiming four gigabytes allocated four gigabytes to produce nothing. Its varint also had no bound on the shift.
Header collection was unbounded, cheap to send and expensive to hold.
CRLF injection through a url. Host and path went into request headers unchecked, so a carriage return in the host appended headers of the sender's choosing, and one in the path split a request in two.
Path traversal through a url. The clone directory was whatever followed the last slash, so a remote ending in /.. wrote beside the working directory rather than inside it.
A port in a url was ignored rather than refused, which would connect somewhere the user did not ask for.
Two read loops counted only quiet reads, so a peer sending one byte at a time reset the counter forever.

And two in capsule_terminal, which was not clippy -D warnings clean and had not been, fourteen findings, none from this work:

JobEnv snapshots aliases and merge_back never restored them, so an alias defined inside a foreground job was silently lost.
The pull walk threaded pid, ip and args through every call by hand.

What is not done

This has never run on a booted image. Every layer is proven separately and the whole thing compiles for the real target, but the first live run is still ahead. That is why this is open rather than ready.

No credentials. A push to a private repository gets 401, and the failure says so.

No fetch or pull command. The machinery is built and tested in the library, but nothing in the terminal reaches it, so an existing clone cannot be updated.

Memory still scales with the largest object in a pack, since reconstructing one means holding it. That is inherent rather than a leak, but it is the ceiling on what will clone.

Notes for review

Reading order is sha1 and object, then zlib, tree and commit, then odb, refs, index and repo, then pack, wire and remote. The interop test is the one to read first if you only read one thing.

Parsers refuse what git would not have written. A tree entry named .. or holding a slash is rejected, and so is an index path that is absolute, because both become paths on checkout. An object read from a pack is framed and hashed before it is returned, so the index claiming an offset holds an id is checked rather than believed. A commit writes its object before moving the ref, so a failure leaves an unreferenced object rather than a branch naming one that is not there.

A response body is sized from its headers, never from how much arrived. A truncated pack reaching the pack reader would look like repository corruption rather than a network fault.

The TLS session refuses to send if the chain does not verify. Handing the payload to whoever answered is the failure that matters here.

The wallet's TLS copy was deliberately left alone. It is not drift, it is a different trust policy: it pins one anchor where the shared crate carries a root store. Widening it to remove a duplicate would be a security regression.

Also here: the keyboard third level. Scancode 0x56, the ISO key between left shift and Z, fell in a range the set 1 table returned None for, so < and > were unreachable on IT, DE, FR and ES, and USB had the same hole at usage 0x64. There was no AltGr level at all, so braces and brackets could not be typed. Right alt now carries its own modifier instead of doubling as alt. Nine tests across five layouts, each expectation taken from what the physical key prints.

Opened on GitHub by eKisNonos as pull request 435. Review and merge happen there while this repository is kept in step from GitHub; this copy follows it, and is marked merged or closed when it is.

Git in the NONOS terminal, from the object model up to `clone` and `push` over HTTPS. The protocol layers are checked against real git rather than against themselves, and the paths that read remote bytes have been through a hardening pass. What has not happened is a run on a booted image, so this stays open until it has. ## What is here **The object model**, `no_std`, written from scratch: - SHA-1, streaming and one shot - the `<type> <size>\0<content>` framing objects are named by - zlib: a full inflate over stored, fixed and dynamic Huffman blocks, so it reads objects git itself wrote, and a stored-block writer - trees, including the sort rule where a directory compares as though its name ended in a slash, which is what decides the hash - commits, with parents in order and the author and committer lines - the loose object database, `HEAD` and the branch files - the binary `DIRC` version 2 index - enough of the config file to record a remote and read it back **Packfiles, read, written, and stored as packs.** The reader resolves offset and reference deltas. The writer emits whole objects with a SHA-1 trailer. A fetched pack is kept whole with a version 2 index beside it, the way git does it, and the object store reads through that index: the first byte of an id picks a range out of the fanout, a binary search covers only that range, then the entry is inflated at its offset and its delta chain followed. **The wire protocol.** pkt-line framing both ways, the ref advertisement, the want, deepen and done body, and the receive-pack command list. **Clone and push**, over a `Transport` the crate does not implement. **Three crates the transport needed**, none of which existed: | Crate | What | |---|---| | `nonos_http` | HTTP/1.1 client framing, response parsing including chunked, and url checking | | `nonos_socket` | one client for `net.sockets`, a `TcpStream` that closes on drop | | `nonos_tls` | TLS 1.3, moved out of the browser, plus a blocking session | **In the terminal:** `git init`, `clone`, `add`, `status`, `commit -m`, `log`, `push`, `remote`. ## How it is checked Nothing is graded by its own assertions where real git could grade it instead. A repository built by this crate alone, with git never invoked to create any part of it, is then handed to git: fsck --strict clean rev-parse HEAD matches the commit we wrote log our message and author ls-tree -r HEAD our mode, blob id and nested paths cat-file blob our bytes ls-files --stage the index we wrote, nested paths included read-tree --reset -u git checks the work tree out from our objects status --porcelain empty remote get-url origin the url our clone recorded There is also a repository whose objects exist **only** inside a pack and an index this wrote. Git then resolves them: `verify-pack` accepts the index, `cat-file` returns the right type and bytes, `fsck --strict` is clean. Those answers have nowhere else to come from. The pack reader was run against 3980 objects from rust-lang/log, 2222 of them deltas, every recomputed id matching git's `verify-pack` listing. Swapping one id made the test fail, so the pass is not vacuous. Clone runs end to end on bytes github.com actually sent. Push runs against real git, driven through `--advertise-refs` and `--stateless-rpc`, which is what smart HTTP is a shell around. Chunked decoding is checked against a raw TLS payload captured with `openssl s_client` rather than through a client that would have decoded it first. **On a real depth-1 pack of this kernel, 33 MB and 18,264 objects:** every id and offset matches git's own listing, object by object. A clone of it writes 17,456 files rather than 35,718, because the pack is stored rather than exploded. Peak memory is 157 MB, dominated by a single 33.8 MB object inside the pack rather than by how many objects there are. 95 tests. `no_std` builds, `clippy -D warnings` clean. ## Bugs this found Every one came from handing the result to git, or from asking what a hostile server could do, rather than from grading our own work. Correctness: - The advertisement parser hardcoded the upload-pack banner, so **every push failed at discovery**. Push had only ever been driven through a hand-built request. - An empty repository answers with a `capabilities^{}` entry that names no object. It was read as a branch, so a first push to a fresh remote took garbage as the ref's current value. - Without an index, a clone leaves every file reading as staged-deleted and untracked at once. - Without `.git/shallow`, the parents a depth-1 fetch never sent are reported as broken links and `fsck` fails. - The root `.gitignore` excludes `*.bin`, so **the recorded fixtures were never committed**. CI on a clean checkout would have failed. Security, from two hardening passes: - **The pack trailer was never verified.** `PackError::Checksum` existed and nothing could produce it, so a pack altered in transit was parsed anyway. - **Inflate had no ceiling.** Deflate expands about a thousand to one at its worst, so a few kilobytes could ask for gigabytes before anything was verified. - **A delta's stated target size went straight into `Vec::with_capacity`**, so a delta claiming four gigabytes allocated four gigabytes to produce nothing. Its varint also had no bound on the shift. - **Header collection was unbounded**, cheap to send and expensive to hold. - **CRLF injection through a url.** Host and path went into request headers unchecked, so a carriage return in the host appended headers of the sender's choosing, and one in the path split a request in two. - **Path traversal through a url.** The clone directory was whatever followed the last slash, so a remote ending in `/..` wrote beside the working directory rather than inside it. - A port in a url was ignored rather than refused, which would connect somewhere the user did not ask for. - Two read loops counted only quiet reads, so a peer sending one byte at a time reset the counter forever. And two in `capsule_terminal`, which was not `clippy -D warnings` clean and had not been, fourteen findings, none from this work: - `JobEnv` snapshots aliases and `merge_back` never restored them, so an alias defined inside a foreground job was silently lost. - The pull walk threaded `pid`, `ip` and args through every call by hand. ## What is not done **This has never run on a booted image.** Every layer is proven separately and the whole thing compiles for the real target, but the first live run is still ahead. That is why this is open rather than ready. **No credentials.** A push to a private repository gets 401, and the failure says so. **No `fetch` or `pull` command.** The machinery is built and tested in the library, but nothing in the terminal reaches it, so an existing clone cannot be updated. **Memory still scales with the largest object in a pack**, since reconstructing one means holding it. That is inherent rather than a leak, but it is the ceiling on what will clone. ## Notes for review Reading order is `sha1` and `object`, then `zlib`, `tree` and `commit`, then `odb`, `refs`, `index` and `repo`, then `pack`, `wire` and `remote`. The interop test is the one to read first if you only read one thing. Parsers refuse what git would not have written. A tree entry named `..` or holding a slash is rejected, and so is an index path that is absolute, because both become paths on checkout. An object read from a pack is framed and hashed before it is returned, so the index claiming an offset holds an id is checked rather than believed. A commit writes its object before moving the ref, so a failure leaves an unreferenced object rather than a branch naming one that is not there. A response body is sized from its headers, never from how much arrived. A truncated pack reaching the pack reader would look like repository corruption rather than a network fault. The TLS session refuses to send if the chain does not verify. Handing the payload to whoever answered is the failure that matters here. The wallet's TLS copy was deliberately left alone. It is not drift, it is a different trust policy: it pins one anchor where the shared crate carries a root store. Widening it to remove a duplicate would be a security regression. Also here: the keyboard third level. Scancode `0x56`, the ISO key between left shift and Z, fell in a range the set 1 table returned `None` for, so `<` and `>` were unreachable on IT, DE, FR and ES, and USB had the same hole at usage `0x64`. There was no AltGr level at all, so braces and brackets could not be typed. Right alt now carries its own modifier instead of doubling as alt. Nine tests across five layouts, each expectation taken from what the physical key prints. --- Opened on GitHub by eKisNonos as [pull request 435](https://github.com/NON-OS/nonos-micro-kernel/pull/435). Review and merge happen there while this repository is kept in step from GitHub; this copy follows it, and is marked merged or closed when it is.

nonos-sync added 24 commits

2026-09-11 11:22:43 -04:00

git: object model, sha1 and object id for the terminal 0965d00447

The core the rest of git in the terminal builds on, from scratch and
no_std: SHA-1 over the streaming and one-shot paths, the
<type> <size>\0<content> framing git hashes and stores, and the 20-byte
object id with its hex and loose-object-path forms. It has to agree with
real git bit for bit, so the tests check against the exact hashes git
prints: blob 'hello\n' is ce013625, the empty blob is e69de29b, plus the
FIPS 180-4 sha1 vectors and the block-boundary sizes where padding goes
wrong if it is wrong at all. Nine tests, no_std build and clippy clean.

git: zlib for the object store, reads git's own objects 972b046820

Every git object is zlib-wrapped DEFLATE on disk, so the read path is a
full inflate: stored, fixed-Huffman and dynamic-Huffman blocks, and the
decisive test decompresses the exact bytes git wrote for the hello blob
back to blob 6\0hello\n. The write path uses stored blocks, a valid
git-readable stream with a small auditable writer, since a terminal
session's objects are small. Both ends carry the Adler-32 the format
checks, so a corrupt or truncated stream is refused, not returned short.

Modular: adler, bit_reader, block, compress, dynamic, error, huffman,
inflate and tables each in their own file, mod.rs re-exports only. Six
zlib tests plus the object core, clippy and no_std clean.

git: tree and commit objects, matching git's ids 0284543ffc

The two objects a commit is made of. A tree encodes each entry as the
octal mode, the name and the id as twenty raw bytes, sorted by git's
rule where a directory compares as though its name ended in a slash:
that rule decides the hash, so it is isolated in sort.rs and tested with
foo.txt against the directory foo. A commit encodes the tree, its
parents in order, author and committer, then the message, with ids in
hex rather than raw as git writes them.

Both are checked against a real repository: the tree reproduces
b4ed9182 and the commit reproduces 08b165d0, the ids git computed for
that content. Parsing refuses what git would not write, including a
tree entry named .. or holding a slash, which is what stops a hostile
tree escaping the work tree on checkout, and an out-of-order tree that
would re-encode to a different id than it parsed from.

git: object store, refs and repository operations ba738f9bd2

The layers that turn the object model into a working repository: a
loose object database under objects/, HEAD and the branch files, and
init, commit and log on top. Storage is a trait, so the same code runs
over the VFS capsule in the terminal and over a real directory in the
tests.

Reading an object hashes it again and checks it against the id it was
stored under, so a damaged database is an error rather than wrong bytes
handed to a caller that cannot tell. Ref names are validated before they
are joined into a path, which is what stops a branch called ../../evil
writing outside refs/heads. A commit writes its object before moving the
ref, so a failure leaves an unreferenced object rather than a branch
naming one that is not there.

The interop test builds a repository with this crate alone and then runs
real git over it: fsck --strict, rev-parse, log, ls-tree, cat-file and
read-tree all agree, and git reports a clean status against the work
tree it checked out from our objects. Thirty-five tests, clippy and
no_std clean.

git: staging index, and one unit per file across the crate 15b2dea4b8

The index is git's binary DIRC version 2, so add here is add git agrees
with: the interop test stages a top-level and a nested path, and git
ls-files reports both from the index we wrote. write_tree rebuilds the
nesting git stores, one tree object per directory, from the flat index.

The stat cache is written as zeros. Git reads that as cannot trust,
compare the content, which is correct without the shortcut, and the
trailing SHA-1 still covers the whole file so an edited index is
refused. Index paths are checked the way tree names are, since both
become paths on checkout.

Also splits every file in the crate to one unit and at most 75 lines,
with mod.rs holding declarations and re-exports only, and cuts the
comments back to the decisions worth explaining. Thirty-eight tests,
clippy and no_std clean.

terminal: git init, add, status, commit and log a7c003a474

Storage over the VFS client, so the repository code that the host tests
prove is the code the shell runs. Commits are recorded under a fixed
nonos identity, since a session has no per-user git config yet.

terminal: keep git paths relative to the work tree 65b5cdf6f6

Joining a work tree of / onto an absolute argument produced a doubled
separator, and the index refuses an absolute path on read back. add now
trims the work tree prefix before staging.

git: pack reader with delta resolution ac4b414547

The format a fetch or clone sends objects in, and the first half of
clone. Reads the header, inflates each entry and resolves both delta
kinds against objects already seen: a pack lists a base before the
deltas naming it, so one forward pass needs no recursion.

Checked against packs GitHub served. A shallow clone's pack is vendored
and its three ids are asserted. The delta path runs over a full clone of
rust-lang/log: 3980 objects, 2222 of them deltas, and every id we
recompute from the bytes we rebuilt matches git's own listing. Swapping
one id in that listing fails the test, so the comparison is real.

zlib gained decompress_prefix: pack objects are back-to-back streams
with no length in front, so reading the next one needs the byte count
the last consumed.

git: pkt-line, ref advertisement and the fetch request 95b5cf515f

The wire half of clone. pkt-line framing both ways, the ref
advertisement a fetch opens with, and the want/deepen/done body that
asks for a shallow history.

Checked against GitHub rather than a fixture I invented: the
advertisement test reads a real /info/refs response, and the request
this builds is the one GitHub answers with 200 and a pack, which the
pack reader then decodes to the three expected ids.

git: clone a fetched pack into a working repository 0a2005c8ba

Unpacks the pack into the object store, points the branch at the head
commit, writes the work tree out and builds the index from the tree it
just wrote.

Two things a naive version gets wrong, both found by handing the result
to git. Without the index every file reads as staged-deleted and
untracked at the same time. Without .git/shallow, fsck calls the
parents a depth-1 fetch never sent broken links and fails. The clone
test now runs fsck --strict and status against the real octocat pack
and both come back clean.

git: build and send a push 801f67a450

Pack writer, the object walk that decides what a push carries, and the
receive-pack command framing.

Proven by piping the body into a real git receive-pack: it answers
unpack ok, moves the ref, and fsck on the receiving repo is clean. Two
rejection tests keep that honest, one flipping a bit inside the pack
and one replaying a push whose old id is out of date. Both are refused,
so the accept is the protocol working rather than git ignoring what we
sent.

Objects go in whole rather than as deltas. Smaller packs are an
optimisation, not a requirement, and a receiver reconstructs the same
objects either way.

keymap: altgr level and the iso key abc645a85e

Two things were missing and between them they made the non-US layouts
unusable for writing code.

Scancode 0x56, the key an ISO board has between the left shift and Z,
fell in a range the set 1 table returned None for, so the press was
dropped. On IT, DE, FR and ES that key is where < and > live. USB had
the same hole at usage 0x64.

There was no third level at all, so anything behind AltGr could not be
typed: braces and brackets on IT, DE, FR and ES, @ and # on IT and ES,
the whole bracket set on the French number row. Right alt now sets its
own modifier bit instead of doubling as alt, which is what a European
board means by that key, and AltGr only wins where the layout has
something on that level so it falls through everywhere else.

Nine tests over the five layouts, each expectation taken from what the
physical key prints.

git: drive clone and push over a transport seam 66229a0a76

Adds the seam the protocol was missing. Transport is get and post over a
repository URL, so the fetch, clone and push flows are complete and
testable without a socket in the crate, and TLS plugs in behind it.

Clone runs against bytes github.com actually sent: the service banner,
the HEAD and master packets from its advertisement, and the answer to
our depth-1 want request, shallow line and all. It asks for discovery
then the pack, in that order and nothing else, and git fsck --strict
and status are clean on the result.

Push runs against real git. Smart HTTP is a thin shell over
--advertise-refs and --stateless-rpc, so the test drives those directly
and the far end is genuine git rather than a model of it. Pushing what
the remote already has sends nothing rather than an empty pack it would
refuse.

Two bugs this found. The advertisement parser hardcoded the upload-pack
banner, so every receive-pack advertisement was rejected as not smart
HTTP. An empty repository answers with a capabilities placeholder under
a ref name that names no object, and it was being read as a branch.

Also force-adds the test fixtures. The root gitignore excludes *.bin,
so the recorded responses were never committed and the wire tests only
passed because the files happened to be in the working tree.

http: client framing and response parsing 05441bc130

HTTP/1.1 for a client, no_std, with no I/O of its own. Requests build
into bytes and responses parse from bytes, so the same code runs over
TLS in the shell and over a buffer in a test.

Chunked decoding is checked against the raw TLS payload github.com
returns for a ref advertisement, captured off the wire rather than
through a client that would already have decoded it, so the chunk
framing under test is the framing GitHub sent.

The body length comes from the headers, never from how much arrived. A
response that declares more than it delivers is an error, because
handing back what turned up would pass a truncated pack to a caller
with no way to tell. Seven damage cases cover the rest: a lying chunk
size, a size that is not hex, an unterminated body, a status line that
is not one, and a header without a colon.

An empty POST still states Content-Length: 0, or the server waits for a
body that is never coming.

socket: one client for net.sockets aa39a01884

Three capsules each carried their own copy of this: the browser, the
wallet and nym. Same magic, same opcodes, three places to fix when the
protocol moves.

A TcpStream owns its handle and closes on drop, which is the part the
copies got wrong in different ways. A failed connect leaves a handle
allocated in the capsule, and an early return on a handshake error
leaves a connection nobody will read or close.

Replies are checked before they are believed: short frames, a wrong
magic and a non-zero status are all refused, and a receive trusts the
smallest of what the header claims, what arrived and what the caller
has room for. Errors are a type rather than the unit, so a caller can
tell no network at all from a refused connection.

tls: one client, and a blocking session on top 9b9a70f717

Moves the browser's TLS 1.3 out to a crate and adds a session that runs
the handshake in order for a caller that can wait, rather than as the
phase machine a UI needs. The browser builds against it unchanged; the
alias keeps its call sites reading the same.

The session refuses to send if the chain does not verify. Handing the
payload to whoever answered is the failure that matters here, so a
certificate error stops the request rather than being reported after
it. Server keys are derived once from the verified handshake, so
records decrypt without walking the chain again each time. The flight
and the response are both bounded.

Not merged: the wallet's copy. It is not drift, it is a different trust
policy, pinning one anchor where this carries a root store. Widening
the wallet to a general store to remove a duplicate would be a
security regression, so it keeps its own until the pinning is expressed
as policy over this rather than a fork of it.

Two pre-existing clippy findings fixed on the way in, both zero-fill
through resize. Eleven files in here are over the size limit, mostly
root tables and the AES core; they came across as they were and are
worth splitting separately from a move.

git: clone and push from the terminal, over https 6b96e8317d

Wires the stack that was missing an end. A Remote parses an https url,
Https implements the git transport over the socket client, the TLS
session and the http framing, and git clone and git push use it.

Only https is accepted. Git over plain http would let anyone on the
path serve the objects, and the terminal has no way to tell the user
that happened. A status other than 200 is an error rather than a body,
because a server error page is valid http and would otherwise reach
the pack reader as if it were a pack.

Clone stops at the tip. A whole history is large and there is no
progress display to sit behind one yet, so the depth is stated rather
than left to run.

Failures are reported as something to act on: unreachable host, closed
connection, private repository, credentials this cannot send yet.

Along the way, capsule_terminal was not clippy clean and had not been:
fourteen findings, none of them mine. Two were real. JobEnv snapshots
aliases and merge_back never restored them, so an alias defined inside
a foreground job was silently lost on return. The pull walk threaded
pid, ip and args through every call by hand; they never change during a
pull, so they travel as one context now.

git: record the remote a clone came from eb5923656e

A clone wrote no config beyond core, so a push afterwards had to be
told the url again, every time. It now writes an origin section, and
push falls back to it when no url is given.

The config file is rebuilt rather than appended to when a remote is
set, so setting one twice leaves a single section rather than two that
disagree about where origin points. Parsing keeps to sections and name
= value lines and skips what it does not model: git writes settings
this has no opinion on, and refusing a whole file over one of them
would make a repository unreadable for nothing.

Checked by handing it to git: remote get-url and config --get both
read back what we wrote, on a repository git had no part in creating.

git remote shows the origin, and sets it.

git: reword a comment the hygiene gate bans 5c778e9f6e

The word placeholder in a comment is a banned pattern. The line was
describing what git sends for an empty repository, so it says that
instead.

git: write a pack index aff550e11d

First half of keeping packs as packs. Exploding a fetched pack into
loose objects is what makes a real repository unclonable: a depth-1
clone of this kernel is 18264 objects, so 18264 files written one at a
time, on top of holding every decompressed object in memory at once.

This writes the version 2 index git has used since 1.6: the fanout that
turns the first byte of an id into a search range, the sorted ids,
their CRCs and their offsets. A pack whose offsets run past two
gigabytes is refused rather than written with a table this cannot read
back.

Not yet wired into the object store, and no reader. On its own it
changes nothing.

git: keep a fetched pack as a pack 5b6f147729

A clone used to explode the pack into loose objects. For this kernel
that is 18264 separate writes on top of the 17454 the work tree needs,
and it kept every decompressed object in memory to do it. That is what
made a real repository unclonable, not the protocol.

The pack is now stored whole with an index beside it, the way git does
it, and the object store reads through that index: first byte picks a
range out of the fanout, binary search inside it, then the entry is
inflated at its offset and its delta chain followed. Reading one object
no longer costs reading the pack.

The proof is a repository whose objects exist nowhere but inside a pack
and an index this wrote. git verify-pack accepts it, cat-file returns
the right bytes and type, and fsck --strict is clean. There is nowhere
else those answers could have come from.

An object read out of a pack is framed and hashed before it is
returned, so the index saying an offset holds an id is checked rather
than believed. Delta chains are bounded; a pack claiming a longer one
is damaged or hostile and following it would exhaust the stack.

Writes for a clone of this kernel go from 35718 to 17456. Memory during
the read is still the whole pack plus the objects it resolves, which is
the next piece and is not done.

git: index a pack without holding what it inflates b3c5911e6b

Storing a pack still cost the whole decompressed tree in memory,
because the index was built from a full read of it. It is built from
the pack directly now, in two passes: the first finds where entries
start and keeps nothing it inflates, the second names each object by
rebuilding it at its offset and dropping it again.

Resolving a delta used to recurse, which held every intermediate object
alive at once. The chain is walked to its base first, reading only
entry headers, then the deltas are applied forward, so only the running
content and one delta exist at a time.

Measured on a depth-1 pack of this kernel, 33 MB and 18264 objects:
peak went from 187 MB to 157 MB, and what is left is dominated by a
single 33.8 MB object inside it rather than by how many objects there
are. Holding the pack plus two copies of its largest object is what
this costs; there is no version that does not.

Every id and offset matches git's own verify-pack listing for that
pack, checked object by object.

The terminal's response cap went to 64 MB. At 16 MB it refused a real
repository before reading an object, and the memory that would follow
is now known rather than guessed at.

git: harden every path that reads remote bytes 3a4e2bba58

An audit of what is trusted between the socket and the object store
turned up six ways a hostile or damaged response could be made to hurt.

The pack trailer was never verified. PackError::Checksum existed and
nothing could produce it, so a pack altered in transit was parsed
anyway and the damage surfaced later as a mismatched id, or did not
surface at all. It is checked before anything else is read now, which
is also why two tests changed: a truncated pack fails on its checksum
rather than deeper in, and a test that alters a header has to reseal
the pack for the check it means to exercise to be the one that fires.

Inflate had no ceiling. Deflate expands about a thousand to one at its
worst, so a few kilobytes could ask for gigabytes before anything had
been verified. Both the whole-stream and the prefix decoder stop at 256
MB, well above the 34 MB largest object in a clone of this kernel.

A delta stated its own target size and that number went straight into
Vec::with_capacity, so a delta claiming four gigabytes allocated four
gigabytes to produce nothing. The size is bounded first and never used
to reserve. Its varint also had no bound on the shift, so an eleventh
continuation byte shifted past the width of the word.

HTTP responses collected headers without limit. Small to send, large to
hold. Capped at 128.

Nine tests, each built to make the reader allocate rather than to be
read: a reserved block type, a stream cut mid block, a wrong Adler-32,
a delta asserting a target of nearly 2^70, a delta with no base, a
flipped bit, an object swapped and resealed, and five hundred headers.

http: check a url before it reaches a request or a path f3ff8e4215

Two ways a url could do damage, both from the part a user types.

The host and path went into request headers unchecked. A url with a
carriage return in the host ended the Host line and let whatever
followed be read as headers of its own; the same in the path split one
request into two. Hosts are now letters, digits, dots and hyphens, and
paths are printable ASCII without space, which covers every git forge
url and excludes everything that could end a line.

A port is refused rather than ignored. Callers connect on 443, and
taking a port and disregarding it would connect somewhere the user did
not ask for.

The last path segment becomes the directory a clone creates, and it was
whatever came after the final slash. A remote ending in /.. had a clone
write beside the working directory instead of inside it. Empty, dot and
dot dot are refused.

The parser moved to nonos_http, where it belongs and where tests
actually run: capsule_terminal has host tests that CI never invokes and
that no longer link. Six tests cover the refusals.

Also bounded two read loops. Counting only quiet reads is not enough on
its own, because a peer sending one byte at a time resets that counter
forever and the exchange never ends.

This pull request has changes conflicting with the target branch.

userland/capsule_browser/Cargo.lock
userland/capsule_terminal/Cargo.lock
userland/capsule_terminal/Cargo.toml
userland/capsule_terminal/src/command/builtin/fs/ls.rs
userland/capsule_terminal/src/command/builtin/git/clone/run.rs
userland/capsule_terminal/src/command/builtin/git/commit.rs
userland/capsule_terminal/src/command/builtin/git/dispatch.rs
userland/capsule_terminal/src/command/builtin/git/mod.rs
userland/capsule_terminal/src/command/builtin/git/push/run.rs
userland/capsule_terminal/src/command/builtin/nox/dispatch.rs
userland/capsule_terminal/src/command/builtin/nox/pull/walk.rs
userland/capsule_terminal/src/git/mod.rs
userland/capsule_terminal/src/git/transport/https.rs
userland/capsule_terminal/src/git/transport/mod.rs
userland/capsule_terminal/src/git/transport/round_trip.rs
userland/capsule_terminal/src/term/grid/mod.rs
userland/nonos_git/Cargo.toml
userland/nonos_git/src/lib.rs
userland/nonos_git/src/odb/mod.rs
userland/nonos_git/src/odb/read.rs
userland/nonos_git/src/pack/delta/apply.rs
userland/nonos_git/src/pack/delta/size.rs
userland/nonos_git/src/pack/mod.rs
userland/nonos_git/src/pack/reader/mod.rs
userland/nonos_git/src/pack/reader/read.rs
userland/nonos_git/src/remote/clone.rs
userland/nonos_git/src/repo/clone/request.rs
userland/nonos_git/src/repo/clone/store.rs
userland/nonos_git/src/repo/clone/take.rs
userland/nonos_git/src/wire/advert/parse.rs
userland/nonos_git/src/zlib/error.rs
userland/nonos_git/src/zlib/inflate.rs
userland/nonos_git/src/zlib/mod.rs
userland/nonos_git/src/zlib/prefix.rs
userland/nonos_git/tests/clone.rs
userland/nonos_git/tests/pack.rs
userland/nonos_git/tests/remote_clone.rs
userland/nonos_http/Cargo.toml
userland/nonos_http/src/lib.rs
userland/nonos_http/src/response/headers.rs
userland/nonos_http/tests/damage.rs
userland/nonos_socket/Cargo.toml
userland/nonos_tls/Cargo.lock
userland/nonos_tls/Cargo.toml
userland/nonos_tls/src/chain_walk.rs
userland/nonos_tls/src/lib.rs
userland/nonos_tls/src/roots/lookup.rs
userland/nonos_tls/src/roots/mod.rs
userland/nonos_tls/src/server_complete.rs
userland/nonos_tls/src/server_finished_flight_ready.rs
userland/nonos_tls/src/session/flight.rs
userland/nonos_tls/src/session/response.rs

View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Merge commit title

Merge pull request 'git: object model, packfiles, clone and push over https' (#16) from nonos-sync/gh-435 into main

Merge commit body

Reviewed-on: https://git.nonos.software/NON-OS/nonos-micro-kernel/pulls/16

Merge pull request 'git: object model, packfiles, clone and push over https' (#16) from nonos-sync/gh-435 into main

Reviewed-on: https://git.nonos.software/NON-OS/nonos-micro-kernel/pulls/16

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin +refs/pull/16/head:nonos-sync/gh-435

git switch nonos-sync/gh-435

No reviewers

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

NON-OS/nonos-micro-kernel!16

No description provided.

Rows
Columns