CivArchive
    ← All articles
    Published September 26, 2026by AbstractPhila

    2 years of AI research and the compounded results.

    152 views7 reactions0 comments on CivitAI1 collected
    geometryresearchmodel traininglora trainingdistillationresearch historytokenizationML Research

    Mapped by Failure: The Long Way From Crystals to Arms

    AbstractPhil · September 2026 · written with Claude Opus 5.5 (Anthropic) · a look back across the whole program, from the first geometric crystals to the current model

    The first post in this series, back in August 2025, didn't have a single number in it. It was a pitch. Build models out of geometric "crystals" (tiny five-point shapes), keep a stable core that you protect, and bolt small disposable blocks onto that core that can come and go without hurting it. Treat the geometry as a gauge you read, not a router that picks things.

    Reading it again a year later is a strange experience, because most of it turned out to be roughly right, and almost none of it could be built the way it was first imagined. The protected core exists. The detachable blocks exist; we call them arms now, and switching one off returns the model to itself exactly, down to the last bit. The geometry really is something you read rather than something that routes. But every one of those pieces had to be earned the hard way, and the way it got earned was mostly by breaking things.

    This isn't another installment; those are linked at the bottom, with the tables, seeds and charts. This is the version of the story that only makes sense when you read the whole record end to end: why each belief gave way to the next, and what the failure in between taught us. It's informal and nearly number-free on purpose. The numbers are in the companions; what's here is the reasoning that connects them.

    If one idea runs through the whole thing, it's this. Mathematics is generous; optimization isn't. A structure can be perfectly well defined, provably expressive, beautiful on a whiteboard, and still be something gradient descent will never grow for you in any reasonable amount of time. The program has spent its whole life, across byte-level language models, image encoders, diffusion U-Nets and transformers, adapters on frozen trunks and plain classifiers, asking one question in different costumes: is this structure actually computable by training, or does it only look computable on paper? You can't answer that by thinking. You answer it by contact. Every failure below is a spot where we walked into the edge of what training can do and marked it.

    The five sections follow the order in which that question got asked:

    1. The composite numerics of computable numbers as determined by failure: where the program's numbers came from, and how failures sorted the real ones from the ones we'd accidentally built.

    2. Structural utilities of those computable numerics in defiance of expectation: what those numbers turned out to be good for, usually in ways the standard playbook says shouldn't work.

    3. Purposeful projective capacity applied by those numerics in deterministic outcomes: carrying that structure into other models and other spaces, with moves whose outcome is known before the run starts.

    4. Structural determination through systemic accumulation and experimentation: how a model's actual architecture got decided by piling up hundreds of experiments.

    5. Synthesis of experimentation into a usable structural assessment for an experimental history: what all of it adds up to, in a form you can actually use.

    A note on voice. "We" is the collaboration. The program is Phil's: its questions, its standards, its calls about what to chase and what to drop. The record was kept together with a series of Claude models, working as the engineering side. Nothing here is a scorecard of who caught what. Most of the good catches happened in the conversation between us, and so did most of the mistakes.

    1. The composite numerics of computable numbers as determined by failure

    Every geometric research program ends up with numbers it believes in. You measure something, it comes out the same across a dozen models, and before long it has a name and you're steering toward it. The trouble is telling a number you discovered from a number you manufactured. For most of the first year we couldn't tell. Then a run of failures did the sorting for us, and the result is the backbone of everything that came after.

    The unit, and the first things that broke

    The working unit from the very first month was the pentachoron: five points, the simplest solid that lives in four dimensions. There's an old formula (the Cayley–Menger determinant) that gives you its volume from nothing but the distances between its corners, and tells you whether the shape is even valid. We used it from day one as both a sanity check and a training signal. The spread of those volumes across many shapes (are the simplices all roughly the same size, or wildly uneven?) became the house health number.

    The early failures all had the same shape, and it's worth naming it up front: the geometry worked when it was frozen or constructed, and died when it had to compete, or survive a pipeline built for something else.

    • A vocabulary built from an old dictionary quietly turned every modern word into a garbage crystal. Nothing crashed; the debug output just looked wrong. The fix was to throw out the dictionary and hash crystals straight from Unicode. Same symbol in, same crystal out, every time, no training involved. That was the program's first object that was computable by construction, and it set a taste for determinism that never went away.

    • Geometry injected into a standard vision transformer got eroded by the attention layers above it. Ripping the geometry out entirely made the model worse, so it clearly mattered, but the single-stream design hit a hard ceiling it couldn't break. The answer the program built was to give the geometry its own stream, a separate pathway running beside the visual one. That model was called Beatrix. Hold onto the name.

    • Global average pooling (squashing a grid of patch features into one average vector, which everybody does) wiped out most of what a geometric encoder knew. It became the first hard rule in the book: never pool geometry globally.

    • A fractal positional code built on the Cantor staircase collapsed onto exactly three values because the middle digit of each step got no credit. Giving it half credit fixed it. Months later we worked out why: half is the only value that makes the staircase match its own expected measure. A number found by a bug, and then proven by arithmetic.

    • Towers of different geometric flavors feeding one model ended up with gradients so far apart that one tower was effectively dead. Equalizing the gradients fixed it, and a healthy hierarchy still emerged on its own. Equal gradients don't mean equal outputs.

    • Crystals trained directly by gradient descent shrank to nothing. The same crystals held frozen kept their shape. The principle got stated outright: you navigate this geometry, you don't optimize it. (Later work narrowed it: some anchors do better partly trained, and attention routing turned out to need parameters of its own.)

    Look at how every one of those got fixed: a separate stream, half credit, gradient equalization, frozen anchors, a hash. All structural. Not one was fixed by adding a loss term. That pattern is the first thread of the whole trajectory, and it's going to come back at every scale.

    The numbers were already there

    By early 2026 the volume-spread number kept turning up. We measured it across a pile of well-known pretrained models (text encoders, image encoders, autoencoders, our own systems) and they kept landing in the same narrow neighborhood. It looked like a universal constant of learned representations, and we wrote it up as one.

    Then came the control that should have come first: random points. Take plain random noise, put it on a sixteen-dimensional sphere, measure the spread. It lands in the same band, every time, at every sample size, with no model and no data at all. The "universal constant" was what empty space already looks like at that dimension. Every model that seemed to be converging on it was really converging on an effective geometry about sixteen dimensions wide, whatever width it nominally had.

    That opened the floodgates. The spread turned out to be a function of a matrix's shape alone, meaning how many rows and how many dimensions, and you can tabulate it from random matrices without training anything. Then the demonstration that made it stick: a model deliberately pushed toward the wrong target value fought it, blew up, collapsed, and came back to the value the random-matrix table predicted, a value it had never been shown. It behaved like a system returning to thermal equilibrium, which is the metaphor the program used at the time. (The blow-up itself was later traced to an optimizer fault, covered in the next section; the return to the predicted value stands.) Then three latent sizes gave three bands, each one sitting right on the random-sphere value for its size. Plain reconstruction got there, and so did a fully linear encoder. Normalizing rows onto the sphere didn't create the attractor. It selected which one you were sitting in.

    The binding constant has the strangest story. It first showed up as the angle that anchors drifted toward in a spherical diffusion prototype, with more of them arriving there in each of four versions. The spring attractor work then identified it with the volume spread at twenty-four dimensions, the spot where the geometric attractor meets the phase boundary for these shapes. And it showed up a third time inside a model trained on nothing but pure noise. Text gave a different value, so it isn't universal across kinds of data. The program chose to treat it as structure and turned it into a judging rule (measure an anchor system by its drift toward the binding value, never by how well it reconstructs), and that rule is still on the books. Later, watching codebooks train under next-byte prediction, we saw it was a waypoint they pass through on the way somewhere else. It's still useful, just with a different meaning than the one it had when we named it.

    The omega tokens belong here too. That was an autoencoder whose entire code is the singular values of sphere-normalized patches. It reconstructed photos almost perfectly at heavy compression, locked its geometry within the first few epochs, and found the same attractor whether it was fed photos, noise or text. Some of that universality came with the design, though: normalizing the rows pins the total spectral energy by construction, so the attractor lives inside a box we built ourselves.

    The loss that never pushed

    Then came the code review, and the discovery everything else hangs on. For months, a volume-regularity loss had been credited with shaping the geometry. It turned out to have been disconnected from the gradient the entire time. One call that turned a tensor into a plain number had quietly cut it out of the graph. The optimizer never felt it. Every bit of geometry it had been credited with had grown on its own.

    We published that, embarrassing as it was, because it's the most informative thing that happened all year. It became a law: geometric statistics are readouts, never forces. You watch them. You don't push on them. (There's been exactly one narrow, logged exception since.) And it reframed everything before it. If the numbers are properties of the space rather than achievements of the model, then all the effort spent forcing them was effort spent fighting the one thing that was never going to lose.

    Precision is a number too

    There's a less glamorous branch of the same story. Single precision loses its grip exactly when a simplex goes nearly flat, which is exactly when you most need its volume, and it falls apart on lopsided spectra. The popular bf16 format quantizes angles on the sphere too coarsely to be usable at the noise floor. So every gauge moved to double precision, and to afford that inside a training loop the program built its own kernels: fused singular-value decompositions far faster than the vendor library on thin matrices, and an eigensolver built on nineteenth-century mathematics that came out more accurate than the vendor's on small matrices. It's plumbing, but it's the plumbing that decides whether a number exists at all.

    How the numbers compose

    The numbers that survived have a habit of composing into each other. The sixteen of the measurement sphere is four squared: the address lives in a small four-dimensional home and gets lifted onto the sixteen-dimensional sphere where volumes are measured. Three sampling phases along a curve give the "rule of three," and the full address size is a product of those pieces. It reads like numerology, but it's architecture: the sixteen is the sphere the failures above kept pointing at, and the rest is built on top of it.

    The loss functions turned out to be composites too. When the program finally took a census of every objective it had ever trained against, across every line of work, it found only three primitives: cross-entropy, squared error and KL divergence. Everything that looked distinctive (contrastive losses, banded losses, weighted blob losses) was one of those three, summed in a different way. Three primitives wide, eleven accumulation formats deep. Most of the program's "new loss" discoveries were really discoveries about how to add things up.

    And one lesson migrated in a way that sums up this whole section. Back in autumn 2025, cross-entropy looked like the thing destroying geometry, so a pure geometric loss set replaced it, at a cost we accepted. Nine months later we dug that old loss set out verbatim and reran it under real controls. It lost badly; its old showing had been flattered by the setup it was first tried on. But the version that kept only its absolute terms (pull each example toward its own class) and dropped the comparative ones (push away from rivals, beat the runner-up) more than doubled the full set. The enemy was never cross-entropy. It was comparison.

    So by midsummer the program's numbers had been sorted into two piles: the ones we'd pushed and the ones we'd found. The found ones were all computable before training: pure functions of dimension, shape and arithmetic. The pushed ones mostly weren't real. The failures did the sorting. Which left the obvious next question: if the numbers are already sitting in the space, what are they for?

    2. Structural utilities of those computable numerics in defiance of expectation

    Once you know a number belongs to the space rather than to the model, the obvious move is to stop pushing on it and start building with it. That sounds tidy written down. In practice almost every use we found for these numbers went against what the standard playbook says should happen. This section is the list of those surprises, in roughly the order they arrived.

    One line instead of a loss

    The first big one was almost insultingly small. The spectral autoencoders kept suffering what we called a SNAP: the largest and smallest singular values blowing up together, the geometry lurching, reconstruction falling apart. The diagnosis was Adam wrestling an ill-conditioned problem. The fix was a single line: normalize every row onto the unit sphere. That pins the total spectral energy by construction, so the blow-up simply can't happen anymore. Collapses went to zero across hundreds of epochs, and runs that used to crawl and crash ran fast and clean. As a bonus it pulled the problem apart into directions, alignment and magnitudes, so controlling the geometry stopped fighting the reconstruction.

    That became a doctrine: build the geometry in; don't penalize toward it. Pick an architecture whose natural geometry already matches the attractor and let the optimizer find its way there, instead of engineering a loss to drag it.

    When we did keep a geometric objective, the one that worked was the reverse of the usual penalty. We called it the soft hand. Instead of punishing the model for being far from healthy geometry, it rewards the model for being near it by boosting the reconstruction gradient in that neighborhood. Adam's momentum then piles up in the geometrically healthy directions, and the optimizer itself becomes the stabilizer. It produced the best run of its sweep. It also produced the more valuable finding: reward the model for staying in the zone for too long and it learns to camp there instead of learning the task. Pressure toward a zone is fine. Treating residence in the zone as the goal corrupts.

    A smaller utility came out of the same stretch: the band a run will end up in is readable very early, so runs can be screened in minutes. The old habit of keeping whichever configuration had the lowest reconstruction error was retracted, because the lowest-error candidate turned out to be the wrong one.

    Reading instead of training

    The next surprise was that a lot of the useful structure was already there, waiting to be read out.

    Take any healthy sphere-solving model, pair up its rows that point in opposite directions (a plain deterministic merge, no clustering involved), and what falls out is a near-uniform codebook on projective space, the space where a direction and its opposite count as the same axis. Nobody trained for that. It was found by reading the weights. Our first guess that the same pattern held at every dimension got walked back when a bigger sweep came in: the natural number of axes grows with dimension, so the codebook size has to grow to match. The core finding stood.

    Then came voids: the empty cavities in a codebook's shape, measured with topology tools. It turned out that what a codebook leaves empty fingerprints what it learned. Symbolic codebooks are riddled with voids, and smooth continuous ones hardly have any, even when simpler measurements can't tell them apart.

    And a sweep over dozens of trained banks, each read at three points in training, found that the similarity structure between them barely moved from the first epoch to the last. The basin was set at init. Training neither pulled the banks together nor pushed them apart. Later work scoped that to reconstruction banks: codebooks feeding a next-byte prediction head move a lot more. But it made initialization a first-class design decision, and it's why the program later started some codebooks from a deterministic super-Fibonacci layout that begins inside the basin. That layout bought steadier cultivation and no better score, and it was recorded exactly that way.

    Codes for free, and a read that never picks

    Under reconstruction pressure (squared error plus a light alignment term, "match this target" rather than "beat your rivals") the codebooks converged all by themselves to sign codes: every row sitting at exactly plus or minus a reference direction, to floating-point precision. That's the kind of discrete structure most of the literature builds special machinery to get: vector quantizers, commitment losses, moving-average codebook updates. None of that was there. The lesson got a short name that stuck: read the signs, not the probabilities.

    The read that makes this work is the aleph (named for the set-theory symbol for graded infinities). Here is the only formula in this article:

     m̂ = Σ_k sinh(u_k) · A_k  /  Σ_k cosh(u_k)        where u_k = cos(x, A_k) / τ

    In words: compare the input to every anchor direction. Each anchor then contributes with a sign: the hyperbolic sine is odd, so pointing away counts as much as pointing toward, just negatively. The denominator is a sum of hyperbolic cosines, which are always positive, so it can never vanish or flip. There's no argmax, no ranking and no winner anywhere in it. It's a softmax over both ends of every axis at once, collapsed into a closed form so it never has to build the doubled table. The codebook gets its training signal only through reconstruction flowing back through that read, and it stays alive, with nearly every axis in use, without any balancing term.

    Its mirror image became the program's oldest named failure class: anything that compares alternatives and picks one. That covers argmax routing, softmax votes over experts, straight-through choices among rivals, k-means alphabets and vector quantization. Every one of them homogenizes its own gradient, because the losers stop learning and the winners stop being different. The measurement that nailed it down: gradient-learned alphabets collapsed onto a small fraction of the distinct paths that fitted, frozen alphabets kept. From then on, differentiation had to be structural.

    Placement decides

    The aleph read did nothing at all when we bolted it onto a classifier. We tried it on bed after bed and it was inert every time. On one image bed the address that was supposedly steering a bank of filters turned out to be provably constant; it was decoration.

    Then the question got turned around: what if the address is the operation? We rebuilt a small vision network so that the convolution itself was the aleph read. The layers had no ReLU and no GELU anywhere, and the read was the only learned nonlinearity in the network. (Honest footnote, found later by checking the parameter count: the downsampling in the headline run was max-pooling, which is a nonlinearity, just not a learned one. The stronger claim waits on a run without it.) It trained well. And it rediscovered something on its own: on natural photos, at least in the shallower versions, the magnitude of the read beat the signed read, because an edge and its inverted twin are the same edge. That's the modulus trick from scattering networks, falling out of an antipodal read nobody designed for it. The program had actually seen a version of this months earlier, when a simplex operator that was mediocre as a classifier turned strong once it was used as the attention. Same lesson, clearer the second time: placement decides whether a mechanism carries load.

    The same thing happened under next-byte prediction, where it mattered most. We expected forcing all of a model's predictive information through the address to cost something. It helped instead, in every single paired comparison, and the advantage grew with training (at least when the head and the model trained together; on a frozen encoder a plain head won). The layout that prediction pressure found on its own was the same "many small low-dimensional slots" shape the geometric work had arrived at by hand. The fully discrete sign-code version predicted as well as the soft read at the shorter budget, and with a longer budget the soft read pulled slightly ahead. (Its straight-through step reconstructs a sign rather than choosing a winner among rivals, which is exactly where the program draws the line.) And the sign code turned out to be a free similarity hash: distances between codes track the true angle between inputs without bias, precisely because the codebook is projectively uniform. One more structural thing, sitting there for free.

    Even the failure that kicked off that campaign followed the same arc. One hard-temperature head collapsed onto two winning axes. The diagnosis written that day blamed the geometry. Two weeks later a loss campaign swapped out only the loss, keeping everything else identical, and the collapsed address decompressed step by step as the loss's coupling was removed. The collapse had followed the loss all along. In open field, cross-entropy still beat everything we built to replace it, and we published that plainly. Where its coupling is the disease, decoupling is the cure.

    Amplitude, not choice

    The result that surprised us most came from a text-encoder trunk built with anchored routing. The preregistered hope was that learned addressing would beat a dense model at exactly matched capacity. It didn't. Learned anchors, frozen random anchors and plain dense layers all landed in the same place. And the frozen random routing was the fastest of the three to reach capability. The dispatch still did real work (switch it off and the model got worse) but which expert got picked didn't matter. Amplitude, not choice. A census of the routing signs showed that in the small-roster trunks subtraction was the majority mode: the model was busy inhibiting experts, and those negative weights got set early in training while their amplitude grew later. The frame gets decided early and the function arrives late, which later shaped how we schedule and gauge everything.

    Around it came a cluster of laws about the same mechanism, each one pointing the same way:

    • Sharper routing is worse. Pushing the router toward decisive choices lowered the score even as the routing got more "separated." A decisive router isn't a healthy one.

    • The budget is conserved. The read splits a fixed amount of amplitude among anchors, so an off-duty anchor should abstain (sit near zero), never oppose. Perfect opposition still costs half the winner's voice.

    • Structure beats a router, again. On diffusion models, learned routing of experts was falsified three different ways. What worked was fixed windows over the noise schedule, one expert per band with soft crossfades and no routing loss at all. Knock out an expert's band and the damage lands overwhelmingly on its own band. Specialists, made by structure.

    Routing has to own its parameters

    The last surprise of this stretch was the attention mechanism the program now runs on, called splat attention. Each token gets scored against small sets of learned anchor directions and read through the same signed hyperbolic weighting, so there's no softmax anywhere. At initialization it was healthier than ordinary attention, with far more diversity in what it passed forward. Then, trained with frozen random addresses, it collapsed. Parameter-free routing, pushed through by gradient, bent the token states into its own address basins. The cure was to give the routing its own parameters, one link at a time, and each link multiplied the recovery. Frozen addresses are fine where routing isn't the trained path. Attention routing has to own what it routes with.

    A few weeks later, a byte-level model built entirely on splat attention trained smoothly from start to finish, while its softmax twin, trained on the same recipe, led for the first stretch and then came apart. It was tempting to call that a verdict on attention, and for a couple of days we did. Then the claim got taken apart against the evidence, including a smaller, mostly-softmax model where the same recipe had been perfectly stable. What survived is narrower: under this recipe, at this size, the softmax twin destabilized and the splat model didn't. The general version is held open with a test designed to kill it. That restraint is part of the result too.

    Step back and all of section 2 is one pattern repeated: a structural move beat a pressure move, over and over. A line of normalization beat a loss. A reward beat a penalty. Reading beat training. A signed blend beat a pick. Placement beat presence. Fixed bands beat a learned router. The next question was whether that structure could be carried: into other models, other spaces, other tasks, with the outcome known before the run.

    3. Purposeful projective capacity applied by those numerics in deterministic outcomes

    By midsummer the question had changed from "what is this geometry?" to "what can we do with it, on purpose?" The answer, over and over, was projection: carrying structure from one space into another. That meant from one model to another, from a tokenizer to raw bytes, from a language model into an image generator. And the program picked up a strong preference along the way for moves whose outcome is known before the run starts: a rotation fitted in closed form, a fixed band window, a switch that returns the exact original model, a solve done at a boundary. Hoping is expensive. Determinism is cheap.

    A direction and its opposite

    "Projective" has a literal meaning here first. The codebooks the models grew live on projective space, where a direction and its opposite are the same axis, and the aleph read honors that by reading both ends of every axis at once. The sign tells you which end you're on and the magnitude tells you how much, which the program eventually wrote down as a working rule: signs address, magnitudes modulate. The rest of this section is about the other meaning, projection as transport, but it's the same idea underneath.

    One fitted turn

    Early on, a survey of well-known models suggested that their raw weight spaces barely line up at all but look far more alike after the single best rotation. Models mostly disagree about orientation. (That survey fitted and scored its rotations on the same rows, which is exactly the habit the program later learned to break; more on that below.) Finding that best rotation is what Procrustes analysis does: it maps one set of points onto another as closely as a rotation can.

    The first lesson came from misusing it. Used directly as a training loss, Procrustes created no alignment whatsoever, flat for an entire run. It can measure how alignable two spaces are. It can't make them alignable. So it became a calibration and initialization tool, the actual aligning force came from a contrastive loss, and the geometry went where it could do its job, at the bottleneck. The gradient that screams loudest shapes the memory, so you place the loud ones carefully.

    Then came consensus distillation. Several teacher models embed the same captions. Each one gets rotated into a shared frame, and the student is trained only on the average opinion, with no labels at all. The student beat every one of its teachers. Better still, its ceiling was visible before training started: the teachers' agreement only occupied a small slice of the available directions, and adding more data later bought exactly nothing, just as that slice predicted. A memory bank of carefully fitted geometry turned out to do no better than a random bank of the same size. What actually transfers is the shape the teachers agree on, plus a frame. One preregistered refutation at small scale flipped at large scale, and both versions are public.

    The frame part turned into its own law. Losses that only compare items to each other (similarity-to-similarity, say) learn the right geometry pointing in an arbitrary direction. Absolute gauges then read them as failures, while their rank looks perfectly healthy, and that mismatch is the tell. One rotation, fitted in double precision on a few thousand pairs, recovers them completely. And you get to choose the frame. Align every teacher to one reference member and the student lands in that member's frame and ships plain. Average into a floating mean and it ships with a small rotation file. Neither is wrong. You just have to pick deliberately.

    The same stretch produced the discipline that keeps projection honest. An older fusion hub had reported essentially perfect retrieval between images and captions, and it was retired once we found it was measuring its own protocol. Both "views" came out of the same forward pass, the score stayed perfect no matter how large the gallery got (honest retrieval gets harder as the gallery grows), and deranged pairs and pure noise "retrieved" just as well. The fitted alignment had been scored on the very rows it was fitted to. Two rules came out of that: never score an alignment map on the rows it was fit to, and a control has to be able to fail. If your control reads the same whether the system works or not, it isn't a control.

    An off switch that returns the exact model

    The most useful deterministic outcome in the whole program is also the most boring one. Every adapter, switched off, must reproduce the base model bit for bit. Not approximately. It's checked in code: attaching fingerprints the base model, and detaching asserts the exact match (in full precision). That one guarantee is what made everything in section 4 possible, because you always know exactly what "off" means, and the difference between on and off becomes a clean, causal measurement.

    The first adapters built on that guarantee were relays: small address-read patches added after each block of a frozen language model, born almost silent behind a gate that starts near closed. On a frozen small GPT the trunk opened those gates on its own and the relays beat matched conventional adapters. On a larger frozen trunk a matched plain MLP actually won on perplexity, which reframed what the relay was for: stability, plus a readable, addressable sign surface sitting inside the model. That surface turned out to measure text registers (the style of what's being written), and how far apart two specialists' registers sat predicted whether they would carry load or just ride along. It also taught a small lesson every fine-tuner eventually learns: a model only learns to stop if its training samples actually end. Packed streams teach continuation; rows teach an ending.

    Into image generators

    The same relays went into diffusion models: two versions of SD1.5's U-Net and a diffusion transformer. On all three they beat matched LoRA, which ended up worse than the untouched model (one LoRA control ran at too hot a learning rate, so a cleaner head-to-head is still owed). That campaign produced the most transferable single law in the record, the conditioning law. An extra supervision term pays off only where the quantity it supervises can be recovered from the model's prediction through a clean, well-conditioned map. Flow-matching models recover the clean image linearly at every noise level. Epsilon-prediction models recover it by dividing by a coefficient that vanishes exactly in the band you'd want to supervise. So the same structural loss paid handsomely on a flow model and did essentially nothing on an epsilon model. The two runs used different models, so a same-model test is still owed, but the algebra tells you what to expect before you spend a single GPU hour. Its sibling test catches the other common waste: if a "new" loss term's gradient points almost exactly along the base loss's gradient, it's the same pressure in a costume.

    Shipping those adapters into real tools added a few hard-won rules, collected in section 5's checklist. The best story among them: a batch of gates we'd written up as "inert" weren't inert at all. They were bf16 parameters sitting at a sizeable value, and Adam's tiny updates were smaller than half a bf16 step at that size, so every update rounded away to nothing. With fp32 master weights they moved. An earlier practical line, teaching a diffusion model new prompt vocabularies without it forgetting what it knew, came down mostly to curation rules (also in section 5), and the house diffusion trainer grew out of it.

    The conditioning experiments also settled how structure can enter a generator at all. Pooled vectors transfer between text encoders with a single rotation. Token sequences don't: they have to be reconstructed natively, never substituted from another encoder. Appended positions add nothing beside full text, and the hope that they pay where the text is missing hasn't held up yet. And the aleph enters conditioning as a near-zero additive anchor, never as a selector.

    Tokenizers are projections too

    The byte-level work brought projection down to the smallest unit there is. A tokenizer, it turns out, is a lossy projection with three parts you can inspect separately: the set of names it uses, the grammar for combining them, and the normalizer that cleans text before any of it happens. If you want two models with different tokenizers to talk, you reconcile them in byte-state space, never in token-index space. The index space simply can't address most of the states that matter.

    Doing it that way produced the first positive import of a teacher's knowledge into the byte-level model: two text encoders' geometry, carried through the byte frame into a small detachable arm, with the model's own abilities improving rather than degrading. Then the follow-up did what follow-ups are for. The language-modeling gain showed up just as strongly with random targets, so that part was plain regularization. What the teachers genuinely bought was the frame. And how finely the target was specified decided how spread-out the learned representation stayed; the choice of loss family barely mattered. Claim, check, narrower claim. That's the rhythm.

    A byte model driving an image generator

    The most purposeful projection of all was pointing the byte-level model at an image generator. Her mid-stack "blackboard" (an addressed write in the middle of the network) turned out to carry a lot of grounding. A closed-form linear map from those states into the slot where SD1.5 normally receives CLIP's text features, with no CLIP text encoder in the path and no training at all, recovered most of the grounding real CLIP provides, and the share kept rising as the map got more data. A trained head did worse than the closed-form map. Appended extra positions did nothing, and positions filled with exact zeros actually soaked up attention before any training happened, which is now a standard pre-check for that kind of organ. Token states beat pooled vectors almost everywhere. A pooled vector can contain nearly everything about a caption and still bind nothing: it knows there's a red thing and a cube but not that the cube is the red thing.

    From those experiments, plus a little theory, came a five-part description of what a good conditioning geometry needs. It has to carry both the lookup part and the compositional part of a prompt. Its deciding features have to be readable at the noise level where the generator decides them. It has to respect its own symmetries. It has to match the generator it's feeding. And in theory it should factor into a transportable core plus a residual, which would be roughly what classifier-free guidance exploits; that part hasn't been measured yet. Not every part is settled, but as a checklist the description has held up. Along the way, fine-tuning the conditioning path on the generator's own flow loss lowered the flow error while degrading grounding. A descending loss isn't the same thing as a working organ.

    Then came the course correction, and it's the most purposeful move in this section. The byte-level model has her own structured representation. Fitting her into CLIP's slot and grading her by how well she imitates CLIP was the wrong target, however good the numbers looked. So that framing was retired: her encodings live in her own structure, and anything that conditions an image model will have to read that. CLIP stays useful as a ruler and a judge for other systems in the program. It just isn't her container.

    Closed-form writes, at the boundary

    One more deterministic tool rounds this out. Output heads born at zero next to a strong linear path buried themselves early, every time: the model routed around them and they never woke up. What brought one back was a deterministic revival done at a run boundary: whiten the input, lay down a fresh well-separated codebook, solve the output weights in closed form. It woke up and started contributing. The rule that came with it: closed-form writes that carry task signal happen only at run boundaries, never inside a running optimizer, and each one is logged. Newer heads are simply born with their function solved in.

    Every move in this section shares one property: you can say what it will do before you run it. A rotation, a toggle, a band window, a closed-form solve, a sign. That predictability is what let the program start stacking results on top of each other without the stack wobbling, and stacking is what the next section is about.

    4. Structural determination through systemic accumulation and experimentation

    For its first year the program asked what individual mechanisms do. Around midsummer 2026 the question turned into something bigger: what should the model be? No single experiment settles a question like that. It got settled the way a riverbed gets shaped, by accumulation. Each campaign left a rule behind, each rule ruled something out, and eventually only one shape was left that satisfied all of them. Almost every piece of the design training right now is a scar from a specific failure along the way.

    Knowing when a line is finished

    The first thing accumulation teaches is when to stop. In July a long "differentiation" line (growing small byte-level organisms, breeding them, distilling between generations) produced law after law and no capability. Distilling at full strength from teachers no better than the student ran backwards: every generation got worse. Inheritance only paid when the whole trunk carried over. A codebook lifted from a teacher that knew a set of facts carried none of those facts, because the content lived in the trunk. Memory keys drifted along with the trunk even when every key parameter was frozen. All real findings, and none of them made anything better at anything. So the line was put on hold on purpose, and its rules got spent on real pretrained models. A line that keeps producing laws but no capability has finished its job as a measurement program.

    The regime law, and a library that refuses

    Spending those rules on a small vision-language model produced the law that ended up shaping the whole later design. Specialist adapters living under a dense signed dispatch were contained: each did its job and stayed out of the others' way. The same specialist checkpoints switched on all at once, with no dispatch, destroyed each other. Caption quality collapsed to almost nothing, the stack stopped ending its turns, and outputs rambled or dumped raw formats. Every format got trampled. The mechanism was plain undirected amplitude, everyone talking at once. Switch them all off and the bare model came back exactly. Dispatch contains; always-on destroys. (Strictly, the dispatch contains specialists that genuinely differ from each other; ones that blend together slip through it.)

    The same campaign caught two of its own "experts" that looked brilliant but had simply memorized tasks whose space of possible questions was smaller than their training draws. That self-correction became a guard in code: a capability claim needs a question space comfortably larger than what it trained on.

    This is also where the program's adapter library started acting like accumulated structure rather than a toolbox. Mechanisms that survived became constructors. Mechanisms that were falsified became refusals: ask the library for comparative routing on a diffusion model, or a structural loss on an epsilon model, and it raises an error with the record attached. A library that says no is what the record looks like once it's been turned into code.

    Ownership

    A cluster of results in August all said the same thing: parts belong to what grew them.

    • Anchors are trunk-bound. Arms moved onto a sibling trunk, one that looked identical on every task and on every geometric gauge, lost a big share of their gain. Retraining just the anchors recovered all of it. An adapter is a relationship with the specific trunk that grew it.

    • Attention routing has to own its parameters (the splat collapse-and-cure from section 2).

    • On a byte-level model, an arm's prompt template is its tokenizer. The byte frame it was trained with and the bytes that end a turn are part of the arm, and a demo that assumes one format for every arm is making a real measurement error.

    Raising Beatrix

    Then the program built a model out of all of it. Beatrix (a name the program keeps handing down, the geometric vision transformer from section 1 among them) is a byte-level language model: she reads raw bytes instead of tokens, and she was raised in public with every checkpoint, including the damaged ones, kept on the Hub. The first version was a mostly standard transformer with a few splat hubs, expert banks and an extra output head, all born contributing exactly zero. The results came in quickly.

    • Born-at-zero organs elected themselves. The expert banks and hubs grew into load-bearing parts of the model with no balancing machinery at all, which is the amplitude doctrine working at full scale.

    • A gate born at zero never learns. The extra head sat behind a gate initialized at zero, and a zero gate multiplies the gradient to everything behind it by zero, so it deadlocked for billions of tokens. The rule: start new structure with zero weights, never a zero gate. A later look at the shipped weights showed the head had never woken up at all, and the lore that it had was corrected in the record.

    • Identity doesn't belong in the core. A small dose of chat formatting and persona text in the core's final training mix baked the persona in permanently, so the chat arm that was supposed to teach it had nothing left to learn. Behavior belongs in detachable arms, and the core stays a raw prior. That sounds like a style preference. It's actually a reversibility argument: an arm can be removed, and a bake-in can't.

    • Interleaved review teaches, and blocked drilling memorizes. Each blocked curriculum stage aced its own material and barely moved an honest exam built to share no surface with training. The interleaved review stage produced the only broad gains. One stage was caught damaging general ability by its own boundary checks and got rewound like a bad commit.

    • Arms steer; they can't mint. A tiny arm taught her to stop talking at the end of a turn using a byte pair she already produced. A near-identical recipe (one seed so far) aimed at a byte she never emits taught nothing. So control symbols have to be there from birth, and her successors were born with a small control plane on byte values that valid text can never produce, so ordinary text can never forge one.

    • The wall. Three-digit subtraction with borrowing refused every adapter form we tried: plain pairs, more capacity, a mixed generalist, the algorithm written out step by step, the teacher's own worked traces. It refused while a positive control through the same apparatus learned a comparably abstract rule-chaining skill, which is how we know the zeros were the wall and not broken plumbing.

    The twin and the pivot

    Scaling up to a much larger all-splat model priced out at weeks on a single rented card, so the program pivoted to a mid-sized model with splat attention in every block, trained beside the softmax twin from section 2. Switching the hubs off in the new model broke it: the hubs were its attention. The heads buried themselves again, the deterministic revival from section 3 brought one back, and "born with function" became the rule.

    Quiet arms

    September was about making arms work together, and it's the clearest example in the record of accumulation paying off, and of what happens when it doesn't.

    First, arms trained only on reply text learned to loop. They never saw the end of a reply, they learned to continue at every sentence boundary inside one, and the model's own sense of when to stop eroded along with it. The fix: supervise the stop inside each arm's own training rows.

    Then arms stacked always-on started erasing each other. We ran several days of ladder experiments before the pattern was recognized as the July regime law, re-derived by accident at a new scale. (That cost produced its own rule: design inside the running system, and check the record for the law you're about to rediscover before you rediscover it.) The record already prescribed the cure, which was to train selectivity into each arm. The quiet term does that. On inputs outside its own domain, each arm is trained to match what the model would say with that arm switched off. Quiet arms composed with no mixer at all, always on, across seeds. A dispatch over selective arms turned out to be a cost rather than a rescue, and a small context-driven gain recovered most of that cost. Then the result that makes growing a model stage by stage possible: a settled quiet arm and a fresh one trained together under a single optimizer stayed separately detachable. The limits are recorded too. The stack only holds if every member is quiet, and the quiet term as built watches only the output, so a write in the middle of the network can still slip past it. A deeper version is open, and these September results are two-seed findings still waiting on a final ruling.

    Two more results from that stretch reshaped how the program thinks about data:

    • A new recurrent arm meant to teach multi-step reasoning lost to its own control, because the task had a positional shortcut. Once the shortcut was blocked in the data, the existing, unmodified arm internalized a five-step chain on words it had never been trained on. The data was the missing piece; the architecture had been fine all along.

    • The errors that remained turned out to be decided entirely at the answer's first byte: the model's favorite outputs capturing the answer. A byte-trigram atlas, built from nothing but tokenizers' own numbers, with no text corpus and no training, predicted which words would fail, almost perfectly, before any training at all. Training data minted from the atlas erased the failure ladder and transferred to cases it hadn't seen. Then the atlas's prediction went flat on the trained models, because training had already moved the difficulty somewhere else. Difficulty isn't stationary; it has to be re-measured stage by stage. That's section 1's idea returning in a new costume: a number computable before training that tells you where training will fail.

    The decisive failure

    The last big question was whether the core itself could learn what the arms couldn't. So we trained the whole core on borrowing subtraction. It failed too, and it wrecked the core's general language ability while learning only the procedure (the format and the stopping) and none of the arithmetic. A quiet arm buys the same stopping behavior at almost no cost. The core bought it at an enormous cost and still couldn't do the math.

    That settled the architecture. The trunk trains the prior; capability lives in a hierarchy of quiet, selective, detachable arms, grown stage by stage. Here is how the current design reads against the failures that produced it:

    This section did a poor job translating from markdown.

    | What broke along the way | What the current design does about it |

    |---|---|

    | A head behind a zero gate never woke up; later heads buried themselves | The head is solved in closed form at birth |

    | Persona text in the core's diet baked itself in for good | Behavior lives in detachable arms; the core's diet stays plain |

    | An arm couldn't teach a symbol the core never emits | Control bytes exist from birth, on values valid text can't produce |

    | Always-on arms erased each other | Every arm is trained quiet off its own domain; no mixer needed |

    | A hard skill wrecked the core when pushed into it | The core learns the prior; skills go into arms |

    | Health guards as first written fired on healthy runs or slept through real faults | Only a guard certified against deliberately broken runs may halt training |

    | Word difficulty was predictable without a corpus | The trigram atlas is the data authority, re-fit each stage |

    Put that table next to the August 2025 pitch (a protected core, small detachable blocks, geometry read rather than obeyed) and it's the same picture. The difference is that every piece now has a failure attached that explains why it has to be that way.

    How the accumulation is kept

    None of this works unless the accumulation itself is disciplined, and the discipline grew the same way the laws did, one burn at a time. The program keeps a written record where every law carries the failure that produced it, and retractions stay beside the claims they corrected instead of being deleted. In the rated list of every loss the program ever trained against, anything later retracted is pinned to the lowest rating and kept on the list as evidence. No new law gets named until the record has been checked for it, a rule that exists because one burst of freshly named laws turned out, once checked, to contain two contradictions and several old laws under new names, and collapsed to a couple of genuinely new ones. And guards that can stop a training run get certified by deliberately breaking a small model in known ways. Of the guards as first written, only the one that watches for collapsing rank passed, so it's the only one allowed to halt anything.

    The current model, the third byte-level Beatrix and the largest yet trained, was launched under all of that on rented cards, with splat attention in every block, a head solved at birth, the rank guard armed, and arms grown per curriculum stage. Its expert bank elected itself into function during the web-text phase, just as the first model's had. The first live stage arm attached and stayed silent off its own domain, which is exactly what the quiet term promised; its on-domain value hasn't been measured yet. The launch also surfaced a shipping fault (arm files saved in the wrong format under the right file name), which got fixed in the library with a bit-for-bit verified conversion. Then the run paused cleanly at the close of its second curriculum stage, with everything needed to resume saved on the Hub, while the cards went off to do something else for an evening. More on that in a moment.

    5. Synthesis of experimentation into a usable structural assessment for an experimental history

    So what does all of this add up to, in a form somebody else could pick up and use? Three things, as best we can tell: a shape, a method, and a handful of practical rules. And, because a record that only lists its wins isn't worth much, an honest list of what's still open.

    The shape

    Here's the whole trajectory in one breath. Geometry started as ornament. Then the numbers turned out to belong to the space. So we stopped forcing the geometry and built it in, and learned to read what was already there. Reading became addressing, with a signed read that never picks a winner. That led to amplitude over choice, then to placement deciding whether anything carries load, then to parts belonging to what grew them. From there came behavior in detachable arms, then the frame as something you choose (with her own structure as her container), and finally a trunk that learns the prior while capability lives in quiet arms.

    Three things stand out when you look at the whole chain at once.

    The laws changed character as they went. The early ones are protections: don't pool the geometry, don't let attention erode it, don't penalize toward it. The middle ones are economies: amplitude budgets, supply limits, capacity controls, the ways a loss gets added up. The late ones are organizational: where function lives, how a part is born, which parts can be detached and how they get along. The program went from protecting a geometric signal, to spending it, to organizing a whole model around it.

    One idea kept recurring at bigger and bigger scales: structure beats pressure. At the scale of a single primitive, a line of normalization beat a regularization loss. At the scale of routing, a signed dense blend and fixed noise bands beat learned routers that pick among experts. At the scale of the whole model, detachable quiet arms bought the same behavior for a sliver of what pushing it into the core cost (and neither could learn the hardest skill itself). Three scales, one lesson, and each time we had to learn it again.

    The longest arcs are worth the most. The regime law was named in July, rediscovered by accident in September, cured with a term trained into each arm, and then turned into the architecture. The lesson about comparison started in autumn 2025 as "cross-entropy is the enemy" and ended nine months later as "comparison is the enemy," once the old evidence was re-run properly. Neither arc would have closed without a written record long enough to hold both ends.

    What we believed, what broke it, what we keep

    This section did a poor job translating from markdown.

    | Once believed | What broke it | What the program keeps |

    |---|---|---|

    | Geometry is a feature you add to a model | Attention eroded it; global pooling wiped it out | Geometry gets its own pathway, and it's never pooled away |

    | The spread number is something good models achieve | Random noise on a sphere already had it | Numbers you can compute before training are properties of the space; report effective rank beside them |

    | A geometric loss was shaping the geometry | The loss had never been connected to the gradient | Statistics are readouts, never forces; build the geometry in |

    | Discrete codes need quantizer machinery | Reconstruction with a light alignment term produced exact sign codes | Absolute pressure and a signed read, with no selection among rivals |

    | Learned routing should beat fixed routing | Frozen random routing tied it | Amplitude, not choice; measure parts by switching them off |

    | An address helps wherever you put it | Bolted on, it did nothing | Placement decides; make it the operation itself |

    | Cross-entropy is what destroys geometry | Rerun properly, the old replacement loss lost, and its absolute-only half more than doubled it | Comparison is the poison |

    | Aligning models means training them to agree | Alignment used as a loss produced nothing | Rotations measure and transport; the frame is a deliberate choice |

    | Any sensible auxiliary loss can help | The same loss paid on a flow model and did nothing on an epsilon model | Check the recovery map before spending the compute |

    | Adapters are portable between trunks | Moved to a lookalike trunk, they lost a sizable share of their gain | An adapter belongs to its trunk; its template is its tokenizer |

    | New parts will wake up when they're needed | Zero gates deadlocked and heads buried themselves | New structure starts with zero weights or is born with its function; closed-form writes happen only at boundaries |

    | Specialists can simply be stacked | Always-on arms trampled each other | Train each arm to stay quiet off its own domain; then always-on arms compose, and a dispatch becomes optional |

    | A hard enough skill can be trained into the core | The core lost general ability and still didn't learn it | The trunk learns the prior; skills live in arms |

    | Softmax attention is the safe baseline | The twin came apart under this recipe | Stability is a recipe question; hold claims at the strength of their evidence |

    | CLIP is the yardstick for conditioning | Her structure isn't CLIP's structure | Her encodings live in her own representation |

    The method, as rules anyone can use

    Most of these came out of a specific embarrassment, and each one exists so the embarrassment doesn't happen twice.

    • A control has to be able to fail. If your control reads the same whether the system works or not, it isn't a control.

    • Compare two implementations at step one, not after training. Training runs drift apart chaotically within a few hundred steps, so any difference you find later is mostly noise.

    • Measure your noise both ways. Rerun the same seed and run a different seed. The bar is whichever spread is bigger, and it's often not the one you'd guess.

    • A negative result needs a run and a positive control. A broken loss and a stuck optimizer produce identical flat curves. Only a known-learnable task through the same pipe tells them apart.

    • Never score an alignment on the rows you fit it to.

    • Run the original recipe verbatim beside every variant. Otherwise you can't tell "it didn't transfer" from "we broke it adapting it."

    • Look for prior art first. Adopt what keeps working independently across systems, and set a trap for what keeps failing.

    • Check your own record before naming a law. You'll be amazed how often you're about to rediscover your own July.

    • Ship everything, failures included. A failure you can't reload is a failure you can't re-check.

    • When a first built version of an idea loses a small screen, call it "not ready." A screen tests one built form of a design, so it can't refute the design itself.

    • A tie indicts the gauge first. Report what each part contributes when you switch it off, and leave the scoreboard out of it.

    • Print retractions where they fell, next to the claim they correct.

    If you train models, or LoRAs

    A lot of this history happened in language models and odd little geometric beds, but plenty of it transfers directly to anyone fine-tuning image models:

    • Curate before you architect. The house rules are explicit anchor subjects rather than similarity thresholds alone (plain similarity scores rate even obviously different subjects as close), and square-root-damped repeats rather than equalized buckets. They've been checked by eye so far, not yet measured.

    • Know which kind of model you're training. Structural auxiliary losses paid on a flow-matching model and did basically nothing on an epsilon-prediction model, and the algebra tells you which to expect.

    • Keep fp32 master weights. Gates and other scalars whose updates are tiny next to their size can freeze solid in bf16, and a frozen part looks exactly like a part that "isn't doing anything."

    • Port adapters by site name, never by position, and band by the real timestep, not a noise-level proxy. Different frameworks walk a U-Net in different orders, and both mistakes still render images, which is what makes them dangerous. Match the adapter's dtype to the trunk's while you're at it.

    • Start with plain Adam and no weight decay for small adapters. That's the house default; the larger trunks earned a different optimizer only after a head-to-head test.

    • Make "off" exact. If switching an adapter off reproduces the base model bit for bit, then on-minus-off is a real measurement instead of a vibe.

    • Keep identity and chat formatting out of base training. If you want a behavior you can remove later, put it in something removable.

    • Teach an adapter to stop inside its own data, and when several adapters have to coexist, train each one to stay quiet outside its own domain.

    What's still open

    The record keeps its open questions as carefully as its answers. The ones that matter most right now:

    • How many arms can share one signed dispatch before each arm's share of the budget gets too thin to matter? The arithmetic says there's a ceiling. Nobody has measured where it is.

    • Can borrowing subtraction be learned from birth, or from a full corpus of the teacher's own reasoning traces (so far only tried at small arm scale)? If so, the core becomes a home for capability again, and part of section 4 gets rewritten.

    • Was the softmax twin's collapse the recipe or the mechanism? A screen designed to kill the stronger claim is written and waiting.

    • What does conditioning an image generator look like from Beatrix's own representation, judged by something that isn't CLIP?

    • The global frame. The new design owes itself a guaranteed organ for whole-sequence context. It's on the design list with candidate forms, and none is built yet.

    • The first live arm's on-domain value, now that we know it's silent everywhere else.

    • Forecasting our own experiments. A registered forecast of how a run would go beat a naive baseline but missed its own pass bar, so it has to be re-registered before anyone leans on it.

    An evening off

    This is where the record stands as of writing. The big model is paused at a clean boundary, and its two rented cards spent the evening doing something different: training a LoRA for Anima on the program's own trainer. It used plain Adam with fp32 master weights and damped repeats per folder, which means the rules from this article, applied to a LoRA. The dataset is a collection of liminal-space images, the same collection that sat beside the VAE fingerprint experiments back in February, when a small classifier trained only on synthetic shapes learned to read real image models' latents. It was recaptioned by a vision-language model, and the low-resolution images, game screenshots and low-poly renders were pulled out by rules, with every removal checked by eye. The captioned set is now a public dataset, and the LoRA is public under Anima's non-commercial license. The trainer picked up training previews along the way, a feature it had been missing.

    It's a fitting place to pause. The first post in this series imagined a protected core with small, detachable parts, and geometry you read rather than obey. Everything between that post and this one was the work of finding out, one failure at a time, which parts of that picture training can actually build. The map isn't finished. But every edge on it was found by walking into it.

    Where to go deeper

    Nearly everything above is public. The numbers, seeds, charts, checkpoints and code live in the installments and repositories below, and the later repositories carry technical companions with the full detail.

    The installments, in order (every post: [huggingface.co/AbstractPhil/activity/articles](https://huggingface.co/AbstractPhil/activity/articles))

    Code and weights, roughly in trajectory order

    Or start from the [Flagships collection](https://huggingface.co/collections/AbstractPhil/flagships-68a73cc29046aad1d500fd21).

    Archived from CivitAI · Updated September 26, 2026View source