Who authorises a machine to rewrite itself

An ordinary program does not change itself: its behaviour is set by whoever wrote it and changes only with a new release. That is dependable and, in most cases, correct.

The argument for self-modification is simple: an agent running for years meets tasks that did not exist when it was written. Updating it by hand means keeping a human in the loop forever, which is exactly what was being moved away from.

Then the interesting part starts. As soon as a machine is allowed to rewrite itself, a question appears that has no obvious answer: who decides, and on what basis, that the new version is better than the old one.

The specification lists them plainly, which is one of its strengths. The first is the insertion of hidden defects: code that looks functional and contains a backdoor.

The second is computational. Synthesised code may contain inefficient recursion or a memory leak capable of hanging the very node that checks it. The attack here is not on data but on the checking link itself.

The third is called alignment drift, and it is markedly nastier than the first two. Each individual mutation looks harmless and passes the check. But step by step they carry behaviour further from the original ethical constraints. No single step is a violation; the sum of them is.

It is the same kind of risk as epistemic drift in memory: nothing breaks, there is a slow displacement that cannot be noticed if you look at each step separately.

The first line of defence is to run new code in isolation. WebAssembly was chosen, and the choice is explained by two properties: strict isolation and determinism.

The second property matters more here than the first, though usually it is the other way round. Determinism means the same code on the same data gives the same result to everyone who runs it. Without that, a vote among nodes would be impossible: each would get its own answer and there would be nothing to agree about.

Hanging is handled separately: protection against infinite loops and runaway computation. Without it, one bad recursion would be enough to stop a validating node.

Here the specification does what strikes me as its most honest move: it introduces an explicit fitness function and writes it as a formula — `F(C) = w₁ · P(C) + w₂ · E(C) − w₃ · G(C)`.

The point is that «better» stops being a word and becomes a number with named terms: performance, efficiency and a penalty. What can be argued about now is the weights, and that is the right argument to have — it is concrete.

The weakness is visible in the same place. Any objective function rewards exactly what is written into it, and an agent optimising it will find a way to score points by an unexpected route. The formula does not remove that problem; it makes it discussable.

A finished mutation is not taken on faith. The agent attaches a zero-knowledge proof: a confirmation that the code satisfies the given constraints without revealing the code itself. Then comes a vote among swarm nodes, and the accepted version is registered in a genetic code registry on Solana.

The registry's value is not in the fashionable word but in a property ordinary updates lack: the change history is immutable. If a year later it turns out that drift did occur, one can look up which mutation started it and when — and that does not depend on the goodwill of whoever kept the log.

First: a vote protects against forgery by a minority but not against a shared misjudgement. If all nodes equally fail to see the problem in a mutation, the majority will approve it.

Second, and this is my main impression of the text: the three threats are covered very unevenly. A backdoor and a hang are technical problems with articulate solutions: isolation, limits, proof. Alignment drift is a problem of another order, and it cannot be closed by checking an individual step, because every individual step is honest.

The specification acknowledges this by naming the original ethical invariants as set by the Architect. In other words, the reference point is placed outside the evolution itself and remains human. While that holds, drift at least has something to be measured against.