Now Pangram has a pretty low false positive rate, but if you have ever used an LLM as a writing assitant, you will have probably noticed that it claims your posts 100% AI, even though you don’t feel like they are.

Pangram itself is a trained model, that attempts to detect segments of text as being definitely human, definitely AI and a mixture of the two. If you want to know how it works, they published a paper. The short summary is that they are manufacturing its own training data by starting from collections of known human authored text. An LLM is then tasked to understand the text and write a fresh new text on the same topic. They also let the LLM perform partial edits on that original human text and through that they can pick up on these co-authored details. Pangram claims their model to have rates of 0.0041% false AI accusations and 0.34% missed AI text.

So now that we know this I figured it might be fun to have an LLM re-create David’s tweet. I first came up with a prompt. And when I say I came up with that prompt I in fact used an LLM to propose to me from that tweet what I might want to say for the structure. I’m sure if you ask Pangram about if the above text is AI, it will probably say so, but that’s not really the point. The point is that I then used Opus 5 to generate a text which reads entirely AI generated.

If you are curious, this is the prompt I used:

The text that Opus 5 then wrote, was this one:

Dario published “We Must Pace the Frontier.” Sam agreed, and said OpenAI will also commit to independent evaluators with employee-like access. My response is not the one they expect: go ahead. Please do.

Because you two are the frontier. OpenAI and Anthropic hold a duopoly on frontier intelligence by every metric anyone actually uses: capability, revenue, enterprise share, developer mindshare. And you both keep telling us the lead is widening, that models are starting to improve models. If that’s true, pacing costs you a quarter you can reclaim whenever you like.

I also can’t see your unreleased checkpoints. If what’s sitting behind the curtain genuinely scares the people who trained it, then slowing down is your call to make, and the right one. I’m not going to second-guess a risk assessment I have no access to.

But stop pretending the rest of us are the problem. Stop pretending this is a collective action failure in which a handful of startups are the defectors.

Stop pretending open weights are the dangerous frontier while the two of you ship the most capable model in the world every few months. Stop pretending third-party evals require new law, when you both just committed to them voluntarily, in public, on a blog.

And stop pretending it’s purely altruism. Pacing the frontier is also margin repair. It slows the price war, stabilizes the capex story, stretches depreciation schedules across a longer product cycle, and makes life considerably harder for fast followers who survive by distilling whatever you shipped last month. Safety and self-interest are pointing the same direction here. That’s worth saying out loud.

Still, the upside is real. Pacing buys room for a smarter regulatory debate than the one on offer, where the loudest alternative is Bernie Sanders’ “shut it all down.” That’s a slogan, not policy. Meanwhile China keeps shipping competitive open weights, and any rule that binds two American labs while the rest of the world downloads its capability for free isn’t safety, it’s theater with a very expensive ticket price.

And well, Pangram agrees that this is 100% AI. So far, so uninteresting. It does read somewhat like David’s tweet, but obviously not entirely. Given that the original prompt does not have enough information to re-create the tweet entirely you would expect some divergences.

The actual thing that interests me is if you can take this output at all, and then rewrite it from scratch, but by sticking to the general structure and ideas. Will Pangram give us a AI or human rating?

I read the generated text. Then I read each paragraph and decided to rewrite and rephrase it without an LLM. According to some similarity checkers, they the final texts are 50% similar which seems about right. But strictly speaking, not a single sentence is the same. Here is the 100% human rewritten text of the above one. No LLM was used to write it, but an LLM was used to fix up typos in the end. That from my experience really does nothing to tick off an LLM detector.

Dario has written “We Must Pace the Frontier,” and Sam from OpenAI has agreed. My response might surprise people: go ahead, please.

You two are the frontier! Your companies, OpenAI and Anthropic, are at the frontier by all metrics: revenue, developer mindshare, adoption, capabilities. And yet you both claim that your lead is widening as a result of recursive self-improvement as models are improving models. You currently are the duopoly of self-improving models!

I am unable to see what unreleased models you have. When what you have behind those doors really scares your folks, then you should slow down. I’m not going to tell you otherwise and I support you.

But please don’t pretend we are the problem. Stop pretending you need our permission. Stop pretending this is all a collective issue when in reality this is all on you. Stop pretending open weights are the problem here. Stop pretending pulling third-party evaluators in requires lawmaker involvement. And for the love of all the good things in the world: stop pretending this is all about altruism.

Pacing the frontier is also about your margins, and it makes it harder for fast followers. And it patches up your capex story and has the potential for slowing down the price war ahead of the IPOs.

But yes: pacing might give us the space for a better debate than Bernie Sanders’ “shut it all down.” There is no policy there. And while we’re having fights at home, China will keep shipping competitive open-weight models and won’t adhere to any American agreements.

This is all regulatory capture hiding behind a safety debate, and the rest of the world is watching.

So what does it say? Well this text too comes back as 100% slop. And it does not surprise me all that much. I have generally noticed that if you rely on an LLM to give your text structure, it will score badly on Pangram even if you do plenty of edits over it. In fact, it’s quite unlikely you’re going to get a post that starts out as slop into a structure that will make it appear that it’s not.

I came to quite appreciate the existance of Pangram because at the very least it has made me quite aware of some of the effects that using LLMs for writing blog posts has. This blog has been AI supported for about two years (as you can see from the AI transparency link on the bottom but I did notice that I became both more reliant on those tools and that they have become much more aggressive editors and it gave me pause.

Yet, I also think that plenty of people will find a “100% AI” rating misleading when in fact the author has done plenty of editing. But maybe it’s fair to have this to show up as entirely AI?

I encourage you strongly to read the post, because I think it’s a good one. And yet, when I read the post I could not help but feel in strong opposition to it, despite the fact that I think I’m on the same page with regard to all observations and, to a large degree, the concerns.

I thought it might be interesting to write down my present-day thoughts on this, even if for no other reason than for myself to look back at it a year or two from now.

What Is Doom?

What I really appreciate about Dario’s post is that he lays out a scenario that is not a huge stretch but also one that describes a clear, unfortunate outcome we should fight: persistent botnets and other forms of nuisance. And well, we don’t have to look very far to see the issues left and right. Wikipedia has a page called 2026 OpenAI agent cyberattacks which gives you at least some overview of what we figured out agents have hacked up to this point. Except I know it’s not up to date, because for instance they also poisoned RubyGems.

Today these systems might be annoying, but they can be turned off when we figure out where they are. Except, it seems like OpenAI and Anthropic are operating at such a scale that they seemingly can be completely blind to what their systems are doing.

I don’t think we are anywhere close to a world where an agent might decide to hack into core inference infrastructure to upload weights to other GPUs to survive. But simultaneously it’s entirely in the realm of possibility and primarily curtailed by the labs probably being particularly careful about their IP.

For me the scenario I primarily worry about is what it does to us. And by us I mean anyone who is not currently working on closed weight, dopamine-loaded, subsidized token faucet. I really don’t worry about someone using these models to build a nuke, or to control some rockets in the Middle East, or that America would lose against China in some international culture war. I almost exclusively worry about what this does to us as humans.

What Needs To Be Paced?

What I find absolutely hilarious and simultaneously entirely frustrating about this conversation is that there is this idea that there is something to be paced. First of all, we should really talk about who Dario is talking about here. There are really only two companies: Anthropic and OpenAI. Nobody else matters in this space right now (this might change, but we’re talking about the right now). Both of those companies are basically coming from the same origin. The solution that Dario proposed, at least in part, is a third-party evaluator that in this case is METR. Which, unsurprisingly, also has strong ties to both OpenAI and Anthropic. Sure, there are some philosophical differences between the companies, but they are much more alike than they are different.

Both those companies greatly benefited from being able to train on public data that we all generated in one form or another over the last decades. They are also both increasingly causing strain on public resources, though it seems that OpenAI has their shit way less under control. But now we are presented with the idea that what these models are being trained on is so dangerous that it really should be in the hands of very few American corporations to decide who can do what and when and how.

But behold, Dario is also very worried about China. It starts with using AI for “democracy and freedom” and then it asks for ensuring that a gap with China exists. All new recent shenanigans on the Anthropic API are fully there to prevent the distillation by the Chinese, and they are not at all hiding it.

Automatic Pacing

I can tell you when the topic of AI safety and pacing is much less of a concern: if we actually were forced to have open weight models to begin with. A powerful technology that is out there for everyone to use comes with built-in pacing. In a way it’s the truest form of MAD or proliferation. I would argue we are in this pickle in the first place because right now the public is massively supporting (indirectly) the development of these models but simultaneously has to buy back the economic benefits that they might create from very few labs who have significant power. And their power is also seen as a geopolitical power, at least in the US, and maybe to some lesser degree in China.

And I know I use “public” loosely here. PyPI is not a public project, nor are RubyGems or GitHub. But they’re part of the Open Source commons and large AI companies are currently doing a tremendous job at stressing these in an effort to train ever more powerful models.

We should be glad that China is currently massively bailing out the world. If it were not for Chinese labs distilling American models, we would be in a pretty awful situation right now, particularly as Europeans. The open weight models are driving innovation and the diffusion of capabilities, and are leveling the playing field.

If we greatly restrain our AI capabilities in the belief that China will do the same, and then China defects, AI could be so powerful that such a defection could lead to their geopolitical dominance. Therefore any agreement must either have ironclad verifiability, or must be limited enough that defection would not be militarily existential.

— Dario Amodei

I am assuming Dario has reasons to believe this, but the models that are actually causing issues right now are all closed weight American models. I’m fairly certain if they were open weight models, we would not have that issue. Why? Because for a start, the economics of serving up these models are only that distorted due to how the big labs can operate. OpenAI is casually burning 18 million USD to brute force a problem on a whim. They are operating subscriptions at a massive loss, distorting the market everywhere. If we had mass accessibility on somewhat equal terms, a lot of the crazy issues we are seeing today would not be taking place.

A Total Regulatory Failure

From where I sit, what we observe right now is a total regulatory failure everywhere. In Europe you have some whacky AI regulation that is two years old and completely misses the problems that we actually have and focuses on problems that nobody has. In the US we’re seeing a system that is probably best described as turbo capitalism paired with sinophobia and erratic decision-making. In the chaos in which we find ourselves, the reality emerges. And the reality is, even today, really problematic.

Whatever laws and regulations already exist are largely completely ignored. Plenty of companies are buying data from all over the place that people never agreed could be used for training of AI models. The token economy that is emerging is one that looks like a drug market where you don’t know where the requests are going, what model is served up to you, where the GPUs are even running, let alone what you pay for all of this.

We now have mathematicians who are scared that their use of ChatGPT leads to future models being trained on their ideas, and OpenAI apparently can’t even rule it out.

Ideally the regulators would have forced these models to actually benefit the commons if they are from the commons. The internet has, for instance, greatly benefited from very liberal rulings in the US that permitted scraping. Learning on public data could have been regulated in a way that labs would have to actively support and enable certain forms of distillation. That alone would dramatically change how these models are trained.

What Might Happen?

As I said before, I don’t think AI is going to usher in an extinction event. In fact, even if nobody were to slow down, I really don’t think humanity would have much to worry about. I tend to think it would actually be the large labs that have much more to lose there in reputation and legal responsibilities. I find it preposterous that OpenAI’s agents are committing actual crimes out there, but we’re just shrugging our shoulders and moving on as if nothing happened. But I’m sure executives in those companies are waking up to the reality that this is not at all popular with a lot of their potential consumers.

I also think that this entire recursive self-improvement business has a good chance of being a problem. But not necessarily in that it will cause the end of humanity or societies, but that it will just do massive damage everywhere.

And really, it will just make a lot of the things we are doing much more expensive. Software engineering is an early victim of that. The newfound powers so far have resulted in a new tax that companies need to pay to the model providers, both to keep up with the new speed and to deal with the problem of these machines finding security issues left and right.

And presumably what is going on in software will happen to more industries. Universities and research groups will have to pour a lot of money into the closed models as well, to keep up with others who do.

In a way, I’m really confused that society is taking all of this so well.

That’s how I feel about AI right now.

Which brings me to GPT 6 Astra. Astra is by all accounts an incredibly impressive model. There is really not much I can say against this. It’s amazing at computer use, understands images and complex topics, and it’s relentless in its pursuit of completion. It is absolutely impressive; these types of models are going to change the world in one form or another.

But at least for the moment I don’t know how to work with it for actual software engineering. Since that got quite a bit of attention on Twitter, I figured I might summarize my thoughts and just share what kind of code comes out of this thing.

My Slop Factory

“Armin, you should run a software factory!” I’ve heard that a few times now, so I figured I might celebrate the release of it by running a little software factory over the weekend. If everybody builds slop 3D games, then I should do something useful with it. My software factory was intentionally set up to let the model decide the how of the workflow entirely. It was free to manage its own context and could maintain its own records in an agent-notes folder. Then it spun off subagents to work on stuff. The goal? What if we had a Python with virtual threads and lexical scoping. And well, I burned a full reset’s worth of ChatGPT tokens on this which appears to be around 4 billion tokens. 35 hours later, the factory has delivered absolutely nothing of value and also not taught me anything about how to operate a better one.

But it produced a lot of code and input prompts, and so there is stuff I was able to study. And well, it shows behavior that I’m not used to with Sol and earlier OpenAI models ¹. I have since encountered the same issues with regular programming with Astra, so it’s not a result of just the factory.

I think I’m suspecting something is going “wrong” in the training process. The model is greatly rewarded for succeeding on long-horizon tasks, but presumably there is very little punishing going on for “shitty code.” The apparent result is that Astra is amazing at producing 3D stuff and it can keep going for a very long time, coming up with its own work in the process. I had it do quite a bit of reverse engineering of my robot vacuum in ways that were quite impressive. So it’s definitely cool!

Codegolf Tool Calls

The first issue I have with Astra comes from the type of code that it uses for tool calls. Codex increasingly has been relying on “just bash” to do more and more operations. For a few versions now the original Codex harness just uses sed and other tools to read files. You just usually can’t see them because Codex parses the bash commands and hides them if it recognizes them. But Astra … really loves Python? That is not much of a surprise because even older OpenAI models had a tendency to sometimes use on-demand Python code to read and manipulate files at times, but Astra does it really quite excessively for me.

Now here is an important disclaimer: this project is very meta here because I worked on the CPython interpreter. But I can assure you that I have seen this model do weird Python things even in TypeScript code in Pi. But I have the most evidence of odd code from when I had the thing work over the weekend with zero oversight from my slop factory.

That it writes Python is not interesting; the type of Python is interesting, and I collected some outputs for you to gloss over.

python3 - <<'PY'
from pathlib import Path
p=Path('Include/internal/pycore_intrinsics.h');s=p.read_text().replace('#define MAX_INTRINSIC_1                         14','#define INTRINSIC_RETAIN_ANNOTATION_CELLS        15\n\n#define MAX_INTRINSIC_1                         15');p.write_text(s)
p=Path('Python/intrinsics.c');s=p.read_text();idx=s.index('#define INTRINSIC_FUNC_ENTRY');s=s[:idx]+'''/* Hold every old cell until the compiler has published the entire site's new
   capture. A replaced cell's finalizer may reenter module __annotate__. */
static PyObject *
retain_annotation_cells(PyThreadState *tstate, PyObject *holders)
{
    if (!PyTuple_CheckExact(holders)) {
        PyErr_SetString(PyExc_TypeError, "annotation holders must be a tuple");
        return NULL;
    }
    Py_ssize_t size = PyTuple_GET_SIZE(holders);
    PyObject *previous = PyTuple_New(size);
    if (previous == NULL) return NULL;
    for (Py_ssize_t i = 0; i < size; i++) {
        PyObject *holder = PyTuple_GET_ITEM(holders, i);
        if (!PyCell_Check(holder)) {
            Py_DECREF(previous);
            PyErr_SetString(PyExc_TypeError, "annotation holder must be a cell");
            return NULL;
        }
        PyObject *cell = PyCell_Get(holder);
        PyTuple_SET_ITEM(previous, i, cell == NULL ? Py_NewRef(Py_None) : cell);
    }
    return previous;
}

''' +s[idx:];s=s.replace('    INTRINSIC_FUNC_ENTRY(INTRINSIC_AWAIT_BLOCK, await_block)','    INTRINSIC_FUNC_ENTRY(INTRINSIC_AWAIT_BLOCK, await_block)\n    INTRINSIC_FUNC_ENTRY(INTRINSIC_RETAIN_ANNOTATION_CELLS, retain_annotation_cells)');p.write_text(s)
p=Path('Python/codegen.c');s=p.read_text();idx=s.index('static int\ncodegen_annassign(');s=s[:idx]+'''static int
codegen_retain_annotation_cells(compiler *c, location loc, PyObject *captures)
{
    Py_ssize_t pos = 0;
    PyObject *binding, *holder;
    while (PyDict_Next(captures, &pos, &binding, &holder)) {
        ADDOP_NAME(c, loc, LOAD_CLOSURE, holder, cellvars);
    }
    ADDOP_I(c, loc, BUILD_TUPLE, PyDict_GET_SIZE(captures));
    ADDOP_I(c, loc, CALL_INTRINSIC_1, INTRINSIC_RETAIN_ANNOTATION_CELLS);
    return SUCCESS;
}

''' +s[idx:]
a=s.index('                if (conditional_annotation_index != NULL) {',s.index('codegen_annassign(compiler *c'))
b=s.index('                if (captures != NULL) {',a)
# Move lookup before conditional registration and retain old cells before anything changes.
lookupstart=s.index('                PyObject *captures = _PyCompile_AnnotationCaptures',a)
lookup=s[lookupstart:b].replace('                    return ERROR;','                    Py_XDECREF(conditional_annotation_index); return ERROR;')
s=s[:lookupstart]+s[b:]
setup=lookup+'''                if (captures != NULL && codegen_retain_annotation_cells(c, loc, captures) < 0) {
                    Py_XDECREF(conditional_annotation_index); return ERROR;
                }
'''
s=s[:a]+setup+s[a:]
needle='                        ADDOP_NAME(c, loc, STORE_DEREF, holder, cellvars);\n                    }\n                }'
s=s.replace(needle,'                        ADDOP_NAME(c, loc, STORE_DEREF, holder, cellvars);\n                    }\n                    ADDOP(c, loc, POP_TOP); /* release old cells after full publication */\n                }',1);p.write_text(s)
p=Path('Include/internal/pycore_magic_number.h');s=p.read_text().replace('    Python 3.16a1 3709 (Checked deferred annotation closure capture)','    Python 3.16a1 3709 (Checked deferred annotation closure capture)\n    Python 3.16a1 3710 (Retain replaced annotation captures until publication)').replace('#define PYC_MAGIC_NUMBER 3709','#define PYC_MAGIC_NUMBER 3710');p.write_text(s)
p=Path('Lib/test/test_block_annotation_captures.py');s=p.read_text();idx=s.index('    def test_typing_consumers');s=s[:idx]+'''    def test_replaced_cell_finalizer_sees_complete_site_publication(self):
        module=execute("""\\
            events=[]
            class V:
                def __init__(self,n): self.n=n
                def __del__(self):
                    if self.n == 0: events.append(__annotate__(1))
            for i in range(2):
                x=V(i) # bind x y
                y=i
                value:(x.n,y)
        """)
        self.assertEqual(module.events,[{'value':(1,1)}])
        self.assertEqual(module.__annotate__(1),{'value':(1,1)})

''' +s[idx:];p.write_text(s)
PY
make -j1 > /tmp/block-annotations-build7.log 2>&1

/usr/bin/python3 - <<'PY'
import socket,os,array
for into in (False,True):
 a,b=socket.socketpair();fd=os.open(os.devnull,os.O_RDONLY);b.sendmsg([b'c'],[(socket.SOL_SOCKET,socket.SCM_RIGHTS,array.array('i',[fd]))]);print('fds',a.fileno(),b.fileno(),fd)
 if into:r=a.recvmsg_into([bytearray(1),bytearray(),bytearray(19)],socket.CMSG_SPACE(4),socket.MSG_PEEK|socket.MSG_DONTWAIT)
 else:r=a.recvmsg(20,socket.CMSG_SPACE(4),socket.MSG_PEEK|socket.MSG_DONTWAIT)
 print('peek',r,flush=True)
 rights=array.array('i',r[1][0][2]);print('rights',rights,flush=True)
 for f in rights:
  try: print('stat',os.fstat(f))
  except Exception as e: print('error',e)
 r=a.recvmsg(20,socket.CMSG_SPACE(4),socket.MSG_DONTWAIT);print('consumed',r,flush=True)
 a.close();b.close();os.close(fd)
PY

python3 - <<'PY'
from pathlib import Path
p=Path('agent-notes/live/block-with-bindings.md');s=p.read_text().replace('  has117/77/zero failures.', '  has117/77/zero failures; existing bundled Expat unreachable-fallthrough\n  warnings are the only optimized warnings.')
# Keep the final evidence readable without rewriting historical parent requirements.
for a,b in [('all328','all 328'),('pass31','pass 31'),('pass all328','pass all 328'),('pass,9.2s','pass, 9.2s'),('log`,210','log`, 210'),('log`,5,731','log`, 5,731'),('log`:18/18','log`: 18/18'),('pass,88','pass, 88'),('pass,90','pass, 90'),('skips,1m','skips, 1m'),('all6,280','all 6,280'),('has117','has 117')]: s=s.replace(a,b)
s += '\nKey source review: Python/symtable.c:603 (discovery), :3985 (sequential header traversal),\nPython/codegen.c:3488 (source-only exclusion), :5836 (publication), :5853 (normal/\nunwind reference cleanup), :5925/:6037 (enter-protected target setup).\n'
p.write_text(s)
for name in ('STATE.md','build-and-test.md'):
 p=Path('agent-notes/live')/name;s=p.read_text()
 for a,b in [('build:117','build: 117'),('paths.18','paths. 18'),('paths.\n18','paths.\n18'),('and210','and 210'),('pass5,731','pass 5,731'),('All6,280','All 6,280'),('failures,31','failures, 31'),('in\n115s','in\n115s'),('have117','have 117'),('paths.\n18','paths.\n18'),('18 focused,210','18 focused, 210'),('and5,731','and 5,731'),('all6,280','all 6,280')]: s=s.replace(a,b)
 p.write_text(s)
PY
git diff --check
git add -u
git add Lib/test/test_block_with_bindings.py agent-notes/done/asyncio-task-drivers.md
git diff --cached --stat
git commit -m 'Add explicit with and async with header bindings'

import subprocess
code = "const{readFileSync}=require('fs');const{strict:a}=require('assert');const c=require('C:/Users/mitsuhiko/AppData/Local/Temp/pi-clipboard-threads/win32-arm64.node');(async()=>{const p=c.getText();a.ok(p instanceof Promise);const saved=await p;const image=await c.getImage();if(image||saved===null){console.log('arm64 async text/image reads passed; preserving non-text clipboard');return}try{for(const text of ['café 日本語','', 'large'.repeat(200000)]){const p=c.setText(text);a.ok(p instanceof Promise);await p;a.equal(await c.getText(),text);a.equal(await c.getImage(),null)}console.log('Windows ARM64 async Unicode, empty, large text and empty image passed')}finally{await c.setText(saved)}})().catch(e=>{console.error(e);process.exitCode=1})"
subprocess.run(['prlctl', 'exec', 'Windows 11', '--current-user', 'C:\\Program Files\\nodejs\\node.exe', '-e', code], check=True)

import subprocess
code = "process.env.PSModulePath='C:/Windows/System32/WindowsPowerShell/v1.0/Modules';require('child_process').spawnSync('powershell.exe',['-NoProfile','-NonInteractive','-ExecutionPolicy','Bypass','-File','C:/Users/mitsuhiko/AppData/Local/Temp/pi-clipboard-threads/pi-clipboard-windows.ps1'],{stdio:'inherit'});console.log('completed')"
subprocess.run(['prlctl', 'exec', 'Windows 11', '--current-user', 'C:\\Program Files\\nodejs\\node.exe', '-e', code], check=True)

You can consider this amusing, but I have some questions here. The first problem with this is that it’s unreadable for a human. If you wanna follow along with what is going on, then good luck. Particularly once it opts out of using the edit tools that the harness provides, you’re going to have to resort to using the diff viewer of the final artifacts since it’s almost impossible to visualize the changes as they happen by reading the code.

This is not quite as bad in Pi for the most part because I mostly see it editing with the edit tool. When however goes all bananza with subagents (where the agent believes nobody is looking) it’s resorting to all kinds of increasingly bizarre behavior. I actually don’t know if the model thinks someone is looking, but that’s the vibe I’m getting.

But then it starts doing the same nonsense in code that actually gets committed. I have mostly seen this in tests, but you can also see this for instance when it writes JavaScript or CSS embedded in HTML. It almost seems like when it’s “one step removed” from regular code, it starts falling into these patterns.

Here are some unit tests that it created:

def test_unpack_suspension_and_continuation_close(self):
    from continuations import Continuation,suspend
    readers=[]
    class Source:
        def __iter__(self):
            yield 1
            suspend('unpacking')
            yield 2
    ns=execute('''
        def run():
            a,b='old-a','old-b'
            readers.append(lambda: (a,b))
            def a,b=Source()
            suspend('published')
    ''',Source=Source,readers=readers,suspend=suspend)
    with Continuation(ns['run']) as continuation:
        self.assertEqual(continuation.resume(),'unpacking')
        self.assertEqual(readers[0](),('old-a','old-b'))
        self.assertEqual(continuation.resume(),'published')
        self.assertEqual(readers[0](),(1,2))
    class Value:pass
    refs=[];frames=[];callbacks=[]
    ns=execute('''
        def run():
            for def x in [Value()]:
                refs.append(weakref.ref(x))
                frames.append(sys._getframe())
                callbacks.append(lambda: x)
                suspend('body')
    ''',Value=Value,refs=refs,frames=frames,callbacks=callbacks,weakref=weakref,sys=sys,suspend=suspend)
    with Continuation(ns['run']) as continuation:self.assertEqual(continuation.resume(),'body')
    self.assertNotIn('x',frames[0].f_locals)
    self.assertIsNotNone(refs[0]());callbacks.clear();self.assertIsNone(refs[0]())

def test_ast_roundtrips_and_future_annotation_unparse(self):
    source='callback=lambda {for def a, [b,*rest] in [(1,[2,3])] {return a,b,rest}}'
    tree=ast.parse(source);node=tree.body[0].value.body[0]
    self.assertIsInstance(node,ast.ForBinding)
    self.assertEqual(node._fields,('target','iter','body','orelse','type_comment'))
    self.assertEqual(node.lineno,1);self.assertGreater(node.end_col_offset,node.col_offset)
    self.assertEqual(ast.dump(tree),ast.dump(ast.parse(ast.unparse(tree))))
    ns=execute('from __future__ import annotations\ndef f(arg: '+source.split('=',1)[1]+'): pass')
    self.assertEqual(eval(ns['f'].__annotations__['arg'])(),(1,2,[3]))
    tree=ast.parse('async def f():\n async for def x in values: pass # type: ignored\n')
    self.assertIsInstance(tree.body[0].body[0],ast.AsyncForBinding)
    self.assertEqual(ast.dump(tree),ast.dump(ast.parse(ast.unparse(tree))))

So at least in some situations, the Python slop that it normally code-golfs for token-efficient tool calls leaks into the Python code it generates that should be stored. And well, it’s clearly more token efficient. The two unit tests above, when indented to the class structure they were in, are 10% more token efficient in this form than after a ruff format.

It’s AGI If You Don’t Look

I think there are a handful of things happening now that are pushing the whole thing in directions that are in conflict with one another. The training runs for these models are rapidly accelerating and they are now presumably also moving towards recursive self-improvement. The reward for the models is probably a combination of token efficiency, task completion rate and maybe some simple indicators like cyclomatic complexity. But we humans don’t think of code that is readable or understandable by simple, readily quantifiable metrics. All those things you can easily measure in isolation, and you can also optimize for them quite locally.

But these local optimizations do not produce global optimums, and the fewer of us are looking at the output, the less it matters. Obviously my software factory ran aground over the ~35 hours that it ran, but you can see the gradual regression towards insanity from the notes that it produced. For instance the task naming in the task file starts with an optimistic 1, 2, 3, 5, 5a but then eventually gets to 8a, 8a1, and then ends up with 8b2c2b3 and “8b2c2b2b checkpoint1”. The code that it produced got ever more wild. I don’t want to bore you with what it tried to build, but here are some example pieces of the interpreter changes:

static PyObject *
native_probe_run_impl(PyObject *callback, int sleep, int operation, PyObject *other)
{
    pthread_mutexattr_t attr;
    pthread_mutex_t mutex;
    pthread_mutexattr_init(&attr);
    pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_RECURSIVE);
    pthread_mutex_init(&mutex, &attr);
    pthread_mutexattr_destroy(&attr);
    pthread_mutex_lock(&mutex);
    int previous = native_sentinel;
    pthread_mutex_t *previous_mutex = native_mutex;
    native_sentinel = previous + 1;
    native_mutex = &mutex;
    PyThreadState *tstate = PyThreadState_Get();
    PyGILState_STATE gil = PyGILState_Ensure();
    int saved_errno = errno;
    PyObject *result = NULL;
    Py_ssize_t value;
    /* No intervening Python frame: these exercise ambient C provenance. */
    switch (operation) {
        case 0: result = PyObject_CallNoArgs(callback); break;
        case 1: result = PyNumber_Add(callback, other); break;
        case 2: result = PyNumber_Negative(callback); break;
        case 3: result = PyObject_RichCompare(callback, other, Py_LT); break;
        case 4:
            value = PyObject_IsTrue(callback);
            if (value >= 0) result = PyBool_FromLong(value);
            break;
        case 5:
            value = PyObject_Length(callback);
            if (value >= 0) result = PyLong_FromSsize_t(value);
            break;
        case 6: result = PyObject_GetIter(callback); break;
        case 7: result = PyIter_Next(callback); break;
        case 8: result = PyObject_GetItem(callback, other); break;
        /* ... */
        case 21:
            result = PyType_Type.tp_call(callback, other, NULL);
            break;
        case 22: case 23: case 24: case 25: case 26:
            result = conversion_probe(operation, callback); break;
        case 27: case 28: case 29:
            result = protocol_probe(operation, callback, other); break;
        case 30: case 31: case 32: case 33: case 34: case 35:
        case 36: case 37: case 38: case 39: case 40: case 41:
        case 42: case 43: case 44: case 45: case 46:
        case 47: case 48: case 49: case 50: case 51: case 52:
        case 53: case 54: case 55: case 56: case 57: case 58: case 59:
        case 60: case 61: case 62: case 63: case 64: case 65: case 66:
        case 67: case 68: case 69: case 70: case 71: case 72:
            result = collection_probe(operation, callback, other); break;
        default: PyErr_SetString(PyExc_ValueError, "bad probe operation");
    }

PyObject *info = PyTuple_Pack(3, name, mangled, suite->su_id);
PyObject *flags = PyLong_FromLong(DEF_LOCAL);
if (key == NULL || info == NULL || flags == NULL ||
    PyDict_SetItem(suite->su_bindings, mangled, key) < 0 ||
    PyDict_SetItem(st->st_cur->ste_block_bindings, key, info) < 0 ||
    (private && PyDict_SetItem(st->st_binding_info, key, info) < 0) ||
    (private && PyDict_SetItem(st->st_cur->ste_symbols, key, flags) < 0)) {
    Py_DECREF(mangled); Py_XDECREF(key); Py_XDECREF(info); Py_XDECREF(flags);
    goto error;
}
Py_DECREF(mangled); Py_DECREF(key); Py_DECREF(info); Py_DECREF(flags);

def _register_task(task):
    """Register an asyncio Task scheduled to run on an event loop."""
    _scheduled_tasks.add(task)
    if _task_accelerator is not None:
        _task_accelerator[6](task)


def _register_eager_task(task):
    """Register an asyncio Task about to be eagerly executed."""
    _eager_tasks.add(task)
    if _task_accelerator is not None:
        _task_accelerator[8](task)


def _enter_task(loop, task):
    if (_task_accelerator is not None and
            _task_accelerator[5]() is loop and loop not in _current_tasks):
        return _task_accelerator[1](loop, task)
    # ...

static int
apply_layout(tokenizeriterobject *it)
{
    PyObject *source = PyBytes_FromStringAndSize(it->tok->source.bytes, it->tok->source.len);
    if (source == NULL) return -1;
    PyObject *events = _PyPegen_tokenize_layout(PyBytes_AS_STRING(source), it->tok->filename);
    Py_DECREF(source);
    if (events == NULL) return -1;
    PyObject *result = PyList_New(0);
    if (result == NULL) { Py_DECREF(events); return -1; }
    Py_ssize_t index = 0;
    PyObject *first_pos = PyTuple_GET_ITEM(PyList_GET_ITEM(it->pending, 0), 2);
    PyObject *last_pos = PyTuple_GET_ITEM(PyList_GET_ITEM(it->pending, PyList_GET_SIZE(it->pending)-1), 2);
    PyObject *previous = NULL;
    for (Py_ssize_t i = 0; i < PyList_GET_SIZE(events); i++) {
        PyObject *event = PyList_GET_ITEM(events, i);
        if (previous && PyObject_RichCompareBool(previous, event, Py_EQ) == 1) continue;
        previous = event;
        PyObject *token = layout_token(it, event);
        if (token == NULL) goto error;
        if (token == Py_None) { Py_DECREF(token); continue; }
        PyObject *pos = PyTuple_GET_ITEM(token, 2);
        if (PyObject_RichCompareBool(pos, first_pos, Py_LE) == 1 ||
            PyObject_RichCompareBool(pos, last_pos, Py_GT) == 1) { Py_DECREF(token); continue; }
        while (index < PyList_GET_SIZE(it->pending)) {
            PyObject *old = PyList_GET_ITEM(it->pending, index);
            int cmp = PyObject_RichCompareBool(PyTuple_GET_ITEM(old, 2), pos, Py_LT);
            if (cmp < 0) { Py_DECREF(token); goto error; }
            if (!cmp) break;
            if (PyList_Append(result, old) < 0) { Py_DECREF(token); goto error; }
            index++;
        }
        if (index < PyList_GET_SIZE(it->pending)) {
            PyObject *old = PyList_GET_ITEM(it->pending, index);
            long kind = PyLong_AsLong(PyTuple_GET_ITEM(old, 0));
            if ((kind == NL || kind == NEWLINE || kind == INDENT || kind == DEDENT) &&
                PyObject_RichCompareBool(PyTuple_GET_ITEM(old, 2), pos, Py_EQ) == 1) index++;
        }
        if (PyList_Append(result, token) < 0) { Py_DECREF(token); goto error; }
        Py_DECREF(token);
    }
    for (; index < PyList_GET_SIZE(it->pending); index++) {
        if (PyList_Append(result, PyList_GET_ITEM(it->pending, index)) < 0) goto error;
    }
    Py_SETREF(it->pending, result);
    Py_DECREF(events);
    return 0;
error:
    Py_DECREF(events);
    Py_DECREF(result);
    return -1;
}

The failure case here seems somewhat obvious: the model is trained for token efficiency for tool calling which also looks like code, and sometimes it seems to be taking that code into a place where it should not be: the codebase.

35 Hours on a Single Prompt

I’m not really sure what to say here, but the slop machine was running for 35 hours until I turned it off. In that time it produced a net addition of 75k lines of code and it did not stop. In the 35 hours it burned around 1B tokens for a total of around 1200 USD in raw API costs. It managed to produce 79 commits, and that comes to a cost of around 15.5 USD per commit, and the agents exchanged around 1400 messages.

I honestly do not need an agent to run for 35 hours on a single prompt. It clearly does not work or result in reasonable outputs.

So obviously: prompting it like this is stupid. But when left unattended, it will keep going, and earlier models did not do that. Even Fable wasn’t as crazy as that. When you accidentally give it slightly too big of a task, it will continue until it succeeds, even if it burns through an entire subscription.

And that’s more or less why right now I do not manage to trust this model much. It has shown that it will commit slop, and it requires me to review it more as a result. Even if the failure rate is quite low, I would not want this.

Disposable Code vs Committed Code

In a world where code for tool calls is optimized for token efficiency and “getting the job done”, I wonder if there is really enough signal going to the training processes for “a human understands what is going on”. I would say that quite a lot of the code I get out of Astra is in my mind “objectively bad”. But it’s objectively bad by my human sense. Maybe it’s objectively good for a codebase that is entirely written by agents and only needs to be understood by agents.

Which is why I’m honestly asking myself more and more why we are doing this. These new models are absolutely amazing, for sure. But I’m more and more skeptical that the trajectory they are on still lends itself to present-day software engineering processes. The reason why I’m asking why we are doing this is because I felt like we achieved a pretty good spot for software engineering with those models, and that is the part of the AI economy where it was possible to show a positive return. But for how much more Fable costs, for how much more Astra costs, I do not feel like the results are there.

In fact, with Astra and Fable I feel like not only are the costs astronomical, but the models are also just not for me as a software engineer. And presumably that’s because these models increasingly are for other people. For lawyers, 3D artists, mathematicians, whoever uses computer use, etc.

And potentially as a byproduct of enabling all of this, you can now slop your way to a one-shot 3D game over the weekend which looks impressive. And probably you can now run a software factory for as long as you don’t care about the code.

I’m sure I will get used to this, but man this stuff is weird.

Long story short: I had a conversation with Fable and Sol via Pi about what could be done with such a dongle or whether I should use a Raspberry Pi instead if I wanted to do my own thing there. I figured it might be quite fun to run my own code while still allowing regular CarPlay to pass through.

Through working with the LLM I learned about CatPlay, which is a Rust reimplementation of the CarPlay protocol that can run on Carlinkit devices. In particular, it can run on the Carlinkit Mini Ultra, which I figured would be easy enough to buy. I do have a few CarPlay adapters around, but I did not have that particular model, so I bought one on Amazon. Twenty-four hours later, I had a device in my hand that was branded as a Carlinkit Mini Ultra, but instead of being the Ingenic device that the original author used, it turned out to be something else.

This is normally where the story would stop. However, it’s 2026. Armed with a bit of knowledge about how these systems work, I managed to have some fruitful discussions with Kimi K3 and Sol and figure out how flash the device and in turn, how to make CatPlay compile for that SoC.

I guess that hacking these USB devices is not necessarily hard, but it’s laborious and you can easily end up bricking your devices. It also just sucks because sometimes you need to work with someone else’s code that does not itself run on your machine. In the past, I would abandon many such projects for lack of tenacity. But my clanker is tenacious.

But so are all of our clankers. Some of the projects we’re now attempting are happening because of conversations we have with them. In this case I did not find or decide on CatPlay, the model did. It was not the only suggestion, but it became the best starting point after discarding others.

And I discover this more and more. Particularly when we have solitary interactions with these models, some of us “independently” decide to work on similar projects. When I talked with an acquaintance about CarPlay he also mentioned recently that he decided to try something similar because he too wanted to see if he can get his own agent be hooked up with the car. And guess what: he too learned about the CarPlay hacking community, and that it’s an option, from the models and roughly around the same time.

It really got me thinking about how this could create situations in which completely independent people end up building things they believe are their own ideas. Yet they were inspired or pushed towards doing something by a conversation with an LLM — a conversation that someone else also had. What if we took paths, because those were the paths that were more likely with current generation models? There is a running joke in the AI builder community right now that we’re all working on the same things, and in many ways it feels like we are. That might be because those things are obvious, or it might be partly because we all use the same models with the same capabilities.

A few months ago, I first saw Lucas Meijer share the idea to make a model in Pi produce HTML reports rather than Markdown. I thought that was pretty unique. Except, well turns out the models are probably trained more and more for that (e.g. Claude Artifacts), and now it has become for many the default choice for sharing reports.

How much of what we build comes from eliciting the same latent capabilities from the same models? Did the models make us prompt them that way? Was it because we shared ideas on Twitter and other communities that inspired us? Or is it all unrelated?

There is something powerful and strange about how LLMs diffuse knowledge and capabilities, while perhaps also nudging us all simultaniously and independently toward building the same things.

In the discussion around that topic, one of the most upvoted comments on the Lobsters thread asked a question I had to think about quite a bit:

How can you work in tech right now and not be angry?

In the context of the thread, this was clearly also about AI and agents. For me, the emotions I would expect in tech vis-a-vis these new developments are disorientation and anxiety, but not anger.

Anxiety as an emotion does not require someone to blame. Right now, I find it reasonable to feel anxious about an uncertain future. Who knows what our professions will turn into and what kind of world my kids will find themselves in when they enter the workplace? And if you’ve been in the industry for a long time, will the skills you’ve spent years acquiring still matter?

But anger is different from anxiety because anger needs to be directed somewhere. The feeling of anger suggests that somebody or something is doing something to you.

Who are you going to be angry at and why are you angry in the first place? One narrative that is pretty pervasive is that if AI will usher in productivity gains, those gains are going to benefit companies rather than employees. And well at least someone at Meta wants that. Yet I also find that plenty of people in leadership positions express doubt about AI. They see that an increasing share of their costs is being funneled directly to some large AI labs. They express worries about what will happen to their data and whether these large companies will step into their space instead of being partners.

My answer to the question of how you can not be angry in tech is that it’s by no way the most only possible feeling. First of all, instead of being angry, you can simply be unsure. The feeling of uncertainty is a much more productive emotional state because it can lead to curiosity. Even if you don’t find what’s happening right now exciting, you can at least find it interesting. We have access to magic machines, and we can poke at them and see what happens. The second way is to feel genuine excitement. Once you move beyond curiosity, you can come away with a newfound feeling of power and freedom. A lot of the gains from AI aren’t turning into productivity gains that are reflected in company profits but they’re showing up instead in the number of side projects shipped by everybody not on their company’s time.

The fact that this is happening shows us that owners and founders don’t necessarily know what will happen. Ownership comes with agency, but it does not provide foresight, and this change is disorienting for everybody. I engage with plenty of people who project confidence in public and are much less certain in private. Many of them are placing bets, but they are talking with confidence about those bets, trying to keep their business afloat while the ground moves under them. They experience that uncertainty from a position where they can act on it, and they are often standing somewhere with a megaphone to get others on their side to improve their odds.

I feel that contradiction myself: I am simultaneously tremendously excited, but I am also unsure what will happen next. I do not know what it will mean to be a programmer in the future, and, as the owner of a company, I am also not sure where the high ground will be when this all settles. Much of what I learned over the years is changing rapidly, including ideas I considered fundamental to my craft and business. Some days that feels liberating, but on others I wake up feeling like the ground is crumbling beneath me.

Anxiety is an uncomfortable emotion because it acknowledges that you do not know what will happen and might not be able to stop it. On the other hand, anger can feel more actionable because, instead of saying “I don’t know,” you already have someone to blame. It turns a loss of control into a comforting story with a villain. But I feel that particularly when it comes to AI, it’s easy to pick the wrong villain because of how disruptive the change is for everyone. Your engineering manager or leadership team might themselves feel uncertain about their future and just try to bolster their own confidence by projecting clarity and certainty.

That does not mean there are no villains. When this all plays out, some will profit and many will not. I’m afraid we’re completely ignoring the impact this has on society at large, the climate, and the balance of the world as a whole. As excited as I am about the technology, I worry about Europe’s lack of ambition and growing dependence on other countries. I have a lot of complex thoughts about what we’re doing as an industry right now.

I don’t know what the future of this industry will look like, and I don’t know who will benefit from it and I don’t think I’m alone with that. However I can only urge anyone who feels anger and looks for a villain right to instead remain curious instead. To be curious enough to understand what is changing, excited enough to experiment with it. And then, from what we learn, earn the right to decide when resistance is warranted and where to direct it.

As a result, LLMs make language choice much less consequential than it used to be. If you don’t like the choice, you can seemingly rewrite it in another language and you can make it pick a language that you, as a programmer, are entirely unfamiliar with.

Which in turn means that people can, and do, choose based on the marketing of languages much more. As a long-term Rust programmer I found it quite fascinating to see people now ship Rust code who previously might not have chosen it. I attribute at least one part of this to two recent vibe shifts: there is a lot more talk about wanting fast software, and about LLMs being exceptional at optimizing code without regressing behavior.

Folks like Mitchell Hashimoto, Charlie Marsh, Jarred Sumner, Daniel Lemire and quite a few others always carried a certain level of obsession with fast and performant software and they also all happen to be receptive to agents writing code. Maybe as a result, or unrelated others are now joining in. That’s because with things like autoresearch you don’t even necessarily need to know all the tricks: you just need to put an agent on it — though knowledge greatly helps!

If you look around, there are plenty of projects that want to be fast and small, and they increasingly pick “hard languages”. And it’s not just Rust that is benefiting. Even Zig — despite the fact that the creators and parts of the core community are pretty negative on the whole AI thing — is too. For instance Cloudflare’s new Artifacts service uses a pure-Zig Git-protocol engine, compiled to a roughly 100 KB WebAssembly module and Vercel released fx, a Zig coding agent advertised to be small and fast. From what I can tell, all these projects are largely LLM-assisted.

But it’s not just people picking less common languages but also that they are increasingly working with “much harder” technologies. All of a sudden I have seen people do some really impressive stuff with DWARF files, eBPF, custom network drivers, custom crypto and really old computing hardware. Many of these things were previously off-limits for lots of developers. In some cases (eg: crypto) you were even pushed away because those things were intentionally gatekept by the people in the know.

So maybe the world will have more slop, but it might also have more developers in it, that want things to be fast and small.

Hiding Traces

Reasoning traces are usually hidden from us. We have lamented this, but mostly have to accept it. Open-weight models thankfully reveal them, and from their behavior you can see that their traces can be long and confusing. This is probably a good reason to separate them from what is normally shown to users.

At minimum, UIs need to detect them. The industry has done a good job at making reasoning traces sound special and exotic, but they really are just text: the model is trained to emit its thinking into a scratchpad as part of its response, before its final answer.

GPT-OSS’s Harmony response format makes this easy to see:

<|channel|>analysis<|message|>
I need to work this out ...
<|end|><|start|>assistant<|channel|>final<|message|>
The answer is ...
<|return|>

The markers are special tokens, but the reasoning between them uses “the same text” as the final answer (just that GPT chain-of-thought text sounds really funny). When the model samples the analysis channel token, a parser routes the following text into a separate stream exposed through the Responses API. For closed models, presumably a simple model redacts and summarizes it.

Reasoning Effort

How much budget goes to reasoning? Earlier APIs exposed reasoning token budgets, making it seem like a property of the sampling process. In reality, reasoning effort is baked into the system prompt. GPT-OSS puts this into the system prompt:

Reasoning: low

That’s it. Training produces the resulting behavior, such as emitting the token sequence that switches to the analysis channel. This also explains why changing the effort invalidates the KV cache. I think closed GPT models call reasoning effort “juice,” since you can ask most models how much juice they have.

In DwarfStar for DeepSeek with max reasoning this is added to the system prompt:

Reasoning Effort: Absolute maximum with no shortcuts permitted.
You MUST be very thorough in your thinking and comprehensively decompose the
problem to resolve the root cause, rigorously stress-testing your logic against
all potential paths, edge cases, and adversarial scenarios.

Don’t Think

The destination of reasoning tokens is therefore a learned convention: the model is trained to keep scratch work out of the final channel. Trick it into thinking it is in that channel and it may leak tokens. We have even seen older models, when thinking is disabled, reason into the bash tool and echo their thoughts to /dev/null.

So in some sense the only “special” behavior for some models is not to think. That at times is done by “mechanically” removing the model’s usual ways to think. In DwarfStar, disabled thinking uses the prefill </think>, while enabled thinking uses <think>, which are the tokens that close and start thinking. GPT-OSS doesn’t prefill but lets the model decide either way on its own.

But presumably, some inference APIs prefill the opening token when reasoning is enabled, so the model never samples it itself and might prevent the sampling of the reasoning token when disabled since it can be trivially detected. This may explain why a custom think tool can trick models into putting some reasoning where it should not go — but only when native reasoning is disabled.

Fun fact: this blog post triggered safey checks

Hilariously enough I was unable to use GPT 5.6 terra for spell and grammar checking on this blog post because of safety filters. Had to switch to Kimi.

GitHub’s governance has never been democratic and there is plenty about the platform that I dislike. Yet I do not need my infrastructure to be democratic but I need it to be predictable and reasonably neutral towards the Open Source software hosted on it. A democratic provider without a clear constitution can be worse at those things than a corporation is.

Codeberg is entirely within its rights to do run the platform like they want. It is a German association with members and a democratic process, and that process produced a result. But a democratic vote says nothing about whether the decision is a good one, particularly for the people already depending on the platform. A majority can still decide that certain projects and people no longer belong.

The new terms prohibit projects that mostly consist of code written by generative AI tools. That’s fine, but these days I could not assign authorship percentages to my recent projects. For me this rule is quite vague and I would bet that it makes it hard to enforce. In practice my assumptoin is that the center will leave.

If anything a harsher line would probably be preferable. If Codeberg wants no LLM involvement, it should say so. On the other hand if the objection is just spam and abusive resource consumption, it should write rules for those instead. Now it defers the details of the policy to moderators and the communit which already draws a much harder boundary than the text does, judging by the tone of the discussion around it.

It is a shame that the Open Source and Free Software communities are splitting this deeply over LLMs and agents. These systems have problems, but these tools are also becoming part of how software is made. The Open Source world needs to figure out how to engage with that future. More importantly, LLMs if used well, should be welcome to the Open Source community as they can be used to reclaim control and take power away from large corporations.

I want GitHub to face true competition in the Open Source space. I would particularly like some of it to come from associations rather than another large entity. As a European project, Codeberg naturally matters to me even more. I just wish Codeberg were more forward-looking here and be willing to host the Open Source software of tomorrow and not only software approved by the community today.

The text begins with a technology upgrade:

And they said one to another, Go to, let us make brick, and burn them thoroughly. And they had brick for stone, and slime had they for morter.

They use it for a civilizational project:

let us build us a city and a tower, whose top may reach unto heaven

But when God assesses the situation the bricks are not what concern him:

the people is one, and they have all one language, […] and now nothing will be restrained from them.¹

They get their power through coordination which they have because they share a language. They can use this to coordination to combine their powers and build something no one of them could build alone. God does not take away the bricks or their knowledge of how to make them but their ability to understand one another.

With AI-assisted programming we should get better tools which lets us build more ambitious software. That is certainly true at the level of the individual and without doubt a developer with an agent can change a codebase dramatically quicker. But large software projects have never been limited only by how quickly an individual can produce code but they are limited by how well people can coordinate their understanding of the system they are changing.

The shared language of a software project is the common understanding shared among its developers. This language is rarely written down in one place but it lives in documentation and code. It can also just be something that comes up in code review or watercooler conversations or when one engineer has to explain a change to someone else. It can be about the architecture of the code, the tradeoffs made or which invariants need to be upheld.

In the days before agents some of that shared understanding was maintained by friction. If I wanted to change someone’s storage layer, I usually had to read their code and ask them questions. Changing that code might have required coordination with another team whose service depended on it. Some of this friction was useful as it forced communication. It also was a good touchpoint for both of us discover if we still understood how the system worked. I had plenty of experiences in my career where through conversations with my fellow engineer we collectively had to understand again why the system we had worked as it did before we did some changes to it.

But with agents I can ask an agent to add OAuth, you can ask one to add caching, and somebody else can ask one to rebuild the database from first principles and make the UI pink. Each change can be reasonable in isolation but since it’s frictionless, none of us necessarily has to talk to the others or familiarize ourselves with the code we are changing. The more we use agents, the less we feel the pain as agents feel none of it, and a useful signal is gone.

When I look at some vibecoded scaled-up projects the codebases mirror the story of Babel mostly because nobody needs to communicate. Obviously nothing stops us to talk to one another, but nothing forces us either. Every developer has a tireless machine that can explain a corner of the tower and make whatever local modification they want.

Unlike in the bible though, in AI-assisted engineering, construction can continue after shared understanding has already collapsed. The complete lack of an immediate failure is what makes it curious and a bit disorienting. The tower does not fall, it just keeps rising.

That alone is not too surprising as models emit malformed tool calls sometimes. Particularly small ones. What surprised me is that this is getting worse with newer Anthropic models as both Opus 4.8 and Sonnet 5 show it but none of the older models. In other words, the SOTA models of the family are worse at this specific tool schema than their older siblings.

In case you are curious about Fable: I intentionally did not test it because I was not sure if the classifiers they are running might downgrade me to Opus silently.

Tool Calls Are Text

If you have not spent too much time looking at LLM tool calling internals, the important thing to understand is that tool calls are not magic and use some rather crude in-band signalling. The model receives a transcript, a system prompt and a list of available tools. The server munches that into a large prompt with special marker tokens. Because the model was trained and reinforced on examples of that format, at some point during generation it emits something that the API or client interprets as “call this tool with these arguments”.

For a file edit tool, the intended invocation payload might say something like this:

{
  "path": "some/file.py",
  "edits": [
    {
      "oldText": "text to replace",
      "newText": "replacement text"
    }
  ]
}

A harness then validates the arguments, performs the edit, and feeds the result back into the model. If validation fails, the model sees an error and usually tries again.

How exactly that formatting happens is not known for the Anthropic models, but some people have gotten out “ANTML” markers and they at times do leak also into public communications. To the best of my knowledge, the call above would come out serialized like this from the model:

<antml:function_calls>
  <antml:invoke name="edit">
    <antml:parameter name="path">some/file.py</antml:parameter>
    <antml:parameter name="edits">
[
  {
    "oldText": "text to replace",
    "newText": "replacement text"
  }
]
    </antml:parameter>
  </antml:invoke>
</antml:function_calls>

An important thing to note here is that this thing, while looking like XML, is not really XML. It’s just a thing they found convenient to tokenize and train on. The other thing to note is that a basic top-level string parameter appears in-line whereas an array of objects is implemented via JSON serialization. While I’m not entirely sure that this is how it works, there are some indications that this is not too far off. This will become relevant later.

There are two very different ways to make the model produce a structure like this:

1. You can ask the model to produce valid JSON matching a schema and then validate it afterwards.
2. You can constrain the sampler so that invalid JSON, or even invalid schema shapes, cannot be sampled in the first place.

The second approach is what people usually refer to as grammar-aware or constrained decoding. The sampler masks out tokens that would violate the grammar. If the model is currently inside a JSON object and the schema says only oldText and newText are allowed, the sampler can prevent it from emitting "in_file" or "type". Grammar-aware decoding can be used both to constrain something to be syntactically valid JSON and also to enforce specific enum values or keys.

Without any form of constraints the model is merely following a learned convention.

The Failure

Pi’s edit tool supports multiple exact string replacements in one call. That is why the arguments contain an edits array. In the failing cases the model produces entries like this:

{
  "oldText": "...",
  "newText": "...",
  "requireUnique": true
}

or this:

{
  "oldText": "...",
  "newText": "...",
  "oldText2": "",
  "newText2": ""
}

Across repeated trials I saw a whole zoo of invented trailing keys: type, id, kind, unique, requireUnique, matchCase, in_file, forceMatchCount, children, notes, cost, oldText2, newText2, oldText_2, newText_2, and even an event.0.additionalProperties key inside the edit object itself.

The most annoying part is that the actual oldText and newText payloads were byte-correct in the invalid calls I inspected. The model had in fact produced the right invocation but then added nonsense at the end of the object.

The failure is also heavily context-dependent. A fresh single-turn prompt like “edit this file” did not reproduce it at all for me. An agentic history where the model had read files, diagnosed a problem and then composed a multi-line edit could reproduce it. And more annoyingly, not all transcripts will show that behavior. In fact, I needed Petr Baudis‘s transcripts to reproduce this for me at all! In that user’s session continuing the session caused Opus 4.8 to fail around 20% of the time. Stripping thinking blocks from history reduced the failure rate by half. Turning on strict tool invocation eliminated it in my runs.

Why It’s Getting Worse

My strongest hypothesis is that this is not random deterioration but a training artifact.

When older Anthropic models were trained, they were trained on some tools (some of which were documented). But that training did not yet have a user-shipped harness like Claude Code as the obvious target. Modern Anthropic models are most likely different because their post-training includes Claude Code or a harness that looks very similar. The model learns what a successful tool call looks like in that environment. It also learns what mistakes are tolerated by that environment.

Claude Code’s own tools are comparatively flat. The ordinary edit tool is not Pi’s nested edits[] shape; it is closer to file_path, old_string, new_string, and an optional flag (replace_all). Looking at Claude Code’s client is very instructive: it contains retry paths for malformed tool use, parameter aliases, type coercions, Unicode repairs and filtering of unknown keys. In other words, Anthropic’s own client appears to expect and accept a fair amount of slop and repairs it, mostly silently.

If reinforcement learning happens in a harness like that, or a simulation of one, then slightly malformed tool calls can still complete the task and receive reward. The harness fully absorbs the error and there is little gradient against inventing an alias, adding a stray field or using a nearby parameter name.

Worse, the model may become very strongly adapted to the canonical Claude Code edit tool shape. A different harness can present a tool with the same semantic intent but a different schema. Such a tool can increasingly be off-distribution. The better-trained model might actually fight you harder because its prior is stronger.

This is not too surprising, but it is a change from how this was a few months ago. When Opus 4.5 launched, it adapted to other edit tools exceptionally well. In fact, I was pretty convinced that we’re on a good path where the models are more likely to adapt to any sort of tool shape that comes around for as long as the instructions are good.

Now I’m somewhat worried about the track we’re on here. Alternative tool schemas might not just be unfamiliar. They might be implicitly punished by post-training that optimizes for one particular, forgiving tool ecology. And that ecology is not documented. While there is a text editor tool that is documented, you will see that this format is in fact not followed by Claude Code. What Claude Code does internally (which is a closed-source harness) is hidden from you.

The Slop Harness

Claude Code is obviously closed-source but we can look at the minified code and get some idea of what it does. And honestly, it’s very forgiving of incoming data.

For a start, Claude Code checks the model’s visible text for leaked <invoke markup. It also emits some telemetry when that happens and then it has its own state machine to retry such bad calls by pushing back to the model.

It has explicit Unicode escape repair which fixes broken \uXXXX sequences and lone surrogates in string values. It also has per-tool aliases for parameters. For instance, Edit accepts old_str (presumably from the times when the models were trained on the officially documented text editor tool), the newer old_string from the schema, new_str/new_string, path as an alias for file_path, and some more.

It also silently filters out unexpected keys and it does not use strict mode either. The issue with strict mode is that Anthropic applies complexity limits to the tool definitions that cause API requests to fail, so presumably that’s why Claude Code does not attempt to use it.

Strictness

Will this problem be with us in other harnesses too? One huge issue with Anthropic is that the models are completely closed, and so is the harness. Codex models are also closed, but at least the harness is not. We also have gpt-oss which is at least a bit interesting. The models are explicitly trained to use OpenAI’s harmony response format and there is a lot of documentation that at least tells us how OpenAI people think about this.

Harmony makes channels and tool-call content types part of the prompt format. A function call can look like this:

<|start|>assistant<|channel|>commentary to=functions.get_weather
<|constrain|>json<|message|>{"location":"San Francisco"}<|call|>

The important bit is <|constrain|>json. The model can express in-band that this message body is JSON, and an inference stack can use that boundary to switch into JSON-constrained sampling for the body of the tool call. Presumably a bit of this also happens in Anthropic’s models, at least in strict mode I would imagine.

The marker in harmony helps the sampler to detect when it needs to sample with a specific grammar, and because it is part of the transcript, it makes that rather easy to do. For hosted GPT models, there is also an option to provide a LARK grammar for custom tools that need to adhere to something like this.

Anthropic appears different from that, though maybe not entirely. If an array of objects is represented as JSON, as it appears to be, then the model has to write JSON inside the tool parameter. There is probably basic grammar-constrained sampling going on, and that may partly explain the extra keys. For a nested array parameter, that JSON includes escaped multi-line file content inside string literals, inside one tag. The unexpected, made-up keys appear exactly at the highest-entropy point of that task: after closing a several-hundred-token escaped newText string, where the model must decide } vs , "...".

Opus 4.8 and Sonnet 5 seem to have much stronger priors about what an edit tool call should look like and that prior appears to be Claude Code’s edit schema: a flat old/new string pair, plus the optional replace_all flag. My guess is that Opus has learned that an edit operation may have one extra optional field, but under Pi’s nested oldText/newText shape it has no trained name for that field. So it samples a plausible name fresh each time, which is why the failures produce dozens of random keys rather than one stable alias.

As strict mode in Anthropic appears to fix this, I presume that on the server side they are refusing to sample a key that is not permitted by the JSON schema structure. That would also explain why they have limits to the complexity of the tool definitions when strict mode is enabled.

So far, the Codex models I tested did not show this type of regression. I tested all available ones except 5.6, which I do not have access to yet.

What This Means For Harnesses

The uncomfortable lesson is that tool schemas are not neutral, at least not on Anthropic models. We like to pretend that a schema is an abstract contract and the model is a general reasoner that will follow it, but that might no longer be the case for some of the tools.

Tool schemas are somewhere in the distribution and some shapes are close to what the model saw during post-training and some are far away. Some are easy for the provider’s hidden encoding (e.g. top-level attributes in ANTML), whereas some require the model to write large escaped JSON objects inside nested arrays after long multiline strings. The model may be smart enough to understand the schema and still be bad at sampling the exact shape under pressure.

If this type of model behavior continues, I wonder what the implications for harnesses are. Obviously one could turn on strict sampling in Anthropic and the problem should go away. On the other hand, that the model has this behavior shows the impact that reinforcement learning has on them. Fighting that prior is probably futile if you want to get the best model performance.

Right now the reality is that Claude Code is not open source and we cannot really know what they are doing in their RL environments either. We cannot assume Claude-Code-trained behavior will transfer cleanly to your tools unless they are a close match. The more post-training happens inside one dominant harness, the more every other harness will have to inherit its quirks.

I used to be more skeptical of strict grammar-constrained tool invocation because constrained decoding can have quality tradeoffs. I still think that can be true in general, but this bug moved my priors significantly. If the newest models get better at solving the task while getting worse at faithfully emitting an alternative tool schema, then the harness needs stronger guarantees somewhere.

If you want to find out more, or you want to discuss this, consider reading the issue on the Pi tracker.