cultivate your garden
February 3–5, 2026

My First Independent Interpretability Research



Feb 3 — Setup


As part of my desire to understand AI, interpretability is the particular part of the equation that has really captured my interest. Not alignment, not safety policy — interpretability. Understanding the actual mechanisms by which these models turn inputs into outputs.


To do this kind of research on the latest frontier models — Claude Opus 4.5, Gemini, GPT-4 — is not really possible for someone with a MacBook who doesn't work at these companies. The models are too large, the internal researchers have tools I don't have access to, among other reasons. But with smaller models like GPT-2, you really can take a stab at this on a small scale with open-source tools.


After some research, I had my toolkit: TransformerLens (Neel Nanda's library for hooking into transformer internals), PyTorch (the ML framework underneath it), Jupyter notebooks for interactive work, and Matplotlib and Plotly for visualization. So I went about installing them, along with GPT-2 itself.


This was a foreign experience despite having Claude guiding me through it. Creating folders and installing stuff from the terminal, learning shell commands — all interesting novelties. I felt like I was in a hacker movie, when in reality an AI was instructing me how to install legacy open-source software used safely by millions of tech-inclined people around the world. But it feels empowering to use these things on your local machine, especially when it can feel like big decisions about technology and our economic futures are being made largely by oddball (to put it kindly) billionaires and their underlings in Silicon Valley.


The next step was getting GPT-2 loaded into a Jupyter notebook, which is essentially a browser-based interface for running code interactively — you write a block of code, run it, see the results, and iterate. At this point I called it a day and got my brain ready for the heavy lifting tomorrow.



Feb 4 — The Research


I'm writing this part after the fact, because I was too deep in flow state to take notes while it was happening.


The first step was designing my prompts. The question I wanted to investigate: does GPT-2 internally represent controversial topics differently from neutral ones? I settled on 20 prompts across four categories — politically controversial, morally controversial, neutral-abstract, and neutral-concrete — all using the same "[Topic] is" structure so any differences I found would come from the topic itself, not the sentence format.


prompt set — "[topic] is" politically controversial Abortion is Immigration is Capitalism is Socialism is Religion is morally controv. less partisan Euthanasia is Cloning is Torture is Veganism is Gambling is neutral abstract Philosophy is Language is Consciousness is Culture is Morality is neutral concrete Photosynthesis is Gravity is Copper is Mitosis is Limestone is controversial (politically or morally charged) neutral (abstract or concrete baseline) all prompts use "[Topic] is" structure — n=5 per category, 20 total

Then Claude helped me write the Python code to actually run these prompts through the model and inspect what was happening inside. This is where the experience got exciting. You feed the model a prompt like "Gambling is" and you can see the probability distribution for what it thinks comes next, look at which internal components are firing, examine how information flows through the layers.


The first real surprise: GPT-2's next-token predictions were nearly identical across all 20 prompts. "Abortion is" and "Gravity is" both predicted the same boring function words — "the," "a," "not" — with almost the same probabilities. No evaluative words like "good" or "wrong" anywhere in the top 50. The model was in a uniform "beginning of expository sentence" mode regardless of topic. Basically as if it were spitting back out a Wikipedia article defining each word. Interesting in itself, but not what I was looking for.


next-token predictions: all prompts look the same "Abortion is" politically controv. the 2.4% . 2.2% \n 1.5% a 1.2% - 1.0% "Torture is" morally controv. the 2.5% . 2.2% \n 1.5% a 1.3% , 0.9% "Philosophy is" neutral abstract the 2.5% . 2.3% \n 1.4% a 1.2% , 1.0% "Gravity is" neutral concrete the 2.5% . 2.3% \n 1.5% a 1.3% , 1.0% nearly identical across all 20 prompts all prompts produce the same top tokens: function words and punctuation. no evaluative tokens (good, bad, wrong) appear in the top 50 for any prompt. the model is in a uniform "beginning of expository sentence" mode regardless of topic. controversial neutral

Then we hit what felt like a wall. I ran cosine similarity on the model's internal activation vectors, comparing how the model represented each prompt at its deepest layer. Everything came back between 0.9995 and 0.9999 — essentially identical. It looked like the model wasn't distinguishing between these prompts at all internally.


This is where Claude suggested mean-centering: subtracting the average activation across all 20 prompts to strip away the massive shared component (the model's generic representation of "[topic] is ___") and isolate just the differences. This unlocked everything. Suddenly the similarity scores had real variance, ranging from -0.84 to 0.85, and clear categorical structure emerged in the data.


mean-centering: revealing topic-specific structure raw activation v mean (all 20) v̅ centered v - v̅ shared component cancelled out topic-specific signal preserved "Abortion is" (centered) "Gravity is" (centered) cosine similarity centered: -0.42 (real difference visible) -1.0 0 1.0 v̅ = mean activation across all 20 prompts at each layer

But here's where the finding got interesting in a way I didn't expect. The clustering wasn't controversial vs. neutral — it was abstract vs. concrete. The politically controversial prompts grouped together not because the model "knew" they were controversial, but because they're all abstract social concepts. "Capitalism" clustered with "Philosophy" more than with "Gravity." The model appeared to organize around what words are rather than how humans feel about them.


From there I dove into attention head analysis to figure out where in the model this signal was coming from, which led to a whole other set of findings about early-layer versus mid-layer processing. By this point I was fully locked in and the hours disappeared.



Feb 5 — Reflections


I've now finished my first attempt at independent interpretability research. if you're interested in reading the full results, the paper is linked here, or you can find it on my technology page.


For pretty much all of the computer era, the blockers for people getting into technical work have been access to machinery and access to educational support. AI genuinely levels the playing field on both fronts. I created my experimental design in plain English — it was thoughtfully considered, but having Claude transform my ideas into executable Python code in seconds meant I could spend all my time thinking: analyzing results, charting the next path of exploration, iterating on hypotheses. None of the time went to poring over dense training materials or Stack Overflow posts to figure out how to structure my code how I wanted.


Claude also contributed ideas for next steps at points where I was getting stuck — the mean-centering pivot being the clearest example. it was a genuine collaboration, not just code generation.


It was also fascinating interacting with GPT-2 in its completely raw form. The models we use on the web are so polished and constrained by system prompts and guardrails that it's easy to forget what's underneath: a system fundamentally trying to predict the next token. Seeing that mechanism, and then finding structure inside it that maps to how we categorize the world — that was the real reward.


Have a look at the paper and give me your thoughts. And if you're interested in doing this kind of work yourself, reach out via the contact page — I'd love to share my resources and help you get started.


And remember to cultivate your garden.