Note: Please note that this isn’t meant to be a valid neuroscience or research, rather it is my somewhat crude attempt to understand how the brain functions from my perspective, and try to create a model that mathematically falsifiable to test that understanding.

TLDR; How does a system end up giving an output different from what we intended? We started by perturbing the input, and while we got results, it wasn’t quite the right way to approach our question. We even saw the cat and car representations come closer, but that didn’t mean the cat was becoming a car or that we got our desired prediction. So, what if we perturb the system instead? We moved to internal activation perturbation, keeping the input & model weights fixed, and saw promising results; the next-token prediction changed, and introducing noise during text generation changed the completion that followed.

Introduction

In December 2022, I wrote an article titled “An Engineer’s Point of View on the Brain” where I tried to understand and conceptualize cognitive processes, information storage and retrieval, ideal and non-ideal states, and more using computational models. I concluded the article on the note that the brain is an information dependent undecipherable black box that is faster than we’d like to admit; with two model. A trie which speaks to the interconnectedness of the brain, and the disk, RAM and cache model which speaks to how the brain might store and retrieve information.

These models were an attempt to find an engineering analogue for the human brain. To take this further, I wanted a way to test some of what I said in that earlier post. The idea is to perturb an input and see how the learned representation and prediction change, then keep the input fixed and see what happens when we perturb the system instead. This gives us something we can measure while exploring the analogy.

Motivation

The human brain is a compact biological computer that has inspired research & technological advancements alike, but regardless, it is so easy for the brain to be prone to errors, we can all agree that “I’d like to park my cat” makes no sense, but for some weird reason, it slipped out when you intended to say “I’d like to park my car”. The Medium story I wrote tried to investigate this by proposing models, mainly aimed at understanding where the information storage and retrieval process went wrong that led to the unwanted sequence of words.

The post builds upon those models, but this time I wanted to run an experiment; introduce the noise, see what changes, and go from there.

Review

Unlike the “Disk, RAM, and Cache” model, the “Huge Trie-List” model introduces something quite interesting, an ideal and non-ideal (noisy) state. The ideal state says the brain is stimuli dependent function \(f(X)\), and for every set of stimuli \((X)\) there’s a corresponding set of outputs \((Z)\), and I defined it as;

This is fine if we treat the brain like a pure mathematical function; but as established in the earlier post, just like any biological, mechanical, electronic, or electro-mechanical system, there is “noise” (internal & external noise). What happens when we take them into account; \(X, X{\phi}, X{\rho}\); \(X{\phi} = \text{internal noise}, X{\rho} = \text{external noise}\); \(f(X)\) is now given by;

My starting intuition was that unwanted stimuli could still affect a system’s output. That is the part I want to explore here. It doesn’t establish that the brain processes every stimulus alike, or that it never filters noise.

Experiment Setup

For the first part of this experiment and ease of visualization, I’ll be using images with dimensions, 512 by 512 by 4, i.e. a 512 by 512 image with 4-channels (RGBA), that will be converted to 3-channels (RGB). In the second part, we keep the input fixed and perturb internal activations, starting with an image classifier before moving to a small language model.

We Need a Brain

To test this, we need a stand-in for the brain. Don’t worry, there’s no Frankenstein stuff going on here; I’ll use pretrained neural networks. I take two separate readouts: an image embedding for the learned representation, and a predicted label. These come from different models, so let’s keep that distinction in mind: a change in one model’s embedding doesn’t tell us where the other model’s decision boundary lies.

Learned Representation

The learned representation is the first function, \(y = f(x)\), that takes in our image \((x)\) and return an embedding which is the model’s representation of the image being fed to \(f(x)\). To extract the learned representation I utilized facebook/dinov2-base from HuggingFace. This function is given by;

To get the desired learned representation I selected the [CLS] token from the embedding, which is a global representation of the entire image \((x)\), 1 CLS token + 256 patch tokens. The learned representation function is now given by;

\(Y\) is a 768-dimensional vector which is the model’s global representation of the image. Further down in the experimentation, this representation will be used to perform these calculations.

Label

The facebook/dinov2-base doesn’t have a classification head to predict a given label, so I employed the use of facebook/dinov2-giant-imagenet1k-1-layer which has an ImageNet classification head with \(1,000\) class labels, \(C(x)\). Applying “\(\operatorname{argmax}\)” to the output logits of the classification head’s process from \(f_{\text{DINOv2-Giant}}(x)\) gives us the label of the image \((x)\).

\(C\) is the predicted class from the given image \((x)\), this is a semantic classification readout in addition to the embedding from facebook/dinov2-base.

Establishing the Baselines

Before proceeding, it is good scientific practice that baselines are established for all experimentation that’ll be performed. I used an image of a cat for this experiment and cropped it to the aforementioned dimensions i.e. 512 by 512.

Fig 1. The cropped image of cat image gotten from Dear Sunflower on Unsplash, hosted on Github.

Now, we establish the baseline learned representation \((Y_{0})\) and the baseline semantic classification readout \((C_{0})\) for our input image of a cat \((x)\), these are given by;

After running this baseline experiment we get the following baseline values;

Introducing Noise (Perturbation)

Now that we have established our baseline (ideal condition), we’d like to introduce the “noise”, the question now is “How much can we perturb the input image \(X\), that the learned representation \(Y\) changes?”. Now this shift our function from \(y = f(x)\) into a function that now takes into cognizanse, the noise, \(N\). This is now given by;

Notice the difference? \(X\phi\) and \(X\rho\) are now collapsed into \(N\). But we don’t only want to apply \(N\), we also want to apply \(N\) at varying degrees of magnitude, and hence we introduce the “perturbation magnitude” \((\delta)\). We apply fixed gaussian noise \((N)\) at varying perturbation magnitudes \((\delta_{i})\) for \(i = \lbrace1, 2, 3,…, 255\rbrace\), this function is now given by;

Because each pixel values \((p)\) are represented as \(p \in [0, 255]\), we have to make sure we don’t overshoot that threshold, so we clip the perturbed values between 0 and 255, this is given by;

To see how much \(Y_{i}\) changes with increase in perturbation magnitude \((\delta_{i})\) on our fixed gaussian noise \((N)\). We measure the euclidean distance \((D)\) between the baseline learned representation \((Y_{0})\) and the varying learned representations \((Y_{i})\), this is given by;

In addition to this, we want to see how the distance from the baseline changes as \(\delta\) increases by \(1\). I use \(\Delta D\) for this successive change in \(D\), not for the distance between adjacent embeddings. This is given by;

Now we run the pipeline with a fixed noise realization and increasing perturbation magnitude. The distance from the baseline rises, but the size of each successive increase changes along the way.

The distance rises quickly at first, grows more slowly around \(\delta=100\), then continues upward before flattening towards the end of the sweep. The successive-change plot makes the changing rate easier to see: \(\Delta D\) is initially large, then becomes smaller, with fluctuations. Near the end it is mostly below \(0.2\). This is a change in distance from the baseline, not a measurement of the angle between embeddings.

So, as perturbation magnitude becomes increasingly higher the incremental change in this distance becomes smaller. Looking at the semantic classification \((C_{i})\), and this is the interesting part; \(C_{i}\) remained \(C_{i} = \text{“Egyptian cat”}\) through \(i \in \lbrace1, 2, 3,…, 255\rbrace\). Regardless of increasing perturbation magnitude the semantic classification readout still remained \(\text{“Egyptian cat”}\). To peer into the details, we looked at the behavior of the class probability and we see the actual behavior of the semantic readout.

Table 1. The classification probabilities with increasing perturbation magnitude while still predicting \(\text{“Egyptian cat”}\).

While \(\text{“Egyptian cat”}\) remained the highest probability, it also fluctuated substantially, demonstrating that stable semantic classification did not imply a stable classifier confidence, showing classification invariance without probability invariance.

I repeated the experiment with different noise realizations. On the eighth run, the classifier’s winning label changed at \(\delta=230\). The base-model embedding also moved under perturbation, but that distance doesn’t locate the separate classifier’s decision boundary.

This is exactly the kind of behavior we wanted. At \(\delta = 229\) the probability for \(\text{“Egyptian cat”}\) was \(29.0702\%\), and at \(\delta = 230\) the probability for \(\text{“tiger cat”}\) was \(29.3101\%\) and steadily increased to \(37.3848\%\) at \(\delta = 255\). With the noise realization fixed, increasing \(\delta\) scales that noise before clipping; it doesn’t establish that the embedding keeps the same direction. To vary the noise realization as well, we use;

Now, what happens if we change the noise realization at each step too? We would be changing its direction as well as its magnitude, before clipping. That gives us a different experiment from following one fixed noise direction.

What we notice is that changing the direction of the noise realization by increasing values of \(\delta\) still maintains the upwards trends with fluctuation that become more consequential with increase in perturbation magnitude.

Make it Multiplicative

So far, we have added noise to the image. What happens if we multiply the pixel values by noise-dependent factors instead? This makes the perturbation depend on the input’s pixel values. With the noise field fixed, multiplication is linear in the input before clipping; clipping is the nonlinear step.

We perturb the image multiplicatively, given by the equation above aiming to shift the representation in the representation space by a significant amount.

Okay. The distance from the baseline rises sharply, then settles around a plateau with small fluctuations. Its successive change, \(\Delta D\), fluctuates around zero after the initial jump. The classifier also moves away from \(\text{“Egyptian cat”}\) almost immediately. The plateau tells us that this distance changes little at higher magnitudes; it doesn’t mean the embeddings stop changing, or establish why the plateau occurs.

Something jumps out immediately from our experiments above, we have been treating noise as “noise” from a signal processing perspective, while this is valid, there are many ways of perturbing the input \((x)\) and additive noise is one of them, but maybe not the best for our particular experimentation.

Blur, blur, blur sheep

The noise experiments show that representations can move substantially while the original label sometimes remains unchanged. They don’t tell us how much distinguishing information is left in the image. Now, enters Gaussian blur: what happens when we progressively suppress fine spatial detail instead? A new transformation function emerges, given by;

The new transformation function \((T)\) performs a transformation on input \((x)\) using Gaussian blur \((B)\) and with blur intensity \((\sigma)\), this function progressively removes information from the input image.

Under blur, the winning label changes early and sometimes returns to an earlier class. The image is classified as \(\text{“Egyptian cat”}\) at \(\sigma=1,2,3\), then as \(\text{“Japanese spaniel”}\) at \(\sigma=4\) and \(\text{“Blenheim spaniel”}\) at \(\sigma=6\). It returns to Egyptian cat at \(\sigma=7,8\), before changing again. By \(\sigma=255\), the prediction is \(\text{“digital clock”}\). The table below follows both the winning label and the probability assigned to the original class; those are different quantities.

Table 2. Selected classifications with increasing blur intensity, highlighting the return to Egyptian cat at \(\sigma=7\) (also predicted at \(\sigma=8\)).

Where do we meet?

Before we go further, there have been experiments similar to what we’re doing here. Dodge & Karam compared how humans and deep neural networks recognize images under distortions like blur and noise; the tested networks performed worse than humans, and their errors had little correlation with human errors.¹ Geirhos et al. explored this across multiple image degradations, and saw human and network error patterns increasingly diverge as the signal weakened.²

Blur itself has been investigated too. Vasiljevic, Chakrabarti & Shakhnarovich showed that networks trained on clear images lose recognition accuracy under blur, but fine-tuning with blurred images can recover much of that accuracy and produce blur-invariant representations.³ More recently, Jang & Tong found that training with clear and blurred images improved how well CNNs predicted measured neural responses and aspects of human recognition behavior.⁴ So, the way a network responds to blur also depends on what it was trained on.

What about the internal representation? Yoshihara, Fukiage & Nishida looked at how blur training changes those representations, comparing responses to sharp and blurred images from the same image, different images within the same class, and images from different classes.⁵ Comparing representations under blur therefore already has a body of work behind it.

My question comes from the earlier Medium post; I wanted a way to test part of what I said there with something we can define and measure. For this experiment I keep the pretrained models fixed, perturb the input, and follow the changes in learned representation and semantic readout. This gives us a way to test that part of the engineering analogy, but it doesn’t establish a model of what happens in a human brain.

How close can we get?

With the baselines established, we saw how additive & multiplicative noise, and then blur, moved the cat’s learned representation away from its original representation. We also saw the separate classifier change its prediction. Now, let’s take this further; how close can the representation of a car get to that of a cat?

The idea is to blur both images and measure the distance between their representations, then repeat the comparison while keeping one image unblurred. If they get closer, we would still need to see whether either approaches the other image’s original representation.

To get started I introduced a second image, a car classified as \(\text{“minivan”}\), and applied the same blur strength to both images. The transformed images are \(T_{cat,\sigma}=B_\sigma(x_{cat})\) and \(T_{car,\sigma}=B_\sigma(x_{car})\); passing them through our representation model gives us \(Y_{cat,\sigma}\) and \(Y_{car,\sigma}\). This is given by;

\(B_0(x)=x\) is the unblurred baseline, with representations \(Y_{cat,0}\) and \(Y_{car,0}\). Remember, we are using a 768-dimensional representation for these comparisons, recording 256 embeddings for each image; the baseline and blur strengths from \(\sigma=1\) to \(\sigma=255\).

Blurring both images

We measure the Euclidean distance between \(Y_{cat,\sigma}\) and \(Y_{car,\sigma}\) as we progressively increase blur at the same varying strengths, then observe how close or how far apart their representations are in the representation space.

The baseline distance of \(66.8234\) falls to \(59.4186\) at \(\sigma=4\), and \(58.1713\) at \(\sigma=8\) as we see the representations getting closer, but the distance doesn’t keep falling at every step; at \(\sigma=15\), it rises to \(62.9037\). It then decreases with fluctuations, reaching its lowest recorded distance of \(36.3616\) at \(\sigma=167\), and ending at \(40.3695\) at \(\sigma=255\).

At their closest, the distance between them was \(45.6\% ~ \textbf{lower}\) than the baseline distance. Does this mean the cat’s representation is becoming like the original car’s representation? Well, we are blurring both images, so both representations are moving. Maybe losing the details that distinguish them brings them closer, but let’s take a look at what happens when we keep one image fixed.

Holding one image fixed

What we observed from \({D}_1\) resulted from both representations moving. Now let’s remove that movement from one image. Let’s introduce two more measurements alongside \({D}_1\); \({D}_2\) compares the blurred cat with the original car, while \({D}_3\) compares the original cat with the blurred car. This gives us three pairwise distance measures, given by;

At \(\sigma=0\), all three distance measurements compare the same original pair, so they start at the same distance. Let’s sample the point at which the two blurred representations reach their closest recorded point, \(\sigma=167\), and the end of the sweep, \(\sigma=255\);

Table 3. Pairwise Euclidean distances before blur, at the closest recorded point for the two blurred representations, and at the final blur strength.

The observations above show that at \(\sigma=167\), the distance between the two blurred representations reaches \(36.3616\), while the other distance measures are \(65.2932\) and \(64.7817\). At \(\sigma=255\), those \({D}_2\), \({D}_3\) distances are \(64.6428\) and \(66.4435\); both remain much closer to the baseline distance of \(66.8234\) than \({D}_1\) does.

Fig 10. Euclidean distance of pairs \(D_{1}\), \(D_{2}\), and \(D_{3}\) against the increasing blur intensity \((\sigma)\).

Looking at the plot, the two blurred representations get closer to each other much more than they approach the other image’s original representation. This fits the idea that losing their distinguishing details brings them closer, though we haven’t measured how much information is left. But as established earlier, the distances alone don’t tell us everything. Euclidean distance depends on both the magnitudes of the embeddings and the angle between them. What happens to their direction?

Are they pointing in the same direction?

For this we use a common measure of directional alignment, “cosine similarity”, using the same three representation pairs from above, but instead of measuring the distance between them, we now measure their directions in the representation space. For nonzero vectors \(u\) and \(v\) from our pair, this is given by;

A value closer to \(1.0\) means the vectors point in more similar directions. We use \(\mathrm{cos}D_1\), \(\mathrm{cos}D_2\), and \(\mathrm{cos}D_3\) from the same pairs \({D}_1\), \({D}_2\), and \({D}_3\). These are names for the pairwise cosine similarities; we calculate them from the embeddings, not cosine of the distances.

Table 4. Cosine similarities for the same three 768-dimensional representation pairs, including the strongest recorded alignment of the two blurred images at \(\sigma=167\).

Again, we see the two blurred representations coming closer. \(\mathrm{cos}D_1\) starts at \(0.0180\) without blur and reaches a high similarity score of \(0.6874\) at \(\sigma=167\) where we observed the closest distance. The score then fluctuates, settling at \(0.6216\) at \(\sigma=255\), showing us that increasing the blur strength doesn’t keep increasing cosine similarity.

Now let’s look at \(\mathrm{cos}D_2\) and \(\mathrm{cos}D_3\). \(\mathrm{cos}D_2\) only reaches a score of \(0.0771\), while \(\mathrm{cos}D_3\) fluctuates approximately around \(-0.0432\) to \(0.0322\). At \(\sigma=167\), their scores are \(0.0444\) and \(0.0264\), respectively. They stay closer to zero, showing little to no similarity with other image’s original representation. The slightly negative values in \(\mathrm{cos}D_3\) still indicate near-orthogonality, rather than strong alignment in opposite directions.

Fig 11. Cosine similarity score of pairs \(\mathrm{cos}D_{1}\), \(\mathrm{cos}D_{2}\), and \(\mathrm{cos}D_{3}\) against the increasing blur intensity \((\sigma)\).

We see cosine similarity of \({D}_1\) increase as the blurred embeddings become more aligned, showing partial convergence between the two blurred representations with a substantial distance still remaining. Again, this doesn’t mean that cat and car are becoming the same.

At the highest recorded similarity score, at \(\sigma=167\), the cat image was classified as \(\text{“wire-haired fox terrier”}\), while the car image was classified as \(\text{“television, television system”}\), with probabilities of \(7.9\%\) and \(8.0\%\) respectively. At \(\sigma=255\), we get \(\text{“digital clock”}\) and \(\text{“microphone, mike”}\). Even at the point when they are closest, the separate classifier doesn’t give them the same label or anything similar.

Where this leaves us?

The question that started this was simple; how do we end up saying “I’d like to park my cat” when we meant “car”? By introducing noise and blur, we saw the model’s representation move and classifier prediction change. Blurring both images also brought their representations closer, but keeping one image unblurred showed that they didn’t get close to the other’s original representation.

The interesting part of this experimentation is how two representations became more alike as blur suppresses the visual detail distinguishing their inputs. Getting closer in representation space also didn’t give us the same label from the classifier. These observations have not addressed why a human would say “cat” when they meant “car”. The truth is there are so many things that can make a human “glitch” in the manner that built this hypothesis; heck, through the course of writing this post I made several similar mistakes.

In my case I was exhausted and stressed after a wedding, which is another angle that might be worth exploring; maybe exhaustion or stress could be translated to jittering or perturbing the system (model) itself and not the input.

This experiment gave me a way to measure parts of my idea from the initial post, and showed me where that idea needs more work. For now, I have something concrete to test further, rather than a demonstrated explanation of what happens in the brain; and that is okay.

Updated: October 28, 2026

It wasn’t wrong?

The system was responding to the input it received. Heck, we had been changing the image all along. Why? Well, because input \((x)\) was changing with noise and blur, the system was providing a learned representation and classification based on that changed input. Does that make every prediction correct? Not really, but it does bring us back to what we were trying to investigate in the first place.

We also saw that perturbing an image and trying to get an overlap in learned representation between our baseline and the desired output didn’t translate to the question that started this journey. While that gave us a basis, I took some extra time in the following week to explore the next idea. What if the error was a result of perturbation to the system and not the input?

So, instead of perturbing the input \((x)\) with varying magnitude and direction, we perturb the system instead. We still have the magnitude of the noise and its direction to work with, so instead of;

We now get a function where the input stays fixed, but the system is perturbed, given by;

Now, how would we introduce noise to the system? In the initial post, noise was given as “internal” and “external” noise. Here, we bring them together into a single variable, \(N\), as a way of representing their combined effect in our model. We aren’t testing the two sources separately, so for this experiment we write;

How do we introduce this noise (perturbation) into the system? The answer wasn’t so obvious to me at first, “internal activation perturbation”; we hook into a layer and perturb its output activations.

Keeping things fixed

For this experiment, we keep the model weights & the input \((x)\) fixed; then hook into a selected layer and perturb the layer’s original output activations \((H_{\ell})\) using additive Gaussian noise before it gets passed on. We multiply the perturbation magnitude \((\delta)\) by the standard deviation of the layer’s activation \((\sigma(H_{\ell}))\), and then by randomly sampled Gaussian noise \((\epsilon)\), this is given by; ⁶

This scales the noise to the layer’s activations, but it doesn’t stop higher magnitudes from disrupting the prediction. To get started, we ran the baseline experiment on the facebook/dinov2-base-imagenet1k-1-layer model, using the image of the car \((x)\) from our previous experiment, then introduced the noise. We got;

We selected layer 6 of the model \((H’_{\ell}, \ell = 6)\), and perturbed the output activations with randomly sampled Gaussian noise at a magnitude \((\delta)\) of \(2.0\). We see a shift in the prediction even though the input image hasn’t changed. This is the part of the idea we wanted to test. The layers are counted from zero here, so layer 6 is the seventh layer.

Porting from images

The main idea has always been to understand what turns “I’d like to park my car” into a mistake like “I’d like to park my cat”, we have just seen jittering output activations cause a shift in the predicted class for the image model. Now we move this to text generation, which will stand in as an engineering analogue to speech generation. For this experiment we used a small language model, HuggingFaceTB/SmolLM2-135M, and perturbed the activations of a particular layer to see how two things are affected; the predicted next-token and its probability. ⁷

We also ran a baseline experiment here to establish if the language model will predict \(\text{“car”}\) after being given the prompt “I would like to park my”, and at what probability. The model gives us scores for the possible next tokens, called logits \((L_t)\), and softmax turns those scores into probabilities \((p_t)\). We take the token with the highest probability as the next-token prediction \((z_t)\), given by;

After running the baseline experiment, we got “car” with a probability of;

We now perturbed the output activations of layer 6 with a perturbation magnitude of \(0.50\), as we ran the experiment again; prompt was held fixed & model weights were held fixed, this resulted in;

This is great, we now see that jittering the activations shifts the probability for \(\text{“car”}\) from \(57.8125\%\) to \(54.2969\%\), a drop of about \(3.52\) percentage points. But the next-token prediction still remains \(\text{“car”}\).

When does it happen?

Now that we have seen jittering the activations affect the next-token probability, we want to examine at what point the next-token stops being \(\text{“car”}\) and becomes something else; at this point it doesn’t have to be contextually correct. We introduce perturbation magnitudes within the range of \(0.01 \leq \delta \leq 1.0\), changing the noise each time, and we got the following results.

Table 5. Effect of increasing internal activation perturbation magnitude \((\delta)\) on next-token predictions and their Softmax probabilities.

We notice that the next-token prediction stopped being \(\text{“car”}\) at a perturbation magnitude of \(0.50\) in this run, but how often does \(\text{“car”}\) change between those intervals? So, we ran the experiment again with a new range, running it \(100\) times for each magnitude with randomly sampled noise. We count how often the prediction changes from \(\text{“car”}\), which gives us the change rate below. This counts any change, whether the new token fits the context or not.

Table 6. Effect of increasing internal activation perturbation magnitude \((\delta)\) on next-token prediction stability across 100 trials.

This shows us that even at the highest perturbation magnitude within the given range, the model still predicts \(\text{“car”}\) \(2\%\) of the time in \(100\) trial runs. But notice that even at \(0.20\), the prediction already changed \(3\) times. So, \(0.50\) was where it changed in our earlier run, not a point where it must always change.

Where does it happen?

So far, it’s been happening at layer 6 of the model, but what happens when we try to perturb the activations at different layers? To answer this we perturb the activations at each layer, one at a time. We keep the perturbation magnitude fixed to \(0.30\), where we saw the effect of the perturbation, and run the experiment \(100\) times for each layer. We observed the following results; ⁸

Fig 12. Layer-wise change rate of the baseline prediction of \(\text{“car”}\) of SmolLM2-135M’s 30 layers.

Looking at the table below, we’ll notice something interesting; from layer 11 to layer 27, there’s a \(100\%\) change rate of the baseline prediction, with newline characters, whitespaces, punctuation, and token fragments among the alternatives. But we also notice something interesting at layers 28 and 29, we see a return of the baseline prediction and vehicle-related next-token predictions. Looking back at layers 0, 1, 9, and 10, we also see the baseline remain unchanged in all \(100\) trial runs.

Table 7. Layer-wise sensitivity of next-token predictions to Gaussian perturbations. Reporting the prediction change rate and the top two alternative tokens.

Why are layers 28 and 29 predicting vehicle related next-tokens? Does this mean that layers 28 and 29 give more stable next-token prediction even with the addition of random sampled noise? It is easy to make this assumption, so let’s test it out. We keep the layer fixed at \(29\), the perturbation magnitude fixed to \(0.30\), and run 1,000 trial runs.

Table 8. Alternative next-token predictions across 1,000 trials with perturbations at layer 29 \((\delta = 0.3)\) with a prediction change rate of 7.00%, and baseline frequency of 930/1000.

We see the baseline next-token prediction of \(\text{“car”}\) holding steady with a baseline frequency of \(930\) out of \(1,000\) trials. Does this mean that layer 29 is more robust than the other layers in the transformer? Not really, layer 29 is relatively robust to this fixed perturbation magnitude \((\delta = 0.30)\), at least for this particular prompt and next-token prediction, even with randomly sampled Gaussian noise in each of the \(1,000\) trial runs.

But, there are important things to note; the noise is scaled by the standard deviation of each layer’s activations, so the same \(\delta\) doesn’t mean we added the same amount of noise at every layer. Also, seeing “car” remain the prediction doesn’t mean its probability stayed the same.

Does the stability of next-token prediction at layer 29 remain with increasing perturbation magnitude? How does layer 28 fare under similar increasingly noisy perturbations? We ran the experiment \(1,000\) times at each magnitude for both layers. We see layer 28 approach a change rate of \(100\%\), reaching \(96.8\%\) at a perturbation magnitude of \(0.50\), while layer 29 has a change rate of \(40.1\%\) at the same magnitude.

This brings us back to the previous statement about layer 29; “relatively robust”. For the same prompt, with increasingly higher perturbation magnitude, its change rate eventually reaches \(99.8\%\) at \(2.00\), while layer 28 gets close to \(100\%\) at a lower magnitude. So, we see a difference between these layers for this prompt, but what happens if we change the prompt?

Something different…

To further test the robustness of layer 29, we swapped the prompt twice with different baseline next-token predictions; the first swap is “The pilot prepared to land the” and the second is “The musician played a beautiful melody on the”. We ran the experiment \(1,000\) times at each perturbation magnitude within the range \(0.10 \leq \delta \leq 2.00\), increasing by \(0.05\) up to \(0.40\), then using larger steps after that. We got the baseline next-token predictions for these two prompts, calling them \(z1_t\) and \(z2_t\);

Now we introduce the perturbation and see how these predictions shift and their change rates increase with increasing perturbation magnitude.

We see some interesting results that bring us back to our idea from above; “relatively robust” to a specific perturbation magnitude and a specific prompt. We see the first prompt’s baseline next-token change rate gradually increase as the magnitude increases, reaching \(98.1\%\) at \(1.25\), \(98.0\%\) at \(1.50\), and \(99.8\%\) at \(2.00\). We still see a small fluctuation along the way.

While with the second prompt, we see an earlier, steeper increase in the baseline next-token change rate; at a perturbation magnitude of \(0.35\), the change rate was already at \(63.5\%\). It gets close to \(100\%\) earlier, reaching \(99.2\%\) at \(1.25\) and \(99.9\%\) at \(1.50\), before reaching \(100\%\) at \(2.00\). Looking at all three prompts together, we see that changing the prompt also changes how often the noise shifts the prediction.

Fig 17. Layer 29 baseline comparison of \(\text{“car”}, \text{“plane”} ~ \& ~ \text{“piano”}\) change rate with increasing perturbation magnitude \((\delta)\).

I just wanted to say Hi-llo

“Hi-llo” to you too. I apologize, I wanted to say “Hi” and changed my mind half-way and wanted to say “Hello”. This brings “When does it happen?” and “Where does it happen?” into another experiment, in a slightly different manner; “What happens if we introduce noise while the model is already generating a response?”. For this we moved to a variation of our initial prompt, “I would like to park”, generated a baseline completion and introduced noise with a perturbation magnitude of \(1.00\) at the second step of text generation. For the baseline generation;

After introducing the noise at layer 29 during the second step of text generation, we notice a diversion from the baseline completion. We apply the noise to the layer’s output activations just for that step, then remove the hook; the noise isn’t being added again at every step that follows.

Table 9. Token-by-token comparison of baseline and perturbed generation following transient activation interference at step 2.

Notice how the tokenizer chunks the completion; consid, ert, and ion come together as “considertion”. Now, the noise was only added at the second step, so why does the rest of the completion change too? Well, the model now has a different sequence of tokens to work with. The noise is gone, but the token it generated is still there, and the words that follow are now being generated from that changed sequence.

This is where the “Hi-llo” analogy comes back; something changes while the response is already being generated, and we see the completion take another path. We haven’t got the exact “car” to “cat” mistake here, but we can see how a brief change inside the model can affect what it says next. What happens if we introduce the noise at a different step? That would be another experiment; here, we only tested the second step.

Putting everything together

You started this journey with me as we explored the engineering analogue to the human brain and how errors might occur in speech generation. I’d like to say that we learned a lot of things along the way, but here are a few that stick with me.

We started by perturbing the input, and the model gave us different representations and predictions based on the image it received. Think about trying to recognize a highly noisy image or a blurred one yourself; you’d have a hard time telling what’s in the image. That was the intuition here, though we didn’t compare the model’s predictions with human responses. So, this didn’t fall flat on its face, but it wasn’t quite the experiment that fit our original question.

The idea of exhaustion and stress leading to speech generation errors was the next idea we explored. We kept the input fixed and perturbed the model’s internal activations, which showed a shift in the prediction of the image. We ported the experiment to a language model, and we started seeing parts of the initial idea from the Medium story popping up again; the same prompt, with something jittering inside the system, could lead to a different next-token prediction. Does this mean we have explained what exhaustion or stress does in the brain? Not yet, but it gives us an engineering version of that idea to work with.

Furthermore, something that stuck out was that the change in prediction was not just a result of how much noise was going into the system; the layer where we introduced the noise and the prompt also made a difference. We then introduced noise at one step during text generation and saw the completion take another path, even after the noise was removed. So, it’s not just noise in a system, there’s more to that. We still have things to test, like what happens when we introduce noise at other generation steps, but this brings me closer to the question that started the post. How do we go forward from here? Well, that is to be decided at a later time.