Our In-House Recipe for Juicy, Fact-Stuffed MLPs

TL;DR (click to expand):

Transformers store much of their factual knowledge inside MLP layers, but how they do it is still unclear. Whereas most previous efforts to understand MLP fact storage have focused on probing trained LLMs, we take a generative approach: we explicitly construct MLPs that implement key-value fact mappings. This gives us provable guarantees of correctness, precise facts-per-parameter scaling laws, and new insights into how embedding geometry affects MLP parameter efficiency and how transformers can reliably use MLPs. Our construction is the first to be:

• Provably Efficient: for some embeddings, our construction matches the information-theoretic limit of $\Theta(F \log F)$ bits to store $F$ facts.
• General: our construction handles realistic, non-uniform embeddings.
• Usable: Transformers can learn to use our MLPs for factual recall.

In this blog, we'll dig into the details of our construction.

⏮️ Part 1 of the blog post
📄 Paper

In Part 1, we introduced our generative view of factual recall in MLPs. Here, we zoom in on the construction: we'll explain how to turn a set of key/value embeddings into a provably correct and efficient fact-storing MLP. We’ll keep the math light and focus on the core intuitions throughout.

---

Problem setup

First, let’s formalize what it means for an MLP to store a set of facts.

We're given key embeddings $\mathbf{K} \in \mathbb{R}^{|\mathbf{K}| \times d}$, value embeddings $\mathbf{V} \in \mathbb{R}^{|\mathbf{V}| \times d}$, and a fact map $f : [|\mathbf{K}|] \to [|\mathbf{V}|]$ that assigns each key its correct value.

We'll say that an MLP $g_\theta : \mathbb{R}^d \to \mathbb{R}^d$ stores this fact set if, for every key $\mathbf{k}_i$, its output is closer (under dot-product decoding) to the correct value than to any other:

We use the dot product as our similarity measure because it matches the softmax decoding used in language models; this is also the same setup used in prior work.

We quantify our construction's parameter efficiency via its fact-storage cost: we define this as the minimum number of parameters a model family needs to represent every possible fact map over $(\mathbf{K}, \mathbf{V})$. (Lower is better!)

A simple counting argument gives us a lower-bound on fact-storage cost, assuming constant bits per parameter: any model family must use at least

parameters. Intuitively, there are $|\mathbf{V}|^{|\mathbf{K}|}$ possible mappings from keys to values, so just identifying which mapping you want already takes $\Theta(|\mathbf{K}| \log |\mathbf{V}|)$ bits.

In this blog, we'll focus on using gated MLPs to store facts. Particularly, we'll consider MLPs of the form

where $\sigma$ is any activation function (we focus on ReGLU or SwiGLU MLPs in this blog). Check out the paper for more details on non-gated MLPs!

Our goals

With the basic setup in place, we can now spell out what we want from our constructed MLPs. Compared to prior constructions, our approach is designed to achieve three goals:

1. Handle general embeddings.
   Prior work focuses on key and value embeddings that are uniformly spherically distributed, but embeddings from language models are anisotropic and structured. We'll provide a construction that works for all but a measure-zero set of feasible embeddings (those with $\rho(\mathbf{V}) > 0$, as described below).

2. Match our information-theoretic baseline.
   Ideally, our constructed MLPs should match the baseline of $\Theta(|\mathbf{K}| \log |\mathbf{V}|)$ parameters. We'll show that our construction matches this baseline up to a geometry-dependent constant:

   $$\Theta\!\big([\rho(\mathbf{V})]^{-2}\,|\mathbf{K}| \log |\mathbf{V}|\big),$$

   where $\rho(\mathbf{V})$ is a decodability factor we introduce shortly. For "well-spread" embeddings such as spherical ones, $\rho(\mathbf{V}) = \Omega(1)$, giving a scaling of $\Theta\!\big(|\mathbf{K}| \log |\mathbf{V}|\big)$. This scaling asymptotically matches the parameter count of our information-theoretic baseline and requires a factor of $\log^{11} |\mathbf{V}|$ fewer parameters than the lowest bound from prior work.

3. Be usable inside Transformers.
   We want Transformers to be able to use our constructed MLPs for factual recall, with fact-storage cost that scales similarly to the MLP's fact-storage cost. Our construction handles this too, but we'll leave the details for Part 1 of our blog post.

Here, we'll focus on the first two goals.

---

Roadmap

We’re now ready to walk through the main ideas behind the construction. We’ll build up to the full MLP in a few steps:

1. Encoder gadget.
   We start with a simplified setting and construct a gated MLP that maps $|\mathbf{K}|$ inputs in $\mathbb{R}^d$ to arbitrary scalar outputs using only $O(|\mathbf{K}|)$ parameters. This “encoder gadget” is essentially optimal (up to a constant factor) for the scalar-output case. However, naïvely stacking $d$ copies of this gadget to produce $d$-dimensional outputs still requires $\Theta(d\,|\mathbf{K}|)$ parameters, which is too large to achieve our target scaling.

2. Decoding and $\rho(\mathbf{V})$.
   Dot-product decoding gives us an extra degree of freedom: our MLP doesn’t need to output the value embeddings themselves, only vectors that rank the correct value highest under dot-product similarity. This leads us to the decodability $\rho(\mathbf{V})$, which measures how geometrically "separated" the value embeddings are. Small $\rho$ means the values are tightly packed and require more parameters to separate; large $\rho$ means they’re easier to distinguish. To take advantage of this, we introduce margin-optimal outputs, a set of output vectors that can maximally distinguish each value embedding from all other value embeddings.

3. Compression via a random projection decoder.
   The final idea to achieve our target scaling is to compress the margin-optimal outputs to a lower-dimensional code while preserving the dot-product decoding constraints. We show that a simple random linear projection suffices via a Johnson–Lindenstrauss (JL) argument. With this choice of decoder, projecting into

   $$m = \Theta\!\big([\rho(\mathbf{V})]^{-2} \log |\mathbf{V}|\big)$$

   dimensions achieves perfect decoding accuracy with high probability.

4. Full encoder-decoder MLP.
   The exponential reduction in output dimension from compression now lets us directly apply our stacked encoder gadget and, for many embeddings, allows us to achieve fact-storage cost asymptotically matching our information-theoretic baseline! This yields our final construction:

   $$g(\mathbf{x}) = \mathbf{D}\,\mathbf{E}\big(\sigma(\mathbf{G}\mathbf{x}) \odot (\mathbf{A}\mathbf{x})\big),$$

   where $\mathbf{D}$ is the JL-style decoder and $\mathbf{E}\big(\sigma(\mathbf{G}\mathbf{x}) \odot (\mathbf{A}\mathbf{x})\big)$ is the encoder. We show our construction can store any fact set over generic keys and decodable values with fact-storage cost

   $$\Theta\!\big([\rho(\mathbf{V})]^{-2}\,|\mathbf{K}| \log |\mathbf{V}|\big).$$

---

Our encoder gadget

We first focus on a simplified setting:

Can we construct a gated MLP that maps a finite set of inputs in $\mathbb{R}^d$ to arbitrary real numbers, using only $O(|\mathbf{K}|)$ parameters for $|\mathbf{K}|$ datapoints?

Our plan is to stack multiple copies of this encoder gadget to produce full outputs.

We’ll start with a particularly nice choice of key embeddings as a warmup: two-hot difference vectors of standard basis vectors.

A toy setting: two-hot differences

Fix a dimension $d$ and consider the set of inputs

which has $|\mathbf{K}_{\text{2hot}}| = d^2 - d$ elements. For each ordered pair $(i,j)$ with $i \neq j$, we choose an arbitrary target scalar $h(i,j) \in \mathbb{R}$.

Our goal is to build an MLP $\mathrm{enc} : \mathbb{R}^d \to \mathbb{R}$ such that

The encoder gadget construction

In this setting, we propose a simple weight construction for a 1-hidden-layer gated MLP, which we call our "encoder gadget":

where

Intuitively, constructing the encoder gadget has two steps: 1) pick a gating term to cause different hidden neurons to select different "portions" of the inputs, then 2) set the $A$ matrix so it fits the data.

How does this work?

Plugging in a two-hot input $\mathbf{x} = \mathbf{e}_i - \mathbf{e}_j$ with $i \neq j$:

1. Gating step.

   $$\mathrm{ReLU}\big(\mathbf{I}_d (\mathbf{e}_i - \mathbf{e}_j)\big) = \mathrm{ReLU}(\mathbf{e}_i - \mathbf{e}_j) = \mathbf{e}_i,$$

   since $\mathbf{e}_i - \mathbf{e}_j$ has a $+1$ in coordinate $i$ and a $-1$ in coordinate $j$ (which gets zeroed out).

2. Linear step.

   $$\mathbf{A}(\mathbf{e}_i - \mathbf{e}_j) = \mathbf{A}\mathbf{e}_i - \mathbf{A}\mathbf{e}_j.$$

   The $i$-th coordinate of this vector is

   $$\mathbf{A}[i,i] - \mathbf{A}[i,j] = 0 - (-h(i,j)) = h(i,j).$$

3. Elementwise product + sum.
   Gating zeroes out all but the $i$-th coordinate:

   $$\mathrm{ReLU}(\mathbf{I}_d (\mathbf{e}_i - \mathbf{e}_j)) \odot (\mathbf{A}(\mathbf{e}_i - \mathbf{e}_j)) = \mathbf{e}_i \odot (\mathbf{A}(\mathbf{e}_i - \mathbf{e}_j)) = h(i,j)\,\mathbf{e}_i.$$

   Summing with $\mathbf{1}_d^\top$ gives the scalar $h(i,j)$.

From toy to general embeddings

The two-step core idea behind the encoder gadget extends beyond the two-hot embeddings setting:

• Instead of using the identity as the gating matrix, we choose a more general gating matrix $\mathbf{G} \in \mathbb{R}^{h \times d}$. For analytic activation functions like Swish/SwiGLU and GeLU/GeGLU, our construction works for generic $\mathbf{G}$ (for all but a measure 0 set of possible $\mathbf{G}$), so we can just sample a random matrix with i.i.d. Gaussian entries.
• Once the gating matrix is fixed, the network becomes linear in the remaining weights. This means we can solve a linear system to find an $\mathbf{A}$ matrix that matches any desired outputs.

The encoder gadget recipe for general embeddings:

Fix a generic $\mathbf{G}$. For each key $\mathbf{k}_i$, the encoder gadget produces

$$c_i \;=\; \mathbf{1}_h\left[\mathbf\sigma(\mathbf{G}\mathbf{k}_i) \odot (\mathbf{A}\mathbf{k}_i)\right] = \sum_{r=1}^h \mathbf\sigma(\mathbf{G}\mathbf{k}_i)_r \langle \mathbf{k}_i,\, \mathbf{A}_r\rangle,$$

so each encoder output is a linear function of $\mathbf{A}$.

Stacking these equations for all $i$ and all $r$ gives a single linear system of the form

$$\mathbf{M}\,\mathbf{v} = \mathbf{c},$$

where:

• $\mathbf{v} \in \mathbb{R}^{dh}$ is just all the entries of $\mathbf{A}$ flattened,
• $\mathbf{c} \in \mathbb{R}^{|\mathbf{K}|}$ contains the desired outputs $(c_1,\dots,c_{|\mathbf{K}|})$, and
• $\mathbf{M} \in \mathbb{R}^{|\mathbf{K}|\times (dh)}$ depends only on the keys and the gating terms $\sigma(\mathbf{G}\mathbf{k}_i)$.

In the paper, we show that the construction holds for generic $\mathbf{G}$ with enough rows (roughly $hd \gtrsim |\mathbf{K}|$), for commonly-used activation functions like GeLU and Swish (and more generally any non-polynomial analytic function), for all but a measure-zero set of key embeddings, and for all choices of target scalars. For example, with Swish gate activations, our construction produces SwiGLU MLPs, like those from modern frontier models (e.g., Llama or Qwen).

Encoder Construction (sketch)

1. Choose a generic gating matrix $\mathbf{G} \in \mathbb{R}^{h \times d}$.
2. Compute the gating vectors $\sigma(\mathbf{G}\mathbf{k}_i)$ for all keys.
3. Form and solve the resulting linear system to find $\mathbf{A}$ such that $\mathrm{enc}(\mathbf{k}_i) = c_i$ for every $i$.

Parameter count.
A simple degrees-of-freedom argument shows that we need at least $|\mathbf{K}|$ parameters to specify arbitrary outputs on $|\mathbf{K}|$ inputs. When $dh=|\mathbf{K}|$, our encoder uses

parameters: one set of $|\mathbf{K}|$ parameters for $\mathbf{G}\in\mathbb{R}^{h\times d}$, one for $\mathbf{A}\in\mathbb{R}^{h\times d}$, and minor overhead for the readout. For $d, h = o(|\mathbf{K}|)$, this stays within a constant factor of the $|\mathbf{K}|$ lower bound.

---

Our first construction attempt (and why it’s not enough)

Our encoder gadget already gives us a simple way to solve the full problem.
If the gadget lets us map from $\mathbb{R}^d \to \mathbb{R}$, then to map from $\mathbb{R}^d \to \mathbb{R}^d$ we can just stack $d$ copies of the gadget — one per output coordinate!

Excitingly, this naïve construction is provably correct, and already works for a broader range of inputs/outputs than prior constructions! However, it uses $\Theta(d\,|\mathbf{K}|)$ parameters, which can be much larger than our information-theoretic
$\Omega(|\mathbf{K}| \log |\mathbf{V}|)$ bound when $d \gg \log |\mathbf{V}|$ (e.g., for typical LLM embedding dimensions).

To do better, we need to take advantage of our decoding mechanism.

---

Exploiting dot-product decoding

Let’s step back and look at the dot-product decoding condition again. For a fact set $f$, we require

A simple observation:

We don’t need $g_\theta(\mathbf{k}_i)$ to equal $\mathbf{v}_{f(i)}$! We only need it to rank the correct value with the largest dot-product score.

This leads to two key ideas:

1. We should pick MLP outputs that maximize the decoding margin — and they don't necessarily need to equal $\mathbf{V}$.
2. We can compress these output embeddings into a lower-dimensional space, as long as the inequalities hold after decompression.

Decodability $\rho(\mathbf{V})$

To motivate the first idea, we ask a simple question: how hard is it to tell the value embeddings apart using dot-product decoding? Some sets of value vectors are easy to distinguish; others are tightly clustered and require much larger margins to separate. We capture this with a single geometric quantity, the decodability $\rho(\mathbf{V})$.

Intuitively, $\rho(\mathbf{V})$ measures the best achievable margin between each value $\mathbf{v}_i$ and all competing values under dot-product decoding. Larger $\rho$ means the values are well-separated and easy to decode; smaller $\rho$ means they’re packed together and require more parameters to separate.

Before we define $\rho(\mathbf{V})$, we note that it turns out to be a fundamental quantity for fact-storing MLPs. For $\rho(\mathbf{V}) \le 0$, there are no MLPs that can store certain fact sets over $\mathbf{V}$. Additionally, $\rho(\mathbf{V})$ is predictive of fact-storage cost (equivalently, MLP size) across gradient-descent-trained and constructed MLPs (both ours and from prior work (NTK)!).

Now, let's define $\rho(\mathbf{V})$. Formally,

This is the largest worst-case (normalized) margin $\langle \mathbf{v}_i - \mathbf{v}_j,\ \mathbf{u}_i \rangle$ we can give every value by choosing appropriate output vectors $\mathbf{u}_i$.

Given our definition of $\rho(\mathbf{V})$, a natural choice of MLP output is to take, for each value, the margin-optimal output direction

$\mathbf{u}_i^\star$ is the spherical Chebyshev center of the vectors $\mathbf{v}_i - \mathbf{v}_j$ (for $j\neq i$) — geometrically, it's the direction that most robustly separates $\mathbf{v}_i$ from all other values. Our paper contains a more detailed justification of $\mathbf{u}_i^\star$. Here we'll simply foreshadow that $\mathbf{u}_i^\star$ turns out to be the optimal output for our decoder.

Compressing margin-optimal outputs

The second key idea is to use compression to reduce the dimension of the MLP outputs. Conveniently, dot-product decoding only depends on the sign of certain inner products:

If we can compress and decompress the $\mathbf{u}_i^\star$ while preserving these signs, decoding will still work.

Our decoder stems from the following observation: a Johnson–Lindenstrauss–style random linear projection $\mathbf{D} \in \mathbb{R}^{d \times m}$ with

is enough to keep all the relevant inner-product signs intact with high probability.

Why does this work?

After compression,

$$\mathbf{u}_i^\star \mapsto \mathbf{D}^\top\mathbf{u}_i^\star, \qquad \mathbf{v}_i - \mathbf{v}_j \mapsto \mathbf{D}^\top(\mathbf{v}_i - \mathbf{v}_j),$$

and decoding depends only on the sign of

$$\langle \mathbf{v}_i - \mathbf{v}_j,\ \mathbf{u}_i^\star \rangle.$$

Random projections preserve all pairwise inner products up to a small (less than $\rho(\mathbf{V})$) distortion when

$$m = \Omega(\rho(\mathbf{V})^{-2} \log |\mathbf{V}|).$$

Since the original inner products have a margin of at least $\rho(\mathbf{V})$, distortions smaller than $\rho(\mathbf{V})$ cannot flip the signs. This means that decoding continues to work after compression!

Decoder Construction (sketch)

1. Start with the margin-optimal outputs $\mathbf{U}^\star \in \mathbb{R}^{|\mathbf{V}| \times d}$.
2. Sample a random projection matrix $\mathbf{D} \in \mathbb{R}^{d \times m}$ (e.g., i.i.d. Gaussian).
3. Form compressed codes
   $$\mathbf{C} = \mathbf{U}^\star \mathbf{D}.$$
4. Decode by projecting back up:
   $$\mathrm{dec}(\mathbf{c}) = \mathbf{D} \mathbf{c}.$$
   Then $\mathrm{dec}(\mathbf{c}_i) \approx \mathbf{u}_i^\star$, and the decoding inequalities are preserved.

The key insight: now that we can reduce the output dimension from $d$ to $\Theta(\rho(\mathbf{V})^{-2}\log|\mathbf{V}|)$ using our decoder, we can revisit our stacked-encoder idea and finally make it parameter-efficient!

---

Putting everything together: the full construction

Let's combine the encoder and decoder into a single MLP that stores an arbitrary fact set over generic keys and decodable values. The resulting model has the form

where $(\mathbf{G}, \mathbf{A}, \mathbf{E})$ form the encoder, $\mathbf{D}$ is the decoder, and $\sigma$ is a gating activation function (e.g., ReLU, GeLU, Swish). The encoder produces a compressed code in $\mathbb{R}^m$; the decoder maps it back into $\mathbb{R}^d$ so that the output dot-product-decodes to the correct value embedding.

To see our full recipe:

Our MLP Construction (sketch)

1. Choose compressed outputs.
   Compute margin-optimal outputs $\mathbf{U}^\star\in \mathbb{R}^{|\mathbf{V}| \times d}$ and draw a random projection

   $$\mathbf{D} \in \mathbb{R}^{m \times d}, \qquad m = \Theta\!\big(\rho(\mathbf{V})^{-2} \log |\mathbf{V}|\big).$$

   Form compressed codes

   $$\mathbf{C} = \mathbf{U}^\star \mathbf{D}^\top.$$

2. Construct the encoder.
   Treat $\mathbf{C}$ as the target outputs and use the encoder to find weights
   $(\mathbf{G}, \mathbf{A}, \mathbf{E})$ such that

   $$\mathrm{enc}(\mathbf{k}_i) = \mathbf{c}_{f(i)} \quad \text{for all } i.$$

3. Assemble the full model.

   $$g(\mathbf{x}) = \mathbf{D}\,\mathrm{enc}(\mathbf{x}).$$

Our main theoretical result shows that our proposed construction satisfies our goals of asymptotic scaling and generality:

Theorem (Full construction, informal).
For any fact set $f$, generic keys $\mathbf{K}$, and values $\mathbf{V}$ with $\rho(\mathbf{V}) > 0$, the above construction yields a fact MLP $g$ that stores $f$ and has fact-storage cost

$$\Theta\!\big([\rho(\mathbf{V})]^{-2}\, |\mathbf{K}| \log |\mathbf{V}|\big).$$

For value embeddings with $\rho(\mathbf{V}) = \Omega(1)$, such as uniformly-spherical embeddings, this matches our information-theoretic baseline of $\Omega(|\mathbf{K}| \log |\mathbf{V}|)$, improving over bounds from prior work by a factor of $\log^{11} |\mathbf{V}|$.

Since we have matched the parameter count asymptotics of our information-theoretic baseline, we can next ask how many bits per parameter our construction uses. For the two-hot example, we actually achieve a constant number of bits per parameter (leading to information-theoretically optimal scaling in the total number of MLP bits), and, for embeddings such as the uniform spherical embeddings assumed by prior constructions, our construction uses $O(\log |\mathbf{K}|)$ bits per parameter. In both cases, our construction comes within a single log factor of the information-theoretically optimal rate, when measured in bits.

Table 1 summarizes how our construction compares to bounds on prior fact-storing MLP constructions.

Construction	Parameters	Hidden Sizes	Assumptions on $\mathbf{K}$	Assumptions on $\mathbf{V}$
Info-theoretic Baseline	$\lvert\mathbf{K}\rvert \log \lvert\mathbf{V}\rvert$	$\tfrac{\lvert\mathbf{K}\rvert}{d} \log \lvert\mathbf{V}\rvert$	None	None
Naïve (stacked encoder gadgets)	$d\, \lvert\mathbf{K}\rvert$	$\lvert\mathbf{K}\rvert$	Generic	$\rho(\mathbf{V}) > 0$
Nichani et al. (2024)	$\lvert\mathbf{K}\rvert \log^{12} \lvert\mathbf{V}\rvert$	$\tfrac{\lvert\mathbf{K}\rvert}{d} \log^{12} \lvert\mathbf{V}\rvert$	Uniform on $\mathbb{S}^{d-1}$	Uniform on $\mathbb{S}^{d-1}$
Ours	$[\rho(\mathbf{V})]^{-2}\, \lvert\mathbf{K}\rvert \log \lvert\mathbf{V}\rvert$	$\tfrac{[\rho(\mathbf{V})]^{-2} \lvert\mathbf{K}\rvert}{d} \log \lvert\mathbf{V}\rvert$	Generic	$\rho(\mathbf{V}) > 0$

---

How close are we to optimal in practice?

So far we’ve focused on theory; how well does the construction work empirically?

To test this, we compare several MLP families. We compare our construction (which we label "Ours: Bin + JL") against the following baselines:

• GD MLPs: fully end-to-end MLPs trained directly on the dot-product decoding objective using gradient descent and a cross entropy loss
• NTK MLPs: the construction from Nichani et al. (2024)

In addition to the GD and NTK baselines, we evaluate ablations of our encoder-decoder framework:

• GD encoder (Ours: GD + JL):
  We keep the JL decoder fixed, but instead of using the explicit encoder gadget, we train the encoder parameters $(\mathbf{G}, \mathbf{A}, \mathbf{E})$ with gradient descent and a mean-squared-error loss so that

  $$\mathrm{enc}(\mathbf{k}_i) \approx \mathbf{c}_{f(i)}$$

  for the target compressed codes $\mathbf{C}$.

• GD decoder (Ours: Bin + GD):
  We keep the encoder gadget fixed, but instead of using a random projection as the decoder, we train a linear map $\mathbf{D}$ and compressed codes $\mathbf{C}$ with gradient descent and a cross entropy loss on the logits

  $$z_{ij} = \langle \mathbf{D}\mathbf{c}_{i}, \mathbf{v}_{j} \rangle.$$

• GD encoder + GD decoder (Ours: GD + GD):
  We keep the same encoder-decoder architecture as our construction, but with both the encoder and decoder parameters learned with gradient descent. The encoder and decoder are trained separately.

We fix the embedding dimension ($d=128$), sample uniform-spherical embeddings, and vary the number of facts. For each MLP family, we measure the minimum number of parameters needed to perfectly store a randomly chosen fact set.

Three observations stand out:

• Our MLPs match GD MLPs' asymptotics.
  The explicit construction (Bin + JL) follows the same asymptotic trend as GD-trained MLPs as the number of facts grows. In contrast, NTK MLPs exhibit asymptotically worse scaling with increasing number of facts and fixed embedding dimension.

• We consistently outperform NTK.
  Even in this clean spherical setting, our explicit MLP is roughly $4\times$ more parameter-efficient than NTK-based constructions — and the gap widens with more facts. Around $2^{14}$ facts, NTK MLPs fail to achieve perfect fact storage while ours continue working.

• Our encoder-framework is performant when partially learned.
  Using a GD-trained encoder together with a GD-trained decoder gets us most of the way to the parameter-efficiency of fully gradient-descent-trained MLPs (worse by $\sim 4\times$, better than NTK by $>16\times$). This suggests that our encoder-decoder framework captures most of the structure that gradient descent discovers.

Conclusion

---

Acknowledgements

Thank you to Yasa Baig, Kelly Buchanan, Sam Buchanan, Catherine Deng, Andy Dimnaku, Junmiao Hu, Herman Brunborg, Rajat Dwaraknath, Jonas De Schouwer, and Atri Rudra for helpful feedback on this blog post.

More Resources

📄 Paper
⏮️ Part 1 of the blog post