Expert Emergence in Mixture-of-Experts Architectures

Everything here is dummy for now, fyi! I will be adding real content once I complete the project.

Introduction

Transformer architectures have become the dominant paradigm for natural language processing tasks (Elhage et al., 2021) .

Testing citation here (Vaswani et al., 2017) .

Background

The Dummy Circuit

Some LaTex dummy:

LaTex in a normal paragraph can be added like this:

When $W_{OV} \approx I$ (the identity matrix), the attention head primarily copies information from attended positions without significant transformation.

Measuring Dummy Behavior

We can quantify how close a dummy is to being a dummy using the dummy norm:

where $\delta_{ij}$ is the dummy delta.

Inline LaTex dummy:

Methodology

Our analysis proceeds in three steps:

1. dummy step 1
2. dummy step 2
3. dummy step 3

Model Selection

We analyze the following architectures:

Results

Results to be added in future versions.

Code sample:

import torch
import torch.nn as nn
import torch.nn.functional as F

class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.fc1 = nn.Linear(10, 20) # sample comment to extend the line - more dummy text to see how it looks
        self.fc2 = nn.Linear(20, 10)

    def forward(self, x):
        x = self.fc1(x)
        x = self.fc2(x)
        return x

Conclusion

Conclusion ooga booga