BERT-Base Interactive DataLab Visualizer

Stage 1 • Input Embeddings

Tokenization & Static Initial Embedding

Special tokens [CLS] (start of text) and [SEP] (end of text) are added automatically.

Live Animation

1. TOKEN
[MASK]

→

2. TOKEN ID
ID #103

→

3. EMBEDDING
Matrix Lookup

→

4. VECTOR ROW
[1 × 768] Matrix

Raw Layer 0 Embedding for token: [MASK] (ID: 103) Showing first 12 of 768 dimensions

*The default embedding gets the context from the surrounding tokens in the following layers and eventually predict the word.

Stage 2 • Transformer Layers (1-12)

Self-Attention: Absorbing Surrounding Context

Multi-Head Attention

go

to

[MASK]

everyday

morning

Attention heads compute compatibility scores between [MASK] and surrounding words.

Layer 1 Self-Attention Output for [MASK] (post-attention state) First 12 of 768 dims

Stage 3 • Non-Linear Synthesis

Feed-Forward Network (FFN) & GeGLU Activation

GeGLU Filter

• FFN Expansion & Compression Flow

Before Activation (Linear Projection)

After GELU Activation (Feature Synthesis) Zeroed / Amplified

Deep-Layer Convergence Tracker

Progressive Feature Refinement Across All Layers

Logit Lens Probe

Mechanism: Using the Logit Lens technique, we intercept the hidden state vector at each layer of ModernBERT and project it through the final unembedding layer. This gives us a direct view of the model's trajectory across all 22 layers as its output prediction converges.

Bars: meaning shift per layer (direction)

Hover a bar to see that layer's logit-lens guess.

Stage 5 • The Climax (Layer 22 Output)

LM Head: Dot-Product Vocabulary Matching

30,522 Vocab Matrix

h_mask [768] • E_vocab [50,368 × 768] = Logits [50,368]

Rank	Candidate Token	Vector Alignment (Dot Product Visualization)	Raw Logit Score (z_i)

Stage 6 • Mathematical Resolution

Softmax Calculation: Logits to Probabilities

P(x_i) = e^z_i / ∑ e^z_j

Candidate	Step 1: Logit (z)	Step 2: Exp (e^z)	Step 3: Division (∑ e^z)	Confidence %

Denominator (∑ e^z): Calculating...

• How close was it? (top candidates by probability)

The [MASK] token vector successfully navigated 12 layers, aligned its dimensions with the vocabulary matrix, and resolved the missing word!

The Big Picture

BERT understands words in context

Older models read each word in isolation. BERT reads the whole sentence at once with self-attention — so every word's meaning is shaped by the words around it. That's contextual understanding.

Where this powers things you use daily

🔍

Google Search

Reads the intent behind your query

💬

Chatbots & Assistants

Understands what you actually mean

🌐

Translation

Preserves meaning across languages

❓

Question Answering

Pinpoints the answer inside text

Thank you for exploring BERT.

DataLab: What Happens
Under the Hood

BERT DataLab: What Happens Under the Hood

Tokenization & Static Initial Embedding

Self-Attention: Absorbing Surrounding Context

Feed-Forward Network (FFN) & GeGLU Activation

Before Activation (Linear Projection)

After GELU Activation (Feature Synthesis) Zeroed / Amplified

Progressive Feature Refinement Across All Layers

LM Head: Dot-Product Vocabulary Matching

Softmax Calculation: Logits to Probabilities

BERT understands words in context

Where this powers things you use daily

DataLab: What HappensUnder the Hood

Tokenization & Static Initial Embedding

Self-Attention: Absorbing Surrounding Context

Feed-Forward Network (FFN) & GeGLU Activation

Before Activation (Linear Projection)

After GELU Activation (Feature Synthesis) Zeroed / Amplified

Progressive Feature Refinement Across All Layers

LM Head: Dot-Product Vocabulary Matching

Softmax Calculation: Logits to Probabilities

BERT understands words in context

Where this powers things you use daily

DataLab: What Happens
Under the Hood