BERT-Base Interactive DataLab Visualizer

Stage 1 • Layer 0

Tokenization & Static Initial Embedding

Vector Shape: [1, 768]

The sentence is split into WordPiece tokens. At Layer 0, the [MASK] token gets a static initial vector with zero contextual knowledge.

• Abstract Embedding Mapping Flow Tokens → Token IDs → 768-D Space

Raw Layer 0 Embedding for token: [MASK] (ID: 103) Showing first 12 of 768 dimensions

Stage 2 • Transformer Layers (1-12)

Self-Attention: Absorbing Surrounding Context

Multi-Head Attention

Attention heads compute compatibility scores between [MASK] and surrounding words.

• Dynamic Attention Arc Visualization [MASK] connects to top 2 context words

Post-Attention Vector State (Stage 2 Output)

Active Hot-Spots

Stage 3 • Non-Linear Synthesis

Feed-Forward Network (FFN) & GELU Activation

768 → 3072 → 768

The FFN projects the vector to 3,072 dimensions, applies GELU to filter noise, and compresses back to 768 dimensions.

• FFN Expansion & Compression Flow Linear → Non-Linear → Linear

💡 GELU Gate

Passes positive features; zeros out negative noise.

x • Φ(x)

Before Activation (Linear Projection) Carried from Stage 2

After GELU Activation (Feature Synthesis) Zeroed / Amplified

Deep-Layer Convergence Tracker

Progressive Feature Refinement Across All 6 Layers

Representation Sharpening

Values lock into a sharp alignment pointing directly toward the target word between Layer 5 and Layer 6.

Layer 1: Capturing local syntax & broad lexical cues High Variance

Layer 11: Suppressing distractor concepts & ambiguity Med Variance

Layer 12: Final semantic lock matching vocabulary dictionary Target Locked ✓

Stage 5 • The Climax (Layer 12 Output)

LM Head: Dot-Product Vocabulary Matching

30,522 Vocab Matrix

The [MASK] embedding vector has physically shifted through high-dimensional space. We now compute its dot product (h_mask • E_vocab) against every word in BERT's vocabulary dictionary. Higher geometric similarity equals a higher raw logit score.

💡 What is a logit? A logit is the raw prediction score that BERT assigns to each word in its vocabulary before converting the scores into probabilities using Softmax.

Rank	Candidate Token	Vector Alignment (Dot Product Visualization)	Raw Logit Score (z_i)

Stage 6 • Mathematical Resolution

Softmax Calculation: Logits to Probabilities

P(x_i) = e^z_i / ∑ e^z_j

Converting unbounded logits into confidence percentages summing to 100% using Softmax.

Candidate	Step 1: Logit (z)	Step 2: Exp (e^z)	Step 3: Division (∑ e^z)	Confidence %

Denominator (∑ e^z): Calculating...

Resolved [MASK] Prediction

--

The uncontextualized vector successfully navigated 12 layers, aligned its dimensions with the vocabulary matrix, and resolved the missing word!

BERT-Base DataLab: What Happens Under the Hood

Ready to trace the `[MASK]` embedding

Tokenization & Static Initial Embedding

Self-Attention: Absorbing Surrounding Context

Feed-Forward Network (FFN) & GELU Activation

Before Activation (Linear Projection) Carried from Stage 2

After GELU Activation (Feature Synthesis) Zeroed / Amplified

Progressive Feature Refinement Across All 6 Layers

LM Head: Dot-Product Vocabulary Matching

Softmax Calculation: Logits to Probabilities

--

Ready to trace the [MASK] embedding

Tokenization & Static Initial Embedding

Self-Attention: Absorbing Surrounding Context

Feed-Forward Network (FFN) & GELU Activation

Before Activation (Linear Projection) Carried from Stage 2

After GELU Activation (Feature Synthesis) Zeroed / Amplified

Progressive Feature Refinement Across All 6 Layers

LM Head: Dot-Product Vocabulary Matching

Softmax Calculation: Logits to Probabilities

--

Ready to trace the `[MASK]` embedding