Same signal. Four architectures.
Move a measurement. Trace what changes. Discover why a network’s connections matter before choosing a bigger model.
Which inputs can reach this output?
These eight synthetic measurements are a toy sequence, not patient data. We compute four fixed-weight teaching layers, not trained predictors. Their outputs are activations, not risks or comparable performance scores.
Edit any value with a slider (keyboard arrows work). The highlighted input is the selected position. Each diagram connects contributing inputs to that position’s output. Line width shows coefficient magnitude, not causal importance.
Three experiments, three learning levels
- Beginner · Local or global? Select position 4. Change input 8. Dense and unmasked attention can change; this convolution and forward recurrence cannot. Trace their connections to explain why.
- Intermediate · Shared patterns. Reset, select position 4, and note the convolution output. Shift right and select position 5: the output matches because the same filter follows the same neighborhood (away from padding). Dense weights depend on absolute position.
- Advanced · Order and access. Reset, note attention at position 4, reverse, then select position 5 (the same query value). Its output matches: this head has no position encoding or mask. Recurrence generally changes because its prefix changes. A transformer needs positional information to distinguish sequence order; a causal mask separately blocks future inputs.
Check your reasoning: does convolution make every prediction shift-invariant?
No. Shared filters make interior feature responses shift with the input (equivariance). Padding, stride, and boundaries complicate this. A later pooling or readout determines whether a final prediction stays unchanged.
Choose the structure your task needs
Dense → fixed feature roles
Useful when columns have different meanings. A dense layer mixes all features; it does not assume adjacent columns form a local pattern. Compare with simpler tabular baselines.
Train a dense EHR model →Convolution → repeated local patterns
Shared filters look for patterns across space or time. Stack layers to widen the receptive field. A 1D signal filter here illustrates the same sharing principle as a 2D image filter.
Explore image feature maps →Recurrence → evolving state
A forward recurrent layer carries a state through ordered observations. Our scalar state compresses the prefix; long-range effects shrink here. GRU/LSTM gates offer richer memory control. Irregular observation times still need explicit handling.
Review network training →Attention → content-dependent mixing
Each query weights allowed keys. Our single scalar head is only one component of a transformer; real blocks add learned projections, multiple heads, feed-forward layers, residual paths, and normalization.
Explore a causal transformer →Under the hood: hand-checkable mechanics
Indices in these equations are zero-based (the interface labels 1–8). All biases are zero. Dense and convolution outputs here are linear; recurrence uses tanh.
- Dense: yᵢ = Σⱼ Wᵢⱼxⱼ, where Wᵢⱼ = ((i + 2j) mod 5 − 1.5)/4. A general 8-to-8 dense layer has 64 weights, plus 8 biases if enabled.
- Convolution: yᵢ = 0.25xᵢ₋₁ + 0.5xᵢ + 0.25xᵢ₊₁. Outside values are zero. Three shared filter weights, no bias; stride 1. This symmetric filter smooths local values.
- Recurrence: hᵢ = tanh(0.6hᵢ₋₁ + 0.8xᵢ), h₋₁ = 0. Two shared scalar weights, no bias. Diagram widths show the decay factor 0.6ⁱ⁻ʲ, not the exact nonlinear derivative.
- Attention: qᵢ = xᵢ, kⱼ = xⱼ, vⱼ = xⱼ; aᵢⱼ = exp(qᵢkⱼ)/Σₖ exp(qᵢkₖ); yᵢ = Σⱼ aᵢⱼvⱼ. Scalar dimension is 1, so the usual √d scaling is 1. Identity projections, no learned parameters, no mask. All eight weights sum to 1.
These counts describe the specified toy layers only. Attention forms an n × n score matrix in its standard full form; recurrent computation advances sequentially; convolution shares a local filter. None of these facts alone determines accuracy or runtime on a real task.
Before a health ML comparison: define prediction time and allowed inputs, use the same patient-level splits and tuning budget, and evaluate calibration and external performance. A bidirectional model is inappropriate if it can see observations unavailable at prediction time. See data leakage and external validation.
Further reading: D2L: convolution and weight sharing, recurrent networks, and Attention Is All You Need.