Learn by making the model work — and fail
Two incomplete views.
One learnable pattern.
An image tells you one thing. A measurement tells you another. Can a model learn what they mean together?
A live experiment with synthetic data and models trained in your browser. No patient data. No preset performance scores.
01 / Choose the relationship
What makes the class predictable?
Class 1 means the two features have the same sign. Knowing only one sign tells you almost nothing. Their relationship is the signal.
SYNTHETIC TASK02 / See what fusion learns
The joint feature space
Stress-test the image input noise · wrong pairs · missing modality
Weights stay fixed. Only held-out image inputs change.
Add a fixed random perturbation to each image feature, then clip to [−1, 1].
Replace a subset of image features with another example's image. Labels stay with the original example.
Replace every x with zero. This is an intentionally naive fallback, not a validated missing-input policy.
Same held-out examples. Six actual models.
Accuracy at a 0.5 cutoff · click a model to inspectThe line tracks training log loss for the selected model. Lower is better. Scrub the epoch slider to rewind its weights.
03 / Follow the selected example
Where does the information meet?
Take it back to health research
Fusion is a hypothesis
you can test.
In a real study, each modality might contribute new evidence, duplicate an existing signal, or encode how care was delivered. The experiment above lets you separate those possibilities.
Why does averaging fail in “Only together”?
The single-modality classifiers discard the sign relationship before their scores are averaged. With independent, symmetric inputs, each modality alone has a true class probability of 0.5. Combining two such scores cannot recover the missing relationship. More flexible late-fusion methods and different tasks can behave differently.
Does intermediate fusion always beat early fusion?
No. Switch to “Early · with an interaction feature”: the simple classifier receives x, y, and x × y and can solve the same task. The key is representing the relationship, not the name of the fusion strategy. In “Adding evidence,” a linear early model already matches the data-generating rule.
What would I test in a real multimodal study?
Compare strong single-modality baselines and fusion models on the same held-out patients. Match patient, encounter, and input availability time. Evaluate missing-input patterns, subgroup calibration, external sites, uncertainty, and acquisition costs. Test retrained modality ablations separately from missing-input stress tests.
Leakage-safe evaluation · External validation · Missing data
Exactly what is being trained here?
Image contrast is rescaled as (mean right − mean left) / 0.7, up to pixel quantization. Each task uses 320 training and 180 independent evaluation examples with fixed random seeds. Image and table summaries are x and y. The grayscale image deterministically encodes x; no CNN is trained. All classifiers minimize binary cross-entropy with full-batch gradient descent, learning rate 0.18, for 1,200 epochs. The test set never updates weights.
Image/table baselines are logistic regressions on one feature. Early linear uses [x,y]; early interaction uses [x,y,xy]. Late fusion averages the two baseline probabilities. The intermediate model learns one scalar tanh encoder per modality, combines them in eight tanh hidden units, then produces a sigmoid score. This deliberately small network demonstrates joint representation learning; it is not a clinical architecture benchmark.
“Only together” uses class = 1 when xy > 0. “Adding evidence” uses x+y > 0. “Repeating evidence” uses two noisy views of one latent variable whose sign defines the class. All values are synthetic. Accuracy and log loss describe this finite demonstration, not clinical utility. Model selection on this displayed test set would require a new untouched test set in a real study.