Choosing a Model for 5′UTR Translation Experiments

If you have a library of 5′UTR reporters with measured translation, which model should you use to predict the next batch? We compared three options on 100,017 designed 5′UTRs: ridge regression on cheap sequence features, a small convolutional neural network (CNN) trained from scratch, and a frozen RNA language model (UTR-LM).

The short answer:

• For predicting values on new sequence families, use the CNN. It cut mean squared error by 28% against the best cheap baseline.
• For picking the top 1%, cheap features did about as well. The CNN's advantage was too uncertain to call.
• Frozen UTR-LM embeddings added nothing over one-hot input to the same CNN.

These are exploratory results from one assay. The full methods, tables and figures are in the study paper.

The question

A lab that measures 5′UTR libraries usually wants a model for one of two jobs: estimate translation for sequences it has not made, or choose a small set of candidates to synthesise and test. These are different tasks. A model can get the average right and still rank the best sequences no better than a simpler one.

The three options differ in cost and complexity:

1. Cheap features with ridge regression. Short sequence counts, position-specific nucleotides, and known regulatory features such as upstream start codons (uAUGs) and Kozak context. Fits in seconds on a CPU.
2. A small CNN trained on the library's own labels, in the style of Optimus 5-Prime (Sample et al., 2019). Minutes on a laptop GPU.
3. A pretrained RNA language model whose embeddings feed a trained head. This adds pretrained weights and an embedding step. The question is whether that buys accuracy.

The assay

The data come from the designed library of Sample et al. Each reporter mRNA carries a fixed 25-nucleotide leader, a variable 50-nucleotide insert, and the eGFP coding sequence. The mRNAs are transfected into HEK293T cells, and the cell lysate is separated on a sucrose gradient into fractions carrying different numbers of ribosomes. Sequencing each fraction gives every 5′UTR a mean ribosome load (MRL): the average number of ribosomes on each mRNA molecule.

How a polysome-profiling reporter assay produces MRL. This panel shows a 2024 library with the same 25 + 50 nt construct, not the 2019 data used here. Adapted from Castillo-Hair et al. 2024, Fig. 1b, CC BY 4.0.

The inserts include fragments of 35,212 human 5′UTRs, 3,577 clinical variants, and sequences from design algorithms, including step-by-step evolution trajectories in which every intermediate was made. The library was measured once, with no replicate.

How the comparison worked

Two ways to split the data

Many sequences in this library are relatives: variants of the same parent, or steps along the same design trajectory. We grouped records into families if they shared a parent sequence or any identical 20-nucleotide stretch. That gave 33,575 families.

In the standard random split, 79.3% of test sequences had a relative in training. A model can score well there by recognising near-copies. So we also built a grouped split that keeps every family on one side of the train/test boundary. Both splits used about 70,000 training, 15,000 validation and 15,000 test records.

The grouped test set is not just a harder version of the random one. One large family of 14,101 design-algorithm sequences had to go to training, which changed the test set's composition: 5.5% of grouped-test sequences were high-MRL, against 9.5% in the random test set. Methods should be compared within a split, not across them.

Methods

• Cheap baselines: ridge regression on nucleotide composition, 1- to 3-mer counts, positional one-hot encoding, uAUG and Kozak annotations, and all of these combined. The combined model was best on validation data and served as the reference.
• CNN: three convolutional layers and a small dense layer, trained from scratch on one-hot inserts.
• UTR-LM: a 1.2-million-parameter checkpoint pretrained only on unlabelled 5′UTRs. Its weights stayed fixed. Its embeddings fed three heads: ridge on averaged embeddings, ridge on per-position embeddings, and the same CNN design as above.

Every method chose its settings on validation data and was scored on the test set once.

Two measures of success

• Mean squared error (MSE) measures how close predicted MRL is to measured MRL. Lower is better. Predicting the training mean gives 2.22 on the grouped test set.
• Precision at 150 measures the pick-list task. Take each model's top 150 predictions (1% of the test set) and count how many are truly high-MRL, defined as the top 10% of training labels. Random picking finds about 8 of 150 on the grouped split.

Before scoring, we fixed what counted as a useful improvement: an MSE reduction of at least 0.10 MRL², or a precision gain of at least 0.05, with a 95% interval that excludes zero. Intervals came from resampling whole families, so related sequences were not counted as independent evidence.

Results

The CNN predicted values better, but did not clearly pick better

On the grouped split, the CNN had MSE 0.560 against 0.776 for cheap combined ridge. The whole uncertainty interval for the reduction lies above the 0.10 threshold, and two further training seeds gave the same verdict.

For the top 150 picks, the picture changes. The CNN found 86 high-MRL sequences and the cheap baseline 83, both roughly ten times better than random. The difference was not supported: the interval runs from a CNN advantage of about 0.13 to a disadvantage of about 0.09. The CNN's own precision varied from 0.533 to 0.580 across training seeds, more than the gap between the two models.

On the random split the CNN was better on both measures: MSE 0.427 against 0.849, and 118 high-MRL picks against 87. That is the only supported precision gain in the study, and it comes from a test set where most sequences had a relative in training.

MSE reduction against cheap combined ridge (upper rows) and against the one-hot CNN (lower rows), with 95% intervals. Positive values favour the named method. The dashed line is the 0.10 MRL² threshold. "Historical" in the panels is the random split.

Frozen UTR-LM embeddings did not help

Ridge on UTR-LM embeddings was worse than ridge on cheap features on both splits. The best version, per-position embeddings, had grouped MSE 1.020 against 0.776.

Fed into the CNN instead of one-hot input, UTR-LM embeddings gave grouped MSE 0.570 against 0.560: no supported difference (−0.010 [−0.034, 0.016]). On the random split they lowered MSE slightly, by less than the threshold, and lowered precision.

This is a result about one small frozen checkpoint. The UTR-LM authors fine-tune by default, and in their own ablation fine-tuning far outperformed a frozen model. We did not test fine-tuning, larger checkpoints or other RNA language models. mRNABench reports a similar ordering for this dataset under a different protocol, with a supervised CNN ahead of linear probes on language-model embeddings.

Cost

The whole run took 57 minutes on an Apple M4 laptop. Cheap combined ridge took about 20 seconds and under 0.6 GB of memory. One CNN training run took 3 to 5 minutes on the laptop GPU. UTR-LM added a 2-minute CPU embedding pass and a 1.3 GB embedding cache, without improving accuracy over the CNN.

Test MSE against time to fit, on a log scale. Single runs on a shared laptop, so treat times as rough.

Where the errors sit

Errors were largest for sequences with few sequencing reads. On the grouped split, CNN MSE was 0.785 in the lowest read-depth quarter and 0.419 in the highest. Without a replicate, we cannot tell how much of that is measurement noise and how much is model error.

One family of 49 trajectory sequences accounted for most of the CNN's error in its worst sub-library, where the CNN's MSE was 3.98 against 0.59 for cheap features. A single aggregate MSE hides this kind of failure, so it is worth checking error by family.

Grouped-split MSE by sub-library (left) and read-depth quartile (right). Point estimates without intervals.

No method recovered the one single-nucleotide variant pair we examined: measured MRL rose by 0.66, and predicted changes ranged from −0.06 to +0.32. This study did not evaluate variant-effect prediction.

Which model to use

"No supported difference" is a finding, not a gap. But it is not proof that the models are equivalent: the precision interval still allows a CNN gain of around 0.1.

Limitations

• MRL is not protein output. MRL ignores mRNA with no ribosomes attached, so a high-MRL sequence can still be poorly translated overall.
• Reporter results may not transfer to endogenous mRNAs. Karollus et al. (2021) found MPRA-trained MRL models correlated only 0.11 to 0.25 with endogenous translation measurements.
• One assay. One eGFP reporter, one cell line, one unreplicated measurement, fixed 50-nt inserts.
• About 70,000 labels. No learning curve was measured, so this says nothing about which model wins with a few thousand labels.
• Exploratory, not confirmatory. Both test sets had prior exposure: the random test set had been used before, and whole-dataset label summaries were viewed before the grouped split was registered. The 22 comparisons are not adjusted for multiple testing.
• Pretraining overlap unchecked. Human 5′UTR fragments in this library may appear in UTR-LM's pretraining data.
• Not yet independently reproduced. All numbers come from a single run on 30 September 2026. A same-machine rerun matched exactly, but no reproduction on other hardware has been done.

What would change the conclusions

A fresh set of new designs measured in the same assay would turn these exploratory numbers into a real test, and a replicated measurement would show how much remaining error is noise. A precision interval above zero would move the pick-list recommendation to the CNN. A fine-tuned or larger language model that beat the one-hot CNN by more than 0.10 MRL² on a grouped split would reopen the case for pretrained features on this assay.

The study repository has the paper, the registered protocol, the code and the results archive behind every number here. For a related study where a ranking metric needed a careful definition, see the MS/MS model-selection post.