Protein Colour: Hard to Beat a Simple Average

A rhodopsin wavelength model achieved a respectable-looking rank correlation, yet it reduced average prediction error by just 0.24 nanometres compared with giving every protein the same answer. The model had learned something about ordering. That translated into very little improvement in the predicted values on this test set. The study paper explains the difference.

This matters whenever a model score becomes a practical recommendation. If you need a wavelength estimate for a particular protein, how often the model puts two proteins in the right order is only part of what you need to know.

The question starts with light

Rhodopsins are light-sensitive proteins. Here, the quantity being predicted is peak absorption wavelength: the wavelength of light the protein absorbs most strongly, measured in nanometres, or nm. The FLIP2 Rhomax dataset collects measurements of this property for microbial rhodopsins. This comparison says nothing about whether a protein expresses well, remains stable or performs a useful function in a cell.

The study used 584 examples for fitting, held back 116 for validation, and evaluated the models on a separate set of 184. It compared six configurations: a constant prediction, two ways of describing amino-acid composition, two frozen ESM-2 protein language models, and a lookup based on the nearest training sequence. The full methods explain their shared setup and its limits.

The constant is an important starting point. It predicts the training-set average, about 539.73 nm, for every test protein. It ignores the supplied sequence completely. Any model that uses the sequence should be judged against what that simple answer already achieves on the same test set.

Correct order can hide inaccurate numbers

Imagine three proteins with measured wavelengths of 500, 550 and 600 nm. A model predicts 539, 540 and 541 nm. Those predictions put the proteins in exactly the right order. They still give poor estimates of the actual wavelengths.

That is a constructed example, but it illustrates the distinction. Rank correlation measures how well an ordering agrees with the measurements. Mean absolute error, or MAE, averages how far each prediction lies from its measurement, ignoring whether the prediction was too high or too low. MAE keeps the units of the quantity you care about: nanometres here.

The composition model, which describes how much of each amino acid a sequence contains, reached a Spearman rank correlation of about +0.418. Its MAE was 24.36 nm. The constant's MAE was 24.60 nm. Both were therefore wrong by about 24 nm on average, despite the composition model's ability to order some of the proteins. These are held-out results from the fixed panel, not estimates of what every future protein would do.

The constant has no defined rank correlation because all its predictions tie. Its 24.60 nm average error remains a useful reference. These charts describe one test split, and the rank correlations have no uncertainty intervals.

The uncertainty analysis also needs a careful reading. The estimated composition-minus-constant difference was −0.24 nm, with a 95% interval of approximately [−0.45, −0.05] nm. The interval supports a small improvement under the study's analysis, rather than a tie. But the gain falls below the study's chosen 2 nm reporting threshold. That threshold is a rule for this comparison, not a universal definition of a useful biological improvement.

Those intervals came from resampling groups reconstructed from sequence similarity, because the released data did not supply the original group identities. They describe uncertainty under that exploratory grouping choice. They do not establish performance across independently repeated experiments on new groups of related proteins. The paper's uncertainty section keeps that limitation explicit.

The predictions barely move

The composition model's predictions covered only about 2.2 nm across all 184 test rows. The measured wavelengths ran from 436 to 589 nm. That is a spread of 153 nm. The model could make small adjustments that sometimes improved the order while still returning nearly the same wavelength for every protein. The preserved prediction diagnostics show this directly.

The composition predictions vary very little as the measured wavelength changes. The plotted spreads are descriptive; the uncertainty on the resulting error differences is discussed above.

The fitting choices matter here. Protein language models learn patterns from large collections of protein sequences and can turn a sequence into numerical features. Here, both ESM-2 feature sets and both composition descriptions fed into a small regression model that converted those features into a wavelength. The ESM-2 models remained frozen: their weights were not updated for this task. The regression settings were fixed, and feature scales differed, so the same penalty against large fitted coefficients affected the representations differently. The methods and scale diagnostics document that limitation.

The two ESM-2 configurations had larger average errors: about 28.90 nm for the 8-million-parameter model and 29.16 nm for the 35-million-parameter model. That supports a conclusion about those particular configurations. It does not show that protein language models generally lose to composition, or that a larger or fine-tuned model would fail. Those alternatives were not tested.

A shortlist asks a different question

You might care less about predicting an exact wavelength than about choosing ten candidates near a target. The study also tried that decision, selecting predictions closest to 550 nm. Nine of the composition model's ten selected proteins had measured wavelengths between 540 and 560 nm. Uniform random selection from the same test population would put about 31% in that band on average. The shortlist results therefore show an interesting result that the small MAE improvement alone would miss.

It remains an exploratory result from this candidate set. Its uncertainty interval was wide and added after the main analysis; selecting candidates from a fresh population could behave differently. A useful shortlist, a correct ordering and an accurate individual estimate are three separate outcomes to evaluate.

For rhodopsin wavelength models, the useful habit is to ask for the constant baseline and the error in nanometres alongside the ranking score. Then check whether the evaluation matches your decision. The full research version, paper and maintained code retain the details if you want to examine this comparison further.

The MS/MS model-selection article examines a related distinction: ranking candidates and identifying an answer.