The Best Rank Correlation Buys 0.24 Nanometres: Six Rhodopsin Wavelength Models on One FLIP2 Split

You have a rhodopsin candidate, you want its peak absorption wavelength in nanometres, and it sits in a wild-type background your training set does not contain. Six configurations were run on the FLIP2 Rhomax by_wild_type split and scored on the same 184 held-out rows: a constant training-mean prediction, two archived amino-acid-composition controls with a fixed alpha-10 ridge, frozen ESM-2 8M and 35M residue-mean embeddings with the same head, and a nearest-training-sequence lookup.

The endpoint is peak absorption wavelength in nanometres and nothing else. No number below says anything about activation efficiency, expression level, photostability, ion transport or cellular function.

The best rank correlation belongs to a composition control at +0.418 Spearman. It beats the constant by 0.24 nm of mean absolute error, 24.36 nm against 24.60 nm, on a test set whose measured standard deviation is 32.15 nm. The two numbers disagree because the probe's entire predicted range is 2.2 nm wide: it is a constant with a wobble, and the wobble points the right way. Meanwhile the two worst configurations on every aggregate metric, frozen ESM-2 8M and 35M, both surfaced a 587 nm protein in their red-shift shortlists, and that hit sits inside chance.

So "which model is best" has one answer per decision, and the decisions do not agree.

The decision table

Table 1. Recommendations for one population only: the 184 held-out rows of the FLIP2 Rhomax by_wild_type split, 92 of which have no training sequence even of the same length. Each row names its own comparator, because the estimation rows compare against a constant training-mean prediction while the shortlist rows compare against uniform random selection at the same capacity. The adaptation is a frozen encoder with a fixed alpha-10 ridge head and target scaling fitted on the 584 train rows only, with no hyperparameter search. Intervals are 95% paired cluster bootstrap over 37 reconstructed background groups, which are a declared reconstruction and not an archive label; shortlist intervals are post-hoc and were not predeclared. Endpoint: peak absorption wavelength in nm. Measurements originate with Inoue et al. 2021, CC-BY 4.0, reaching this workspace through FLIP2.¹²

That is the answer. The rest is how it was measured, and where it breaks.

Figure 1. The same panel read as a decision path. The estimation and near-match leaves compare against a constant training-mean prediction; the two shortlist leaves compare against uniform random selection at capacity 10, because a constant defines no shortlist. Population: the 184 held-out rows of the FLIP2 Rhomax by_wild_type split. Adaptation: frozen encoder, fixed alpha-10 train-only ridge head, no tuning. Supported uncertainty: paired cluster bootstrap over 37 reconstructed background groups with a 2 nm reporting threshold, except the 5-row near leaf, where no interval was computed and the number is descriptive, and the shortlist leaves, whose intervals are post-hoc. Measurements originate with Inoue et al. 2021, CC-BY 4.0, via FLIP2.¹²

What was run, and why a constant is a serious opponent

The archive is FLIP2 Rhomax, by_wild_type split, Zenodo record 18433203 version 3, CC-BY 4.0.³ Verified from the bytes rather than from the paper: 884 rows in 584 train, 116 validation and 184 test; four columns, sequence, target, set and validation; targets in nanometres from 436 to 622 across 144 distinct values; lengths 227 to 365 residues over exactly the 20 standard amino acids; and zero sequence overlap across all three split pairs. The decompressed CSV has sha256 8e78f6a16cd5298131dca83130d4b94cc0ab4ae9c699460880441c19a65f656f, checked before anything is scored.

The split is hard in one specific way. Random splits of protein data mislead, because sequences are correlated by evolutionary history,⁴ so FLIP2 trains on the five most common wild types and tests on 36 others.² It reports split means of 539.7, 531.7 and 534.4 nm; the companion recomputes 539.73, 531.73 and 534.43 nm from the bytes, an independent check on the parsing.

Those means being so close shapes everything else. The test median is 539.5 nm, the constant that minimises test mean absolute error, and the training mean of 539.7329 nm sits within 0.25 nm of it. A constant fitted on train is therefore within a quarter of a nanometre of the best possible constant on test, and any model has to earn its error reduction against that.

The bytes show one more thing. Six sequences appear twice, all inside train, and two pairs disagree: 37b98f5e1e0a carries 525 and 524 nm, bb35eb34d803 carries 498 and 517 nm. Whether those are repeated assays or repackaging artefacts is not stated upstream and was not established here, so 19 nm is a caution, not an assay precision estimate. That disagreement for an identical sequence is why the reporting threshold below is 2 nm rather than something tighter.

Those twelve-character identifiers are sha256(sequence)[:12], computed here. The archive publishes no identifiers, names or assay conditions, so they are local handles rather than accessions and they will not resolve anywhere.

The adaptation is deliberately frozen. Ridge alpha is 10 with no feature scaling, exactly as archived, a tenfold stronger penalty than the scikit-learn default,⁵ and target scaling uses train rows only. The ESM-2 checkpoints are esm2_t6_8M_UR50D and esm2_t12_35M_UR50D, read at layers 6 and 12 and pooled by mean over residues.⁶ Five of the six configurations are reimplementations of a historical panel and all five reproduce to absolute difference 0.0 at tolerance 1e-9 in a fresh Python 3.11.13 environment.

Rank correlation and nanometres rank the panel differently

Table 2. Six configurations on the 184 held-out rows of the FLIP2 Rhomax by_wild_type split, each with a frozen representation and a fixed alpha-10 train-only ridge head. Comparator for the error columns is the constant training-mean prediction; differences are candidate minus comparator, so positive is worse. Spearman is undefined for a constant predictor, because all 184 predictions tie; it is not zero. The rank metrics for the nearest-sequence control are new here and absent from the archived panel, which reported only the five historical configurations. Uncertainty: paired cluster bootstrap, 10,000 draws, seed 20261001, resampling 37 reconstructed background groups. The Spearman and NDCG columns carry no interval and must not be read as though they did.

Spearman and NDCG were reported upstream; nanometres were not. Every RMSE here is at or above the measured test standard deviation of 32.15 nm, so none of these configurations explains variance in any useful sense, and the best mean absolute error in the panel is 0.24 nm better than predicting one number for every protein.

Three verdicts sit in that last column and they stay apart for the rest of this article, because collapsing them is the main way a table like this gets misread. A supported difference has an interval excluding zero and a point estimate above the 2 nm threshold. A resolved direction too small to act on has the same interval property but a point estimate below it. No supported difference means the interval includes zero, which is a statement about what this split can resolve, not a claim that two configurations perform the same. Untested is a fourth thing: nothing here speaks to a fine-tuned model, a larger checkpoint, a structure-aware model or any background outside this archive, so none of those is known to fail.

The grouping caveat travels with every interval here. The archive publishes no background identity despite the split being named by_wild_type, so the resampling units were reconstructed from sequence length and Hamming distance under a rule fixed before any error was computed. There are 37 such groups, 22 of them singletons, the largest holding 30 rows, with about six carrying most of the weight. All 37 are used as resampling units, singletons included, and none are dropped: the group count is not a count of informative groups. Resampling whole clusters rather than rows is the standard procedure for grouped data, and the choice of resampling level is not cosmetic.⁷

Figure 2. The same six configurations ordered by rank correlation and by error in nanometres. Comparator: the constant training-mean prediction, drawn as the reference line at 24.60 nm on the error axis, with no defined Spearman. Population: the 184 held-out rows of the FLIP2 Rhomax by_wild_type split. Adaptation: frozen encoder, fixed alpha-10 train-only ridge head, no tuning. Lower is better on error, higher on Spearman. Supported uncertainty: only ESM-2 35M's error difference excludes zero and exceeds the 2 nm threshold, +4.56 nm [+0.80, +9.25] under the paired cluster bootstrap over 37 reconstructed groups; nothing in the Spearman panel carries an interval. Plotted from runs/results/panel-receipt.json; measurements originate with Inoue et al. 2021, CC-BY 4.0, via FLIP2.¹²

NDCG needs a scale before it means anything

All six NDCG values sit between 0.8965 and 0.9548, with the constant predictor in the middle at 0.9207, which invites the reading that everything here ranks competently. The scale was therefore measured rather than argued. On these exact 184 targets, with the same relevance transform, full ranking and tie rule as the historical metric, a perfect ranking scores 1.0000 and the worst possible ranking, exactly reversed, scores 0.8131. Ten thousand random orderings average 0.9207, with a 95% range of 0.8975 to 0.9403. That reference is post-hoc and single-split: a scale, not a test.

Read the panel against that scale and NDCG does order these configurations. Composition at 0.9548 is above every one of the 10,000 random draws, the nearest-sequence control at 0.9453 is above the 97.5th percentile, ESM-2 35M at 0.9073 falls inside the random range, and ESM-2 8M at 0.8965 is just below its 2.5th percentile. The constant's 0.9207 is exactly the random-ordering mean, which is what scikit-learn's default tie averaging gives when every prediction is identical.⁸ Wang et al. prove that log-discount NDCG converges to 1 as list length goes to infinity for any ranking function, and that it keeps consistent distinguishability all the same.⁹

What the measured range changes is how an absolute value reads. The achievable band on this list is 0.81 to 1.0, not 0 to 1, so 0.92 means "about what a random ordering gets" rather than "most of the way to perfect", and small gaps inside that band are hard to interpret. NDCG also measures ordering, not wavelength error, and the decision turns on nanometres. One detail is worth keeping straight: NDCG ties are averaged, while shortlist ties are broken by stable row order. Two tie rules for two different jobs.

Why the best rank correlation buys 0.24 nanometres

Table 3. Spread of predictions against the 32.15 nm measured spread of the 184 held-out rows of the FLIP2 Rhomax by_wild_type split, under a frozen encoder with a fixed alpha-10 train-only ridge head. Comparator is the constant training-mean prediction, whose spread is zero by construction and whose Pearson r is therefore undefined. These are descriptive statistics of the predictions and carry no interval; the supported uncertainty on the error differences they produce is in Table 2. Receipt: runs/results/diagnostics.json.

The composition probe's wobble is why +0.418 Spearman translates into 0.24 nm: the ordering is monotone enough to be worth something, while the magnitudes are almost all shrinkage. ESM-2 produces real variation, 42% to 49% of the measured spread, pointing the wrong way.

Figure 3. Why a rank correlation of +0.418 is worth 0.24 nm. Comparator: the constant training-mean prediction, which would be perfectly flat at 539.73 nm. Population: the 184 held-out rows of the FLIP2 Rhomax by_wild_type split. Adaptation: frozen encoder, residue-mean pooling, fixed alpha-10 train-only ridge head. Supported uncertainty: none in this figure, which is descriptive; the paired cluster-bootstrap intervals on the resulting error differences are in Table 2, where composition is −0.24 nm [−0.45, −0.05] and ESM-2 35M is +4.56 nm [+0.80, +9.25]. Plotted from runs/results/diagnostics.json; measurements originate with Inoue et al. 2021, CC-BY 4.0, via FLIP2.¹²

Part of this is an artefact of the archived protocol, not a result about model families. Alpha is fixed at 10 with no feature scaling, so the same nominal penalty is not the same shrinkage. Total feature variance divided by alpha is 0.00105706 for the 22-feature composition vector against 0.0817550 for ESM-2 35M, a ratio of about 77; for the 40-feature vector it is 0.00034495 against the same 0.0817550, about 237. The panel does not hold regularisation constant across representations, which is a limitation of the comparison.

The two "distinct" archived composition controls are also barely distinct. Twenty of the 40-feature vector's columns are reserved for a second colon-separated component Rhomax does not have, so they are identically zero, and the 22-feature vector adds only log1p(length). That term moves Spearman by 0.0002. They are one control reported twice.

Three decisions, three different answers

Capacity was fixed at 10 and both criteria were fixed before evaluation. Decision A is hitting a band, 540 to 560 nm, ranking by distance from 550 nm.

Table 4. Decision A at capacity 10 on the 184 held-out rows of the FLIP2 Rhomax by_wild_type split, frozen encoder with a fixed alpha-10 train-only ridge head. Comparator: uniform random selection of 10 rows. In-band prevalence is 57 of 184, so 0.3098 is the random expectation. The constant predictor has no shortlist, because a precision computed on a row-order list would describe the file rather than the model. The intervals are post-hoc and were not predeclared; they are wide because resampling reconstructed groups changes which backgrounds are present at all. Receipt: runs/results/exploratory-analysis.json.

Two readings sit together and both are honest. Composition's 9 of 10 is well above the 0.3098 random expectation, and it is sorting on a 2.2 nm signal, which is why the post-hoc interval reaches down to 0.33. The clearest post-hoc separation is ESM-2 35M at 0 of 10, upper bound 0.20, below random here; like every shortlist interval it is post-hoc, not a supported difference.

Decision B is maximising red shift, ranking by prediction descending.

Table 5. Decision B at capacity 10 on the same 184 held-out rows and the same frozen alpha-10 train-only adaptation. Comparator: uniform random selection of 10 rows, whose expected top-ten mean is 534.43 nm computed analytically from the measured targets; the constant has no shortlist. The oracle row is the best achievable, not a model. Intervals are post-hoc and were not predeclared, so composition's [546.0, 560.6], which excludes the random expectation, is a post-hoc lift rather than a supported difference under this protocol. Receipt: runs/results/exploratory-analysis.json.

That table needs both of its halves. Composition's shortlist averaged above random selection, 553.50 nm against 534.43 nm, and its post-hoc interval excludes that expectation, so nudging a shortlist's average towards the red is something this panel did. It also found none of the seven test rows at or above 580 nm: its best pick was 564 nm, against an oracle top-ten mean of 583.30 nm. For finding the extremes, nothing in this panel did better than chance.

Both ESM-2 red-shift shortlists contain 781ca2b1c04b, 350 residues, measured at 587 nm, with no equal-length training sequence. ESM-2 8M ranked it first at a predicted 578.98 nm; 35M placed it ninth at 562.94 nm. That is not skill. Seven of the 184 test rows sit at or above 580 nm, so a random shortlist of 10 contains at least one with probability 0.3283; for 585 nm or above it is 0.2018. One hit at one-in-three odds is a coincidence, and the rest of that shortlist makes the point: ESM-2 8M ranks 5e4e5b44d203 fifth at a predicted 571.15 nm, and that row is the bluest protein in the split, at 436 nm.

A different protein makes the point hardest. c37359b2e9ef, measured at 589 nm, is the most red-shifted row in the test set, sits 0.799 from the nearest equal-length training sequence, and appeared in no shortlist from any configuration. The most red-shifted protein available was selected by nothing. Do not read 587 and 589 as the same row: da9591a13be6 is a third row, a second at 587 nm, which nothing surfaced either.

Figure 4. The same panel scored against two different objectives. Comparator in both: uniform random selection of 10 rows, 0.3098 precision for Decision A and an expected top-ten mean of 534.43 nm for Decision B; the constant has no shortlist under either. Population: the 184 held-out rows of the FLIP2 Rhomax by_wild_type split. Adaptation: frozen encoder, fixed alpha-10 train-only ridge head. Supported uncertainty: none. Every interval shown is post-hoc and not predeclared, including composition's [546.0, 560.6], which excludes the random expectation on Decision B without making that a supported difference. Decision B uses points rather than bars because its axis does not start at zero. Measurements originate with Inoue et al. 2021, CC-BY 4.0, via FLIP2.¹²

Failure modes: distance to the training set

Distance is length-gated normalised Hamming: equal length gives Hamming over length, unequal gives 1.0. It is crude because lengths differ by up to 138 residues and no alignment policy had been fixed. Bins were frozen before any error was computed.

Table 6. MAE in nm per frozen distance bin, 184 held-out rows of the FLIP2 Rhomax by_wild_type split, frozen encoder with a fixed alpha-10 train-only ridge head, comparator the constant training-mean column. Every per-bin value here is descriptive. No paired interval was computed inside any bin, including the 87-row and 92-row bins, so no between-configuration difference inside a bin is a supported difference anywhere in this article. The protocol's n ≥ 10 rule is a minimum size below which no claim may be made, not a licence to claim above it. The two composition variants are identical to two decimal places in every bin. Receipt: runs/results/exploratory-analysis.json.

Only 5 of 184 test rows sit in the regime where a close equal-length homologue exists, and in exactly that regime the trivial lookup is far the best thing in the panel at 6.60 nm. With n = 5 and no interval that is a signpost rather than a difference claim, but it is the signpost worth following: look for a close equal-length relative before you train anything.

The mid bin is empty. There is no gradient of moderately similar backgrounds here: either a near-identical training sequence exists or nothing close does. Nothing in this panel was ever observed at intermediate distance, so that regime is untested. Untested is not measured failure, and it is also not reassurance.

The 92-row bin is the heart of the problem. Those rows have no training sequence even of the same length, and on all 92 the nearest-sequence control falls back to its documented default and emits exactly 539.732877 nm, the training mean to the digit. The two are therefore the same predictor on half the test set, which is why their MAE in that bin is identically 24.97 nm. On half the held-out rows the interpretable control has nothing to interpolate from. One detail matters for your own data: this bin is defined by the absence of a neighbour, not by a distance of 1.0. The largest equal-length nearest distance on test is 0.9375, so no row here is affected, but a sequence you supply could be.

Six held-out rows show what the bins hide.

Table 7. Six of the 184 held-out rows of the FLIP2 Rhomax by_wild_type split, from runs/results/test-predictions.json, with the configuration named per prediction column; the frozen-encoder columns use the fixed alpha-10 train-only ridge head, and the comparator for every row is the constant training-mean prediction of 539.73 nm. The first five are the entire near bin. The last is the 587 nm row both ESM-2 shortlists surfaced; it has no equal-length training sequence, so the nearest-sequence control falls back to the training mean of 539.73 nm. These are illustrations of individual rows and carry no uncertainty: none may be generalised into a claim about a configuration. Identifiers are workspace-local sha256(sequence)[:12] digests, not accessions. Measurements originate with Inoue et al. 2021, CC-BY 4.0, via FLIP2.¹²

The near bin also shows why the length-gated distance flatters nobody. Row 1126648311d3 has as its nearest training sequence 37b98f5e1e0a, one of the two duplicated training sequences carrying disagreeing targets, 525 and 524 nm; the lookup returned 525.0 and the measurement is 523. The three rows nearest to 18414205e448 all got the same 551.0 prediction against measurements of 558, 557 and 548.

Figure 5. Error by distance to the training set. Comparator: the constant training-mean prediction, plotted as its own series. Population: the 184 held-out rows of the FLIP2 Rhomax by_wild_type split, as 5 near rows, an empty mid bin, 87 far rows and 92 rows with no equal-length training sequence. Adaptation: frozen encoder, fixed alpha-10 train-only ridge head. Uncertainty: none; every value plotted is descriptive, no within-bin interval was computed, and the near bin holds 5 rows. Measurements originate with Inoue et al. 2021, CC-BY 4.0, via FLIP2.¹²

There is a biological reason not to expect a composition or nearest-neighbour feature to carry this load. Blue- and green-absorbing proteorhodopsins share more than 78% of their residues while absorbing 35 nm apart at 490 and 525 nm, and most of that difference is attributable to a single residue at position 105 whose swap almost completely interconverts the two spectra.¹⁰ Colour tuning is also complex: in the crystal structures, water-containing hydrogen-bonding networks act as complex counterions to the protonated Schiff base.¹¹ A composition vector throws away position entirely, and a Hamming distance over unaligned sequences of different lengths is barely a homology measure.

What an interval costs

Half-widths were set at the 90th percentile of absolute residuals on the 116 validation rows the historical protocol never read, then applied to test. The probe is post-hoc and was not predeclared.

Table 8. Post-hoc interval feasibility probe, not predeclared and not a calibration result, on the 184 held-out rows of the FLIP2 Rhomax by_wild_type split with a frozen encoder and a fixed alpha-10 train-only ridge head. Comparator is the constant training-mean prediction at 37.7 nm half-width and 0.772 coverage. Widths come from the 116 otherwise unused validation rows; coverage is observed on test. The Wilson interval excludes the nominal 0.90 for all six configurations. Receipt: runs/results/exploratory-analysis.json.

Two things follow and nothing more. A 90% interval needs about 38 nm of half-width for the best configuration, which spans most of the useful range of this endpoint, and it still covers only about 78% of test rows, Wilson excluding 0.90 for all six. Split conformal and its relatives guarantee coverage under exchangeability,¹² which is exactly what a by-wild-type transfer split is designed to break; repairing it needs explicit shift weighting or a nonexchangeable method.¹³ No calibration claim is made: checking coverage under background shift would need several independent held-out background sets, and the archive publishes no background labels.

Measured cost

Across three complete runs on an Apple M4 with 16 GB, on CPU, batch size 4, 4 torch threads, torch seed 0, the whole panel took 49 to 52 s of wall time and 1.37 to 1.48 GB of peak RSS. In the run whose receipt ships with this article, ESM-2 35M encoding of all 884 sequences took 35.2 s and ESM-2 8M 13.2 s, against 0.004 s for the composition features and 0.0001 s for the constant. One machine, one batch size, so not a hardware-normalised benchmark. The accelerator does not help at this scale: on 40 sequences with ESM-2 8M, MPS took 2.22 s against CPU's 0.55 s.

So the cost framing is not that ESM-2 is expensive. Half a minute of encoding is nothing. The cost is the engineering time around it, and on this split that buys a configuration measurably worse than a constant.

Reproduce it

The companion ships as two archives linked from this post: code (rhomax-wavelength-panel-code.tar.gz, no data or weights) and run receipts with per-row predictions (rhomax-wavelength-results.tar.gz, no sequences or weights). Extract the code archive and work inside it. Setup needs Python 3.11 and about 160 MB of downloads.

cd companion
uv venv --python 3.11 .venv
VIRTUAL_ENV=.venv uv pip install -r requirements-locked.txt
VIRTUAL_ENV=.venv uv pip install -e . --no-deps
mkdir -p cache

curl -L -o cache/by_wild_type.csv.gz \
  "https://zenodo.org/api/records/18433203/files/rhomax/by_wild_type.csv.gz/content"
curl -L -o cache/esm2_t6_8M_UR50D.pt \
  https://dl.fbaipublicfiles.com/fair-esm/models/esm2_t6_8M_UR50D.pt
curl -L -o cache/esm2_t12_35M_UR50D.pt \
  https://dl.fbaipublicfiles.com/fair-esm/models/esm2_t12_35M_UR50D.pt

Every input is hash-checked, and the run refuses to proceed on a mismatch rather than quietly scoring a different split. The decompressed CSV must hash to 8e78f6a16cd5298131dca83130d4b94cc0ab4ae9c699460880441c19a65f656f. That hash defines the dataset, not the archive hash, because the FLIP mirror serves the same CSV inside different gzip framing.

Then the panel, the analysis and the diagnostics. The --skip-esm line reproduces three of the five archived configurations with no weights:

rhomax-panel --cache ./cache verify
rhomax-panel --cache ./cache panel --output ./results --skip-esm
rhomax-panel --cache ./cache panel --output ./results
rhomax-panel --cache ./cache analyse --output ./results --baseline training-mean
python scripts/diagnostics.py --cache ./cache --results ./results --output ./results/diagnostics.json
python scripts/device_agreement.py --cache ./cache --model esm2_t6_8M_UR50D
python -m pytest tests/ -q

Expect 584 train, 116 validation and 184 test rows; Spearman +0.41798958 for composition-22, +0.41818225 for composition-40, −0.14635068 for esm2-8m and −0.22175950 for esm2-35m; and NDCG 0.92066675 for training-mean. panel exits non-zero if any configuration fails to match its archived Spearman and NDCG at 1e-9, so silent drift cannot pass. The suite is 44 tests, of which 43 run and 1 skips when no completed panel run is present, aimed at the ways this code could produce a plausible but wrong wavelength: a substituted data file, wrong split counts, a leaked label, a constant scored as though it had a rank correlation, NDCG tie handling, and a tuned ridge masquerading as a historical configuration.

Scoring your own candidates is unlabelled inference, not held-out evaluation, and the CLI keeps these apart:

python scripts/make_example_fasta.py --cache ./cache --output ./example.fasta
rhomax-panel --cache ./cache predict ./example.fasta --configuration composition-22

Input rules are enforced: the 20 standard amino acids only, and 1 to 1022 residues with longer sequences rejected and never cropped. The output gives, per sequence, the predicted wavelength in nm, whether it falls outside the training range, and the distance to the nearest training sequence. Read that last column against Table 6 before you believe the first.

What this comparison does not establish

It is one split, one pooling, one frozen adaptation and one fixed head. It is not evidence about ESM-2 in general, nor about fine-tuned protein language models, which were not run. Frozen embeddings do beat one-hot baselines elsewhere: on catalytic turnover across adenylate kinase orthologs in a low-data regime, nonlinear probing of frozen embeddings outperformed one-hot, and learnable aggregation was the most performant pooling, though only marginally ahead of common fixed pooling.¹⁴ A fixed residue mean into a linear ridge is neither of those choices. That same study found earlier layers no better than the final one, so the final-layer read used here is not an obvious culprit, and nothing in it establishes that a different pooling would have rescued these embeddings on this task.

Untested transfer is not measured failure. No fine-tuned model, no larger model, no structure-aware model and no model on a background outside this archive was run, and none of those is therefore known to fail. ESM-2 runs from 8 million to 15 billion parameters, with capability described by its authors as emerging with scale;¹⁵ these are its two smallest checkpoints, and the 35M was chosen after the 8M result was known, so that pair is exploratory rather than a preregistered scale experiment.

Other things not established. The reconstructed grouping is not the archive's, so repeated held-out-background cross-validation was not attempted. Calibration under background shift is not claimed. An aligned one-hot ridge was not attempted, because no reproducible alignment and insertion policy was fixed. FLIP2's published baselines are not comparators here: its Ridge one-hot scores 0.327 Spearman with NDCG 0.939,² but this workspace's record of the pinned upstream baseline code shows it scales targets on train plus validation rows while every configuration here scales on train alone, so treat 0.327 as context rather than a score to beat. Its model tables also conflict: Table A17 lists CARP-640M supervised at 0.379 Spearman as the best baseline on this split, while Table A15 reports it at 0.072 ± 0.244.² Whether any of these proteins sit in ESM-2's UniRef pretraining corpus is unreported upstream and was not determined here. In one study of melting-point prediction, pretraining leakage consistently inflated measured performance on the original FLIP split relative to a pretraining-aware split; nothing comparable has been measured for Rhomax.¹⁶ No human review of this panel has taken place, and automated review is not human review.

What would change the recommendation

Three results would move Table 1.

A learnable aggregation over the same frozen embeddings, under the same train-only protocol and cluster bootstrap, is the first thing to run: it tests whether pooling or representation failed here.¹⁴ The encoding pass costs about 35 s on the CPU measured above, and beating the constant by more than 2 nm with an interval excluding zero would rewrite the table's first row.

An alignment-based representation, with its policy fixed before any error is computed, would address the objection that a composition vector cannot see a single switch residue: mutations at position 105 almost completely interconvert the blue- and green-absorbing proteorhodopsin spectra.¹⁰ One-hot encodings over aligned positions underlie the VPOD opsin wavelength models, down to 6.56 nm mean absolute error, albeit with far more and more diverse training data than five backgrounds.¹⁷ On this split that approach is untested, not refuted.

Published background labels would change the most. With real wild-type identifiers the reconstructed grouping disappears, repeated held-out-background cross-validation becomes possible, and the intervals in Table 2 could narrow enough to separate what this analysis leaves unresolved.

Until then the defensible position is narrow and usable. Against a constant training-mean prediction, on the 184 held-out rows of the FLIP2 Rhomax by_wild_type split, with a frozen encoder and a fixed alpha-10 train-only ridge head: the composition probe is better by 0.24 nm, 95% paired cluster bootstrap over 37 reconstructed groups [−0.45, −0.05], below the 2 nm threshold anyone would act on, and frozen ESM-2 35M residue-mean embeddings are worse, +4.56 nm [+0.80, +9.25]. If a close equal-length relative exists, look it up: on the 5 such rows the lookup scored 6.60 nm against the constant's 16.16 nm, descriptive only, with no interval. For a point estimate, quote the constant's post-hoc half-width of 37.7 nm, which covered 77.2% of test rows against a nominal 0.90, Wilson [0.706, 0.826], and decline to call it calibrated. For a 540 to 560 nm band, shortlist with the composition probe, which put 9 of 10 in band against 0.3098 for uniform random selection, and read its post-hoc interval [0.33, 1.00] first. None of this speaks to activation efficiency, expression, photostability, ion transport or cellular function.

The field-level context agrees. FLIP2 reports that fine-tuned protein language models were best on only 4 of 16 splits and do not consistently improve on simpler methods for wild-type and position splits,² and a 2026 preprint reports that one-hot encoding matched or outperformed the best protein language model in nearly all the settings it tested.¹⁸ That is not a reason to stop using these models. It is a reason to put a constant predictor in every comparison table, and to report error in the units the experiment is measured in.

References

Artifacts

Receipts for every measured number above: runs/results/panel-receipt.json (the five historical configurations and the bit-exact reproduction check), runs/results/exploratory-analysis.json (distance bins, paired bootstrap, both shortlist decisions, the interval probe), runs/results/diagnostics.json (prediction spread, regularisation, chance levels), runs/results/device-agreement.json (the CPU and MPS known-answer check), runs/results/test-predictions.json (per-row predictions in nm with workspace-local seq_id values, which are sha256(sequence)[:12] digests and not upstream accessions) and runs/results/ndcg-finite-reference.json (the measured NDCG scale). The analysis protocol, including the frozen distance bins, the k = 10 shortlist criteria and the 2 nm reporting threshold, was fixed before any test error was computed; post-hoc additions are labelled post_hoc_not_predeclared in the output files and as post-hoc in the text above. Charts are plotted by companion/scripts/make_charts.py, which reads only those result files. The FLIP2 data is CC-BY 4.0 and attributed to Inoue et al. 2021; the companion downloads it rather than republishing it. No human review of this analysis has taken place.