4 articles tagged with #reproducibility.
Two H100s on different clouds gave bit-identical Evo 2 scores for 3,893 BRCA1 variants. An L40S running the same container reshuffled two thirds of the ranking. Running the model in FP32 brought the two GPUs back into agreement, at three to six times the compute.
In this exploratory study, with annotation, reference genome and variants matched, SpliceAI and Pangolin put 63 to 66 assay-confirmed splice disruptions in their top 100 of the same 8,297 held-out MFASS variants. Masking raised average precision slightly for both tools; no top-100 difference was established.
A recall reversal and two constant-prediction controls show why benchmark comparisons need populations, selection rules and reference scores.
One pretrained genomic model can look dominant on a leaderboard and lose to a linear baseline on the next task. Two evidence ledgers explain why.
