6 articles tagged with #benchmarks.
On 8,297 held-out MFASS variants, a supervised baseline put 61 splice-disrupting variants in its first 100 and four SpliceAI and Pangolin configurations put 63 to 66. No difference at 100 was established. Pangolin's higher whole-list scores are exploratory and unadjusted. Minigene exon recognition only.
In this exploratory study, with annotation, reference genome and variants matched, SpliceAI and Pangolin put 63 to 66 assay-confirmed splice disruptions in their top 100 of the same 8,297 held-out MFASS variants. Masking raised average precision slightly for both tools; no top-100 difference was established.
A recall reversal and two constant-prediction controls show why benchmark comparisons need populations, selection rules and reference scores.
The baseline, SpliceAI and Pangolin found 61, 64 and 65 disruptions in their top 100 MFASS variants. The differences do not establish a reliable winner.
Vision-language models can describe a scene in paragraph-length detail and still fail to tell you whether a red cube is in front of or behind a blue cylinder.
Understanding when you're working with an environment versus a benchmark changes how you design experiments, interpret results, and communicate findings. This guide covers the practical differences every RL practitioner should know.
