Picking Variants for Minigene Splicing Tests: SpliceAI, Pangolin and a Simple Baseline Compared

Suppose you have a list of human single-nucleotide variants and the budget to test 100 of them in a minigene splicing assay, a lab test that places each variant in a small engineered gene and measures whether cells still splice it normally. You want as many of those 100 as possible to be variants that really disrupt splicing. Which score, meaning the number a method gives each variant so that the list can be sorted, should you use to rank the list: a splicing predictor such as SpliceAI or Pangolin, or a simple model trained on earlier results from the same assay?

On one large public assay, the answer at 100 experiments is that the evidence does not separate them. This post explains that result, why a tool can rank a whole list better and still not win at a fixed budget, and where the limits are. The full methods, tables and code are in the study repository, which also holds the original detailed article.

What a minigene measures

A minigene is a small engineered gene. The exon being tested sits between two flanking exons, and a readout reports whether the cell keeps that exon when it splices the RNA. A variant that makes the cell skip the exon has disrupted splicing.

The data come from MFASS (Chong et al., Molecular Cell, 2019). Each variant was placed in a 170 base-pair (bp) construct, a natural exon of under 100 bp with some flanking intron, inside a three-exon reporter integrated once per cell in HEK293T, a standard human cell line. Sorting cells by fluorescence gave an exon inclusion index. A variant counted as splice-disrupting if it lowered that index by at least 0.5 in an exon whose index without the variant was at least 0.5. Of 27,733 tested variants from ExAC, a public catalogue of human genetic variation, 1,050 (3.8%) were disrupting, and 83% of those lay outside canonical splice sites.

Figure 1. Look at panel e: the MFASS reporter, the sorting of cells into fluorescence bins, and the threshold used to call a variant disruptive. Panels a to d show a different assay (Vex-seq), and panel f shows other prediction tools on MFASS, not SpliceAI or Pangolin; neither is needed here. Source: Cheng et al., Genome Biology 2019, Fig. 2, CC BY 4.0, unmodified.

What a score means here

Each method gives every variant a number. For SpliceAI, the matched study used the largest of its four predicted changes in splice-site use. For Pangolin, it used the largest absolute predicted change across genes and sites. The simple baseline, called kmer_cons, is a gradient-boosted tree model trained on 19,409 MFASS variants. It uses short sequence patterns around the variant, its position in the construct and conservation scores.

For a fixed budget only the order matters. You sort the list by score and test the top 100. The useful number is then how many of those 100 turn out to be disrupting, called precision at 100. Picking 100 variants at random would give about 4.

The main result: 61 against 63 to 66 per 100

The comparison used 8,297 variants held out from training (314 of them disrupting) that every method could score. Grouping kept variants from the same exon and gene together, so the baseline was never tested on an exon or gene it had trained on.

In the first 100, the baseline found 61 disrupting variants. SpliceAI found 63 or 65, depending on a masking setting (an option in both tools that ignores predicted gains at splice sites already in the gene annotation and predicted losses at sites not in it), and Pangolin 65, or 65 to 66 with masking on, where three tied scores compete for the last two places. A simple rule based only on distance to the nearest exon boundary found about 13.

Are 63 to 66 better than 61? The data cannot tell. Take unmasked Pangolin as a worked example. Its first 100 held 4 more disrupting variants than the baseline's: a difference in precision of +0.040. The 95% interval for that difference, from resampling whole exon and gene groups, is [−0.038, +0.120]. In plain terms, the data are consistent with Pangolin finding about 4 fewer to 12 more disrupting variants per 100 than the baseline. Across all four SpliceAI and Pangolin settings, the intervals together run from about 7 fewer to 13 more. Every one includes zero.

That is an inconclusive result, not evidence that the methods are equal. A real gain of 10 per 100 and a small loss are both still plausible. These intervals are exploratory and not adjusted for the many comparisons made (see Limits).

Figure 2. Each marker is a predictor's difference from the baseline in the share of disrupting variants among the first k picked (k = 25, 100 or 300); each horizontal line is its 95% interval. A line that crosses the vertical zero line means no difference was established. Only the two Pangolin lines at 300 stay to the right of zero, and both 25 and 300 were chosen after the results had been seen. The fourth panel repeats the comparison at 100 against a version of the baseline with retuned settings. "Realised depth" is the range of list lengths used when resampling. S0 and S1 are SpliceAI with masking off and on; P0 and P1 are Pangolin with masking off and on. Source: companion run outputs in the study repository.

Why ranking the whole list is a different question

Two other common metrics score the order of the entire list, not just the top. AUROC (area under the receiver operating characteristic curve) is the chance that a randomly chosen disrupting variant is ranked above a randomly chosen non-disrupting one. Average precision (AP) averages the precision at each point in the ranked list where a disrupting variant appears. On both, Pangolin did better in exploratory, unadjusted comparisons. Unmasked Pangolin's AP was 0.389 against the baseline's 0.287, a difference of +0.102 [+0.061, +0.142], and its AUROC was 0.876 against 0.778.

The two measures can disagree because the first 100 picks are only about 1% of the list. For every method, most hits in the first 100 lay within 10 bases of an exon boundary: 51 of the baseline's 61, and 57 of unmasked Pangolin's 65. No method put any of the 30 disrupting variants lying more than 30 bases from a boundary in its first 100, although 173 of the 314 disrupting variants lie more than 10 bases out. So every method ranks most disrupting variants below its first 100, and the whole-list measures score that part of the ranking too. Within each distance band, unmasked Pangolin's AUROC (0.84 to 0.90) was higher than the baseline's (0.75 to 0.83); no intervals were computed for the bands.

Pangolin's advantage shows further down the list. With a budget of 300, the baseline would find about 100 disrupting variants and the two Pangolin settings about 137 and 145, and those difference intervals exclude zero. Most of Pangolin's extra finds there lie 3 to 30 bases from a boundary. But 300 was chosen after looking at the curve below, so this is a lead to test on new data, not a finding.

Figure 3. Share of the first k picks that are disrupting, for budgets from 10 to 800 (higher is better). The lines overlap from about 25 to 150; from about 200, the two Pangolin lines are highest. Vertical bars show how much tied scores can move a result. Source: companion run outputs in the study repository.

Try it

The companion code reproduces the comparison on an ordinary CPU; it was tested on an Apple M4 with 16 GB of memory. It needs uv 0.8 or later. It downloads about 82 MB of pinned public inputs and checks each file's checksum. Download splice-shortlist.zip (the same code is in the study repository's companion/ folder). Then run:

unzip splice-shortlist.zip
cd splice-shortlist
uv sync --frozen
uv run --frozen python splice_shortlist.py fetch
uv run --frozen python splice_shortlist.py build
uv run --frozen python splice_shortlist.py controls
uv run --frozen python splice_shortlist.py evaluate --budget 100 --method P0 --out out

Fitting the baselines takes about 30 seconds, and the evaluation about a minute. The last command prints the comparison table and ends with:

wrote out/shortlist.csv (100 rows for budget 100; 1 tied at the cutoff), excluded.csv, metrics.json

shortlist.csv is the 100 variants that unmasked Pangolin would send for testing. Change --budget to your own number of experiments, between 1 and 8,297. The companion scores only MFASS variants; for your own list, apply the same checks with your own tools. The SpliceAI and Pangolin scores are read from saved files; running those tools yourself is a separate, much longer route, described in the study repository.

Limits

• Exploratory. The test outcomes had been inspected in earlier work, the intervals are not adjusted for multiple comparisons, and the 300-variant budget was chosen after the results were seen. A comparison fixed in advance on new variants is still needed.
• Saved-score replay. SpliceAI and Pangolin were not rerun for this post. Their predictions come from an earlier matched study that recorded about 14 hours of wall time on an Apple M4 CPU to produce them.
• Labels needed for the baseline. The baseline only works if you have thousands of labelled results from the same construct design. Without them, a splice predictor is the practical starting point.
• One assay, one list. The 100 and 300 budgets are about 1.2% and 3.6% of this particular list, in which 3.8% of variants disrupt splicing. A different list or assay needs its own check.
• Not a clinical prediction. MFASS measures exon inclusion in an artificial reporter in one cell line. In full-length genes the MFASS authors confirmed 13 of 19 tested variants. As the CAGI5 assessors put it, predicting a splicing assay "cannot directly imply clinical significance".
• Missing variants. 27 held-out variants could not be scored by the predictors, mostly because of a genome-coordinate conversion problem that has not been confirmed by the MFASS authors.
• Review and disclosure. The matched study and the companion code come from this site's own benchmark project. An AI coding agent (Claude) ran the companion commands and drafted this article. The work was checked only by automated AI agents (Claude and Codex), not by independent human scientists. The LaTeX paper and compiled PDF report the original results. Independent scientific reproduction remains pending.

Sources and artifacts

• Study repository with the original detailed article, companion code and completed paper: rewire-bio/splicing-model-selection (original detailed article).
• Chong R, Insigne KD, Yao D, et al. A multiplexed assay for exon recognition reveals that an unappreciated fraction of rare genetic variants cause large-effect splicing disruptions. Mol Cell 2019;73(1):183–194.e8.
• Jaganathan K, et al. Predicting splicing from primary sequence with deep learning. Cell 2019;176(3):535–548.e24.
• Zeng T, Li YI. Predicting RNA splicing from DNA sequence using Pangolin. Genome Biol 2022;23:103.
• Cheng J, Nguyen TYD, Cygan KJ, et al. MMSplice: modular modeling improves the predictions of genetic variant effects on splicing. Genome Biol 2019;20:48.
• Mount SM, Avsec Ž, Carmel L, et al. Assessing predictions of the impact of variants on splicing in CAGI5. Hum Mutat 2019;40(9):1215–1224.
• rewire-benchmarks at 093fd1a: annotation-matched SpliceAI and Pangolin study.
• Companion code: splice-shortlist.zip.