A DistilBERT fine-tune that measures tone, and says so
A from-scratch DistilBERT fine-tune for 3-way financial-sentence sentiment on the Financial PhraseBank, ONNX-served. Macro-F1 0.960 with bootstrap CIs, and an explicit refusal to call it alpha.
00Overview
Not every project is a null. This one trains a real model well and measures it honestly: a DistilBERT fine-tune for 3-way financial-sentence sentiment, scored with macro-F1 and bootstrap confidence intervals against two baselines, and served without torch in the container.
Macro-F1 against baselines
Results
01Method
- DistilBERT fine-tune, 3 epochs, early-stop on validation macro-F1.
- Financial PhraseBank, the sentences_allagree subset (highest annotator agreement).
- ONNX int8 export, served by onnxruntime and tokenizers (torch-free container).
- Bootstrap 95% confidence intervals and a McNemar paired test against the lexicon baseline.
- A class-stratified group split by normalised-sentence hash, so near-duplicates cannot leak across folds.
02The honest framing
The model reports macro-F1 rather than accuracy because the neutral class is ~61% of the data, which makes accuracy flattering. It is anchored against a class-prior floor (0.254 macro-F1) and a lexicon baseline (0.653). Per-class test F1 is 0.93 / 0.99 / 0.96. The headline caveat is stated plainly: sentiment is a text label, not a tradable signal, and no alpha is claimed.