Sentiment classifier

Three ways to tell a good movie review from a bad one, and why the two transformers lost to the simplest model until they were allowed to read the ending.

Shipped — Finished, or finished enough that the write-up is about what it does. 2025

Try the demo Source

87.5%
DistilBERT, reading the ending: 75.5%
MiniLM, whole review: 84.5%
TF-IDF baseline

What it does

The same 2,000 movie reviews classified three ways: TF-IDF with logistic regression, frozen MiniLM embeddings with logistic regression, and a fine-tuned DistilBERT. The simplest model won by 20 points, because both transformers were reading only the first 256 tokens of reviews that run to about 800, and reviewers tend to give their verdict at the end. Letting them read the end took DistilBERT from 73.0% to 87.5%, ahead of TF-IDF.

Classifies a review as positive or negative: Three models trained and tested on the same split of NLTK's movie_reviews: 1,600 reviews to train on, 400 held out, stratified, one seed. Each reports accuracy, F1 and ROC-AUC, a confusion matrix, and every review it got wrong.
Explains what the simple model learned: TF-IDF with logistic regression is readable: its strongest words are bad, worst, boring and great, excellent, best. They also include film titles like truman and bulworth, which means it partly learned which films people liked, and that is worth knowing before trusting it on new ones.
Lets the transformers read the whole review: MiniLM now embeds every 254-token chunk of a review and averages them; DistilBERT keeps the opening 128 tokens and the closing 382. Nothing else changed: same split, same seed, same classifier settings, same two epochs.
Scores any text from the command line: predict.py runs one model or all three side by side on a sentence, a file, or standard input, reading the text the same way each model was trained.

Demo

What each model read

Five reviews, five versions of the models. Pick a review, then pick a model: whatever it never read is greyed out. The first versions of the transformers stop at 256 tokens.

Long, opens sounding negative, and gives its verdict at the end. Written to read as positive

Showing what TF-IDF + logistic regression reads

I nearly walked out of The Lighthouse Keeper's Daughter in its first twenty minutes, and I want to be honest about that before I say anything else. The opening is slow in a way that feels almost hostile to the audience. We watch a woman repaint the same railing three times. We watch her eat dinner alone, twice. The dialogue, when it finally arrives, is clipped and strange, and the score is a single cello that seems to have wandered in from a different film. The man next to me checked his phone four times. I understood why. The setup is simple enough. Maren has kept the light on a small island off the northern coast since her father died, and the shipping company that owns it has decided to automate it. A young engineer named Tobias is sent to install the new equipment and, without anyone saying so, to close the place down. He arrives in a storm, he is seasick, and he brings the wrong tools. For a long stretch the film is about two people who do not like each other being stuck in a very small house, and it does not hurry to make them like each other. What the film is doing in that stretch only becomes clear later. Every repeated task, every meal, every argument about whether the lamp needs oiling, is building a picture of a life that runs on routine because routine is the only thing holding it together. When Tobias starts to learn the routines himself, badly at first, the film quietly changes shape. The railing scene comes back. So does the dinner. The second time, they mean something completely different, and I did not see it coming. The two leads are excellent. Ida Varga plays Maren with almost no visible effort, which is exactly the point: this is a woman who has trained herself not to show anything, and Varga lets the cracks appear in her hands and her posture long before they reach her face. Tomas Brede has the harder job, because Tobias begins as an irritating city boy and has to earn our sympathy without the script ever announcing that he has changed. He manages it in small moments, like the way he starts closing the door gently once he knows how loud it is in the night. The island itself is photographed beautifully but never prettily. The light is cold and flat, the sea is grey, and the house is cramped and damp. When the weather finally clears, about two thirds of the way through, the change is so sudden and so bright that the audience around me actually gasped. It is one of the best uses of weather I have seen in a film, because by that point we have been living in the fog with these two people and we feel the relief in our bodies. There are flaws. A subplot about Maren's brother on the mainland goes nowhere, and a late scene with a company inspector is played too broadly, as if the film suddenly worried that we would not understand who the villain was. The cello score, which I found grating at the start, never entirely won me over, although I will admit it works in the final scene. And that final scene is the reason to see this. I will not describe it, except to say that it is almost silent, it is about the lamp, and it made me understand everything that came before it. The slowness I nearly walked out on turns out to be the whole point. By the end I was grateful for every minute of it. This is a patient, generous, deeply moving film, one of the best I have seen this year, and I would gladly sit through those first twenty minutes again.

Each version's verdict

Written for this page, not taken from the dataset. The scores are the trained models' real outputs, computed once and stored here: the transformers are far too large to run in a browser.

How it works

Why the transformers lost

The reviews are long: 797 tokens at the median and 1,312 at the 90th percentile. MiniLM reads at most 256 tokens, and the DistilBERT script truncated at 256 too, so 98.8% of reviews were cut, and each transformer saw 36% of an average review. The part it never saw is where a reviewer usually says what they thought. TF-IDF reads every word, so the oldest and simplest model won by 20 points.

The first write-up claimed MiniLM was the more accurate model, and its own results said the opposite: 64.8% against 84.5%, with a McNemar test putting the gap well beyond chance.

What changed

Only how much of each review the models read:

Model	First version	Revised	Reads
TF-IDF + logistic regression	84.5%	84.5%	every word
MiniLM + logistic regression	64.8%	75.5%	every 254-token chunk, averaged
DistilBERT, fine-tuned	73.0%	87.5%	the first 128 and last 382 tokens

Is it the extra length, or the ending? A third DistilBERT run spent the same 510-token budget on the first 510 tokens and scored 81.5%. So doubling what it reads is worth about eight points, and spending part of it on the ending is worth six more: 50 reviews right only with the ending against 25 right only without it, an exact McNemar p of 0.005.

MiniLM is still behind after the fix. A frozen sentence encoder averages away the shape of an argument, and standardising its features with a cross-validated penalty did not help (75.25%).

The face-off

The five reviews above were written for this page, about films that do not exist, to put the finding in front of the reader. The letdown praises its cast for half its length and turns at the end: every model that cannot see the end calls it positive. The revised DistilBERT is not perfect either. It calls the two-sentence review negative, by a thousandth.

Testing

400 held-out reviews, one fixed split

Every model is scored on the same 400 held-out reviews, never seen in training, and the comparisons are paired: the same reviews, so a difference can be tested rather than eyeballed. The first-version results were reproduced exactly before anything was changed.

What the tests defend

DistilBERT's lead over TF-IDF is not statistically significant on this test set: 40 reviews are wrong only for TF-IDF and 28 only for DistilBERT, and an exact McNemar test gives p = 0.18. The page says ahead, not better.
The ending is tested, not assumed: with the same 510-token budget spent on the first 510 tokens, DistilBERT scores 81.5%. Keeping the ending instead fixes 50 reviews and breaks 25 (exact McNemar p = 0.005).
Each number is one run with one seed on one split, which is the honest limit of what it shows.

Built with

Models: scikit-learnSentence-TransformersHugging Face TransformersPyTorch
Data: NLTK movie_reviewspandasNumPy
Evaluation: accuracyF1ROC-AUCMcNemar's test