Haiku 5.5: Cheaper by the Token
Table of Contents
In AI Scraping on the Cheap I talked about using Haiku as a workhorse model to extract sports events out of HTML. Since I saw Haiku 5.5 available today, I decided to have Claude give it a spin on a few event pages and do some before/after analysis.
The TL;DR is that 5.5 is faster and cheaper. There is, however, a difference in how it parses out events. For one event in particular, 4.5 returns the main 5k run as well as a kids fest and a charity walk. 5.5 omits these, but includes them as tags in the main event, which for my setup is arguably the more correct approach. I’ll have to live with this for a bit to see how I like the results, but so far it’s encouraging. I’ll let Claude explain the rest and you can decide for yourself whether you want to believe it.
"Gold Medallion with a portrait of Olympias, mother of Alexander the Great, Roman (Aboukir, Egypt) 242-243 CE" by Mary Harrsch is licensed under CC BY-NC-SA 2.0.
#The setup
My initial question was regarding how Haiku 5.5 could be cheaper if it uses more input tokens for the same task.
#Claude’s report
Contains LLM-generated content not written by Olaf
Yes, Haiku 5.5 uses more tokens, but it’s still much cheaper. Each token costs a tenth as much, which more than covers the extra tokens. Totals for the six event pages that succeeded on both models:
| Haiku 4.5 | Haiku 5.5 | Change | |
|---|---|---|---|
| Input tokens | 35,284 | 45,527 | +29% |
| Output tokens | 7,942 | 7,991 | about the same |
| Price per million tokens, input / output | $1.00 / $5.00 | $0.10 / $0.50 | 10× lower |
| Cost for the six pages | $0.075 | $0.0086 | about 8.8× cheaper |
| Latency | 3.8–18.8s | 2.4–8.4s | about 2–4× faster |
[Note: by my arithmetic, the Haiku 5.5 token counts above work out to $0.00855, not $0.0086.]
- Why input grows: the input increase is the new tokenizer, which turns the same text into about 30% more tokens. Anthropic documents this for Haiku 5.5, and it matches what we measured.
- Why output didn’t shrink: output depends on how many events the model writes out, and that varies by page.
Two caveats on the cost figures:
- Where they come from: they’re computed from the token counts the API returned, multiplied by the published prices. I haven’t checked them against an actual bill. The per-page arithmetic does check out (for example, page 1 on 4.5 is 7,117 × $1/M + 1,684 × $5/M = $0.0155).
- Higher-price tier: Haiku 5.5 charges 5× more per token for prompts over 100K tokens. We can’t reach that tier, because page content is capped at 200KB and maxSafeTokens is 50,000, so every call stays at the cheap price.
Is it a real improvement?
- Cost and speed: clearly yes.
Dates: better. With today’s date in the prompt, 5.5 gets the year right, and it puts one-time events in the correct date field.[The premise was wrong. See “Guessing the year” below.]- Recall: a little worse. It drops side events such as the Grand K walk and KidsFest on the Moon in June page. Whether the Yeti virtual edition counts as a miss depends on #4063.
- Large club directories: no worse. Both models fail there (#4062).
My view: it’s worth shipping, since an extraction costs about a ninth as much and finishes several times faster, at the price of some missed side events. Whether that trade is acceptable is your call, which is why I’d open the PR as a draft.
n.b. there’s a slight rounding error above, but it doesn’t change the overall conclusion: $0.0086 should be $0.0085.
#Guessing the year
One thing that came out of this is that Haiku 5.5 struggled with inferring the year for an event when it was not explicitly stated. On one event page with ambiguity, Haiku 5.5 returned 2025, whereas Haiku 4.5 returned (correctly) 2026. However, that appears to have exposed a bug in my logic, so that one’s on me. I’m actually not clear on why Haiku 4.5 was even getting the right date in the first place. 😬 I can do better.
#Wrapping up
It’s a very small sample size, but the results were fairly consistent. Based on price alone I think I’m happy to make the switch. Speed is not as much of an issue for me, but I’ll take that as an additional win.
Related posts: