Extract FEMA incidents
Extracting data from a .pdf or .jpg of a table. Suggested by Simon Willison at his PyCon 2025 talk about building software with LLMs.
The table is on page 9 of this pdf. Here is the .jpg version:
| Model | Pass rate | Tests | Avg tokens |
|---|---|---|---|
| claude-3-7-sonnet-20250219 | 2 | 1.4k | |
| claude-opus-4-0 | 2 | 1.4k | |
| gemini-2.5-pro-preview-03-25 | 2 | 3.1k | |
| claude-3-5-sonnet-20241022 | 2 | 1.4k | |
| gemini-1.5-flash-latest | 2 | 1.0k | |
| gemini-2.5-pro-preview-05-06 | 2 | 3.8k | |
| claude-3-opus-20240229 | 1 | 1.6k | |
| claude-sonnet-4-0 | 2 | 1.4k | |
| gemini-1.5-pro-latest | 2 | 1.0k | |
| gpt-4o | 2 | 761 | |
| gpt-4o-mini | 2 | 580 | |
| gpt-5-2025-08-07 | 2 | 4.2k | |
| gpt-oss-20b | 1 | 78 | |
| o1 | 2 | 6.2k | |
| o3 | 2 | 2.1k | |
| o4-mini-2025-04-16 | 2 | 2.9k | |
| qwen3-14b | 1 | 299 | |
| qwen3-vl-8b-instruct | 2 | 2.9k | |
| qwen3-vl-8b-thinking | 2 | 9.9k | |
| deepseek-ocr | 1 | – | |
| kimi-k2-thinking (moonshotai) | 2 | – | |
| qwen3-30b-a3b-2507 | 1 | – | |
| qwen3-VL-30b-a3b-2507 | 1 | – |
Full results
Loading results…