
Cost per successful task: how to compare AI tools before you buy
A tool is priced per attempt. You pay for results. Divide the price of an attempt by the share of attempts that come back right, add the minute someone spends checking, and compare that number instead of the sticker price.
Priced per attempt, paid for per result
AI tools are priced in one of two ways. Seat plans charge a fixed amount per person per month. Usage plans charge per million tokens, the units of text the model reads and writes, with input and output priced separately. Neither price moves when the answer is wrong. A draft you throw away costs the same as a draft you send.
A business does not buy attempts. It buys finished tasks: a reply that went out, an invoice that matched, a summary someone trusted. So the price on the page is the wrong number to compare. The right number is what you pay, in money and in time, for each task that gets done.
The formula
Start with the price of one attempt. For a seat plan, divide the monthly price by the number of tasks that person runs through the tool in a month. For a usage plan, take the bill for a week of real use and divide it by the number of tasks. Either way, call the result the attempt cost.
Then measure the success rate. Of the last twenty or fifty attempts, how many came back usable without a rewrite? Divide the attempt cost by that share. The result is the cost per successful task. At $0.10 an attempt and an 80 percent success rate, each good result costs $0.125. At 50 percent it costs $0.20. At 25 percent it costs $0.40. The chart draws that division. Nothing in it is a forecast; it is one line of arithmetic.

Add the minute someone spends checking
Every output gets looked at by a person before it is used. That minute has a price too. One minute at $30 an hour is $0.50. Add it to every attempt, because a person checks the failures as well as the successes. The formula becomes the attempt cost plus the check, divided by the success rate.
Now the model price stops being the deciding number. Take two tools. One costs $0.02 an attempt and comes back right 70 percent of the time. The other costs $0.20 an attempt, ten times more, and comes back right 95 percent of the time. With a $0.50 check on every attempt, the first costs $0.74 per successful task and so does the second. The ten-fold price difference has disappeared into the checking. From here, any gain in success rate makes the dearer tool the cheaper one.

What a benchmark score is
Vendors and researchers publish benchmark scores. A benchmark is a fixed set of questions with known answers, and the score is the share the model got right. It is a success rate, measured on someone else's tasks. Epoch AI, which runs public evaluations, describes GPQA Diamond as a set of 198 questions, runs each one several times per model, and reports the score with a margin of error.
Use a benchmark score as a first guess at your success rate, and nothing more. A high score on a public question set says little about your invoices or your customer replies. The score to trust is the one you measure on twenty of your own tasks. That measurement takes an afternoon, and it is the only number in this article that is about your business.
Two numbers to write down
Before you buy, record two numbers for each tool you are weighing: the attempt cost and the success rate on your own tasks. Add your checking minute. Divide. The tool with the lowest cost per successful task wins, whatever its sticker price.
After you buy, keep measuring. Success rates move when the tool changes or your work changes. Last quarter's number is a guess about this one.
What the formula leaves out
It assumes a failure is caught. A wrong answer that reaches a customer is not in this arithmetic, and it is the cost that hurts most. If the check is skipped, the number on the chart is a floor, not an estimate.
It also leaves out the fixed costs: setup, training people, and the time and money needed to leave a tool that stops working, or to get your data back out of it. Those belong in the same spreadsheet, on their own line, before you sign anything.
Sources
Educational commentary from Loop XXI. Publication record: Block 974,399.