Verbosity ≠ Value. Disconnect between pricing and customer value in LLM's
OpenAI and others charge based on tokens - both input and output tokens. In other words, you pay to talk to the model, and you pay to have the model talk to you. Here’s a snippet of an OpenAI bill:
Their monetization approach is clear — pay for what you use. But I don’t think it maps well to the value realized by the user.
Verbosity ≠ Value
This would be the part where I’d reference some wisdom about the value of brevity from a philosopher, like “If I had more time, I would have written a shorter letter.” Longer responses are costly and often not helpful to the user (unless you’re a high school student or lazy copywriter).
Let’s explore a high-value use case that uses barely any tokens at all.
LLM-driven fuzzy matching for identity resolution:
Imagine you’re a company putting on an event. You have a list of names from the event signup form, and you want to match them to your list of customer contacts to see who went to the event.
I probably wouldn’t suggest running through every unique pair of [name_from_signup, customer_contact]. But you could clean up the strings and narrow down the possible matches using a traditional/non-AI fuzzy match approach. Then the LLM can be the cherry on top that handles the ambiguous situations with a prompt like:
look at [name_from_signup] and [customer_contact] and determine if it’s reasonable to say these are the same person. If you believe they’re the same person, reply with 1. If not, reply with 0
I’m sure you could improve upon that prompt and experiment with what works best, but the point is, the output would be 1 token — validating 333 pairs would cost 1 penny if you were to use the most expensive model (GPT 4 preview as of Jan 2024).
This is also an application where performance isn’t a big factor. We don’t have to solve this problem dynamically. It can be an ETL job that runs when people are sleeping.
String matching and related topics have many customer applications that can be solved (or improved) for little cost with the clever use of LLM’s.
Who knows what will happen to LLM pricing in the future. Will we see prices go up once the initial customer landgrab settles down and vendor lock-in creeps in? Or will the hardware and software become commoditized and we’ll see prices drop? I don’t know, but as long as the pricing model is pay-per-token, I’ll prefer my robot companions to be less talkative.