Every few months a new model name is on everyone's lips, and the question is always the same: which one is better? There is no answer that is right for everyone, because better depends on your job. The method below works the same for writing, summarising, coding and customer replies, and it takes about an afternoon.
Write the job in one sentence
A model is not chosen for AI in general; it is chosen for a specific job. Summarising a long report into five key points needs a different model than helping with code. Next to that sentence, write three things:
- How long is a typical input: a short message or a file of many pages?
- What shape must the output take: free text, a table, or JSON?
- What happens if the answer is wrong: does someone merely get annoyed, or does a wrong report reach a customer?
Put the criteria side by side
Check these five points for each candidate. You do not have to fill them with numbers; it is enough to know which one weighs most for your job.
| Criterion | What to check | Why it matters |
|---|---|---|
| Language quality | Use real text in the language of your work. | Fluency, terminology and tone need representative samples. |
| Input length | Does your longest input fit the model? | Input over the limit is cut or rejected. |
| Price | Look at input and output price separately. | Output usually costs more than input. |
| Speed | Measure time to first word in your own scenario. | It matters more than total time for live chat. |
| Abilities | Do you need images, files, tools or structured output? | Not every model has everything. |
The price per million tokens alone is misleading. Guess the length of a typical chat, multiply by the number of chats in a month and work out the monthly cost. In Hoveydai the price of each model is shown before you use it.
The ten-sample test
Take ten real inputs, write one fixed prompt and give it to two or three models. If you write code, a simple loop is enough:
import osfrom openai import OpenAI client = OpenAI(base_url="https://<HOVEYDAI_BASE_URL>/v1", api_key=os.environ["HOVEYDAI_API_KEY"]) MODELS = ["model-a", "model-b", "model-c"] for model in MODELS: for text in my_ten_inputs: r = client.chat.completions.create( model=model, messages=[{"role": "user", "content": PROMPT + text}], ) save(model, text, r.choices[0].message.content)
Score the outputs without seeing the model name. The name affects judgement even when you think it does not.
In this test your samples are the judge, not someone else's ranking table.
Take the smallest model that is enough
A model that scored acceptably and cost less is the right choice, even if another one writes a little better. If your job has several steps, test each step on its own:
- Give simple steps, such as classifying and extracting, to a small model.
- Move only the low-scoring step to a stronger model.
- Read sensitive outputs yourself before sending them.
Keep the test
Models change, and so do prices. Save the ten samples and your prompt so that whenever a new model arrives you can re-measure in ten minutes. That small file is closer to your work than any outside comparison.