GPT-4o: Great, but not perfect

Is GPT-4o (a.k.a. Omni) really the best OpenAI model you can use?

It depends.

I was genuinely excited when the GPT-4o model was introduced. It significantly faster than GPT-4 and the multimodal abilities (i.e., understanding images and speech) seemed amazing.

After a few weeks, I’m not so convinced anymore.

Here’s what I learned:

1. Omni has been trained on a newer dataset. This means it’s “aware” of events that happened before December 2023 (GPT-4.0 and GPT-3.5 were trained on data older than September 2021). Does this mean a lot? It depends on the use case; for me, this was not a huge differentiator.
2. Omni is fast. Not quite as fast as GPT-3.5, but the trade-off between speed and performance is much better than with GPT-4. Not a limiting factor if you’re only chatting (since GPT-4 can still write faster than most people can read).
3. Omni is supposed to be superior for longer conversations where it needs to pick up the nuances of the entire conversation. I guess I never chat with it long enough, so I can’t confirm whether that’s really true or not. Which brings us to the next point.
4. Language understanding. This was a real bummer for me. Omni is not doing a great job when it needs to deliver text in other languages, but its English is far from perfect too. For generating text and translations, I’m sticking with GPT-4.
5. Omni is multimodal. So is GPT-4. Omni is supposed to be better at understanding images, but I am not truly convinced. I use this function a lot with my work, and when we upgraded our APIs to Omni, the results quality dropped significantly. In the ChatGPT, the difference is not so problematic, so I’m using Omni for personal use and the good old GPT-4 for all the API calls.

The takeaway? Both premium models totally outperform the free GPT-3.5. Omni is great, but not without its flaws.

What’s your experience with the models? And of course, can anyone report on the performance of Gemini or Claude?