Just wanted to write a quick post to sum up my experience of working with the OpenAI API (specifically the chatGPT API) as I've had a feature running in production on this for a few months now.
As far as API integrations go it was incredibly simple to get up and running, Laravel's HTTP client is a pretty clean wrapper around Guzzle and allows you to create an integration in a laughably small amount of code.
My integration consists of an API object that wraps input up in a prompt and sends it off to chatGPT, it's just a bunch of good old PHP arrays and Http::withToken($this->token)->post(), it really couldn't be much more simple.
The good parts of working with the ChatGPT api
The documentation is really rather good, it's pretty straight-forward to get to what you need and the features are documented in a way that's friendly for the average developer (like me) rather than machine learning PhDs.
I've been using the completions api in an application that generates route plans for trips around Europe. For the most part the model returns an impressive response which in 90% of cases is a perfectly valid and interesting route. It took a while to get used to working with the prompt to get the best out of it, but after a while it becomes easier to predict how the model will behave, I'm sure there's a clever remark about irony in there somewhere!
The OpenAI reporting dashboard is easy to follow and you get a decent amount of data about how many tokens your application is consuming over the api.
The challenges of working with the ChatGPT api
The main challenges really are handling the unpredictability of the model's responses. It seems to get it right in 9 out of 10 cases but will still sometimes return a ridiculous or invalid response. If you code your application defensively it's not that difficult to handle this but it does highlight the problems of working with these systems and how much you can actually rely on them.
The planner application works with JSON objects that are serialised back to the db when the trips are saved. So integrating with ChatGPT was just a case of getting the model to read and write this JSON in a way that it can just be fed back into the planner, but rather than having user provided routes they're now from ChatGPT. This seems to work well if you keep the prompt vague around specific details of the trip. For example if you try to specify preferences for things to do on the trip that would be rare or non-existent in ChatGPT's training data then it will quite often return totally made up (or as they like to say 'hallucinated') places and locations.
The response will also sometimes contain invalid JSON, but this seems to improve by tweaking the prompt and being really clear about what it should return. I think this use-case of providing it with JSON and having JSON returned is actually pretty interesting and seems like a good way to control more of what it's doing, rather than just having it return a text stream.
In Summary
Unpredictability of the response and hallucinations are the main issues that I've noticed working with the api. It really does make you think about the real-world application of these models, when the dust settles on the current state of AI I think a lot of us will be questioning where these models can be put to best use. Given how they work I can't see a time soon when they could be trusted with any unsupervised decision making task in any non-trivial application. I think a key part of software development in the future will be writing applications that defensively integrate with GPTs to take advantage of their power to automate the mundane whilst also managing their hallucinations.
Photo by Emiliano Vittoriosi on Unsplash
