Thursday OpenAI announced plugin system for ChatGPT AI helper. Plugins give ChatGPT the ability to interact with the rest of the world over the Internet, including booking flights, ordering groceries, browsing the web, and more. Plugins are pieces of code that tell ChatGPT how to use an external resource on the web.
Basically, if a developer wants to give ChatGPT the ability to access any network service (eg: “view current stock prices”) or perform any task controlled by the network service (eg: “order a pizza over the Internet”), it is now possible if it is does not violate the rules of OpenAI.
Traditionally, most large language models (LLMs) like ChatGPT have been limited to a bubble, so to speak, only able to interact with the world through text conversations with the user. As OpenAI writes in its introductory Blog Post in ChatGPT plugins: “The only thing language models can do out of the box is generate text.”
Bing Chat developed this paradigm by allowing it to search the web for more recent information, but until now ChatGPT was still isolated from the rest of the world. Being closed in this way, ChatGPT can only use data from its training set (limited to 2021 and earlier) and any information provided by the user during the conversation. In addition, ChatGPT can make factual errors and errors (what AI researchers call “hallucinations”).
To get around these limitations, OpenAI burst the bubble and created ChatGPT plugin interface (what OpenAI calls the “eyes and ears” of ChatGPT), which allows developers to create new components that “hook” into ChatGPT and allow the AI model to interact with other services on the Internet. These services can perform calculations and link to factual information to reduce hallucinations, and they can also potentially interact with any other software service on the web – if the developers create a plugin for the task.
What plugins are we talking about?
In the case of ChatGPT, OpenAI will allow users to select plugins from a list before starting a ChatGPT session. They look almost like apps in the app store, each plugin has its own icon and description.
OpenAI reports that the first round of plugins has been created by the following companies:
- Expedia (for trip planning)
- fiscal note (for real-time market data)
- Instacart (for ordering groceries)
- Kayak (search for flights and car rental)
- Klarna (for price comparison)
- Milo (helper for parents with artificial intelligence)
- OpenTable (for restaurant recommendations and reservations)
- Shopify (for shopping on this site)
- Slack (for communication)
- speak (for language learning with AI)
- tungsten (for calculations and real-time data)
- Locked (on the automation platform)
In particular, the Zapier plugin seems to be particularly powerful as it gives ChatGPT access to an existing software automation system. or like Zapier puts it this way: “You can ask ChatGPT to perform any of Zapier’s 50,000 actions (including search, update, and write) with over 5,000 supported Zapier apps, turning chat into an action. He can write an email and then send it to you. Or find contacts in CRM and then update them directly. Or add rows to a spreadsheet and then send them as a Slack message. The possibilities are endless.”
OpenAI also hosts three plugins, a web browser (which can fetch information from the internet in a manner similar to Bing Chat), a code interpreter for executing Python programs (in a sandbox), and a search tool that allows access to “personal or organizational” sources of information hosted in elsewhere (basically getting information from documents).