OpenAI has released a ChatGPT chatbot that can answer questions, admit mistakes, argue, and reject inappropriate requests.
The developers created the model using reinforcement learning based on human feedback (RLHF). They used the same methods as InstructGPT, but with additional human dialogue data.
To collect a dataset of conversations, the developers attracted the so-called trainers. They held conversations on behalf of a person and an AI assistant. Instructors also had access to simulated sentences to help them write responses.
For the reinforcement learning reward model, the team recorded conversations between a trainer and a chatbot. It then randomly selected AI-generated responses and asked the instructors to rank them, says Forklog.
To improve the accuracy of the model, the developers used proximal policy optimization. For this process, they performed several iterations.
According to the developers, the model has a number of limitations. Despite plausible answers, ChatGPT is often wrong, sensitive to wording, overusing certain phrases, showing bias, and guessing the context.
However, the company said that the experience of deploying GPT-3 and Codex has improved the security of the model and reduced the number of invalid responses thanks to RLHF.
ChatGPT is available for free. The developers asked users to provide feedback on problematic model output, false positives of the external content filter that is part of the interface.
“We are especially interested in feedback […] that helps us identify and understand new risks and possible mitigation measures,” the press release reads.
According to the developers, the information received will help them in the development of more modern and advanced language models.
NIX Solutions notes that ChatGPT was trained on the Azure AI supercomputer. The chatbot is a refinement of the GPT-3.5 series model, the creation of which was completed in early 2022.