LW - AI Safety Chatbot by markov

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI Safety Chatbot, published by markov on December 22, 2023 on LessWrong.
Hello World! The
AISafety.info team is launching a prototype of the
AI Safety Chatbot.
The chatbot uses
a dataset of alignment literature to answer any questions related to AI safety that you might have, while also citing established sources. Please keep in mind that this is a very early prototype and despite citing references, it may still provide inaccurate or inappropriate information.
The overall objective is to help people better understand AI Safety issues based on alignment research using an LLM. This helps with tailoring content to the user's needs and technical level. The chatbot can hopefully be used by both newcomers to AI safety, as well as researchers and engineers who want to get up to speed on specific topics.
How it works
This chatbot builds upon
AlignmentSearch. Our work also expands upon the alignment research dataset (ARD) developed during
AI Safety Camp 6. This involved updating and curating the dataset to focus more on quality over quantity. Additionally, we created a process to regularly fetch new articles from selected sources. The ARD contains information about alignment from various books, research papers, and blog posts. For a full list of all the sources being used, look at the readme of the repository on
GitHub or
HuggingFace.
We use a process called
retrieval-augmented generation (RAG) to generate the answers. Since LLM data is static, RAG increases the capabilities of a LLM by referencing an external authoritative knowledge base before generating a response. So the process can be roughly broken into - 1) getting and storing the data in a vector database, and then 2) generating an answer based on that data.
The information storage process is outlined below:
Source: DeepLearning.AI (2023) "
LangChain: Chat with Your Data"
Document Loading: The articles are scraped from various sources such as the ones mentioned above. They are then parsed and stored in an SQL database while making sure that metadata values fields are valid.
Splitting: Then the text content of the documents is broken up into fixed-sized chunks.
Storage: These chunks are then embedded into the
Pinecone vector database using the
OpenAI embedding model.
Once we have a database of alignment literature, we use the following series of steps to generate an answer based on a user query:
Source: DeepLearning.AI (2023) "
LangChain: Chat with Your Data"
Query: A user types in a question.
Storage+Retrieval: We retrieve chunks from the vector database that are semantically similar to the user's question.
Prompt: A prompt is formed that includes all the text retrieved from the relevant chunks provided as context, along with additional instructions on how to format citations and structure the answer.
Output: This prompt is then passed to the LLM, which synthesizes an answer based on the relevant chunk of data along with accurate inline citations to the source material.
Additionally, as the answer is generated, a '
glossary' is injected with manually written one-sentence definitions of common jargon. The following image example shows what Goodhart's Law looks like on hover:

With automatic updates, the ARD will periodically fetch new article entries from trusted sources and add or update items to a SQL database. A separate process adds text to the dataset from
user suggested sources. This dataset is available on
HuggingFace, which includes instructions on how to download and use it. This means that the chatbot will always be able to produce the more relevant and newer information.
We are also experimenting with multiple modes for different audiences. Currently, we offer three options, which produce answers of varying complexity, using the same chunks but adjusting the prompt sent to the LLM.

Hallucinations
Each chun...