Generate legal and compliance answers¶

Fast facts¶

What it is¶

It’s hard to find more document-intensive processes than legal and compliance. Finding answers to complex questions about the law and policies can involve a lot of time spent combing through dense and complicated documents. Any uncertainty here can halt operational and business processes. This kind of work involves highly paid professionals, which additionally increases the costs, as such processes take up a lot of their valuable time.

Generative AI can address this pain point for legal and compliance professionals to scour documentation and find answers to pressing questions, in the exact context that these professionals require. After retrieving these relevant chunks of information, generative AI constructs and delivers a legible answer that the user can utilize in their decision making. The added benefit of this automation is that the LLM can uncover additional insights from those documents, things that a person “grinding” through the documents might miss.

How it works¶

The process involves feeding legal and compliance documents into a vector database, which is then utilized by the LLM to retrieve information for the user, chatbot-style. Most organizations already have stores of legal documents, in places like Microsoft SharePoint, which can be used as the source of the information. An important part of the process is ensuring that all of the possible LLM and vector database parameters are tested thoroughly before deployment, given the specific nature of “legalese.” Things like chunking and embedding strategies need to be reviewed rigorously.

In many RAG cases, a standard publicly available LLM, called via an API, could work. But since the information for legal and compliance purposes can be highly sensitive, a locally-hosted open source LLM will be a better, more secure choice.

The overall workflow will benefit from a predictive AI guard model, designed to monitor outputs and deliver confidence scores for each response, while also blocking unwanted, hallucinated outputs. The lower the confidence score, the more attention the user needs to pay to the answer. The users can also rate the answers or edit them, which then can be sent back to the original vector database for future reference.

User experience¶

There are multiple ways of approaching the deployment of this solution, but the most common one is implementing the chatbot directly in the standard corporate communications environment, like Microsoft Teams.

The user would open the chat window and start asking questions, since the vector database already stores most of the necessary legal documents. For example: “What are the liability rules in the EU AI Act?”, “What are the rules around filing of civil appeals in the appellate court of {insert_state}?”, and so on.

For this to work seamlessly, an additional internal data pipeline could be built to ensure that new documents could be added to the database quickly. For example, an automated solution that scans the location of legal files and automatically adds new ones to the vector database.

The responses come back according to the given system prompt settings (format, length, etc.). An important addition to the output here would be to automatically ask the model to link to specific source documents that it's referencing. This increases trust and streamlines the user review process even more.

Why it might benefit your business¶

Organizations spend a lot of resources or, even, pay a high hourly rate to send legal and compliance professionals searching through large libraries of information. Generative AI chatbots can significantly reduce the time and labor it takes to find relevant information, realizing cost savings while freeing up those professionals to focus on more important work.

This approach also reduces the risk of information being overlooked, resulting in faster, more comprehensive answers to the most pressing legal and compliance questions.

Potential risks¶

There are numerous risks associated with this solution, given the sensitive nature of the information that’s being processed and outputted.

• Inaccurate or off-topic responses (hallucinations) to toxic responses, as well as outputs that divulge sensitive information that other stakeholders shouldn’t have access to.

• A system prompt that’s not fine-tuned may lead to unconventionally worded and structured responses, which don’t satisfy the user, leading to a lot of manual edits or direct and potentially costly errors.

Baseline mitigation tactics¶

• Custom metrics monitoring for toxicity, readability (Flesch Reading Score), as well as informative/ uninformative responses to ensure that the responses are appropriate. This also extends to operational custom metrics, like tokens/ cost monitoring to ensure the financial viability of the solution.

• Guard models that prevent unwanted generative AI outputs, based on the response parameters or halt any potentially inaccurate answers from being outputted to the user. Accuracy is paramount when legal language is involved and guard models can ensure that. This is also important to ensure that the users are utilizing the tool appropriately, as the guardrails in place can prevent users from sending irrelevant prompts, thus ballooning the costs of the solution.

• A feedback loop by which the user can rate the generated response and edit, if necessary. The edited version then gets added back to the vector database to inform future responses, thus improving the system as time goes on.

• Extensive testing of the LLM, its parameters, like the system prompts or the vector database to ensure that responses require minimal oversight and don’t lead to answers that misrepresent legal or compliance norms.