Protect AI - LLM Guard

This is a python library to protect the application from potential misuse and other threats to the sanctity of the application.

In some guards they use smaller Large Language Model (LLM) to predict and asses the threats and the user inputs, they support both CPU and GPU accelerated instances, for much faster performance we can use ONXX (Open Neural Network Exchange) which allows for faster processing times

Memory Usage

As we use multiple fine-tuned models for the predictions, these models have to be loaded in memory, which reaches over 6-7 GiB.
The team of llm-guard have given a few optimizations specifically for memory usage at Optimization Strategies.

Prompt Injection Detection

The LLM guard uses a model to analyze the user prompt and detect if the prompt safe or a potential threat. There is a Protect AI - ReBuff AI which is said to be much better.
This tool is not available as a native component but needs a ReBuff Server that needs to be available, also required Pinecone, Supabase or ChromaDB as vector stores.

References

• LLM-guard