GenOps: the evolution of MLOps for gen AI

As organizations move to deploy Generative AI solutions at scale, they often face operational challenges. GenOps, or MLOps for Gen AI, addresses these challenges.

GenOps combines DevOps principles with ML workflows to deploy, monitor, and maintain Gen AI models in production. It ensures Gen AI systems are scalable, reliable, and continuously improving.

Why is MLOps challenging for Gen AI?
Gen AI models present unique challenges that make the traditional MLOps practices insufficient:

1. Scale: Billions of parameters require specialized infrastructure.

2. Compute: High resource demands for training and inference.

3. Safety: Need for robust safeguards against harmful content.

4. Rapid evolution: Constant updates to keep pace with new developments.

5. Unpredictability: Non-deterministic outputs complicate testing and validation.

In this blog, we'll explore how to adapt and extend MLOps principles to meet the unique demands of Gen AI. 

Key capabilities of GenOps 
Think of a Gen AI model as the tip of an iceberg. Generative AI systems are complex and contain numerous hidden interconnected elements. We’ve extended those elements in the below image from the original paper, Hidden Technical Debt in Machine Learning Systems.

Below are the different elements in GenOps for pre-trained and fine-tuned models:

1. Gen AI experimentation and prototyping: Experiment and build prototypes using enterprise models like Gemini, Imagen and open-weight models like Gemma2, PaliGemma etc.

2. Prompt: Below are the various tasks involving prompts:

1. Prompt engineering: Design and refine prompts for GenAI models to generate desired outputs 

2. Prompt versioning: Manage, track and control changes to prompts over time. 

3. Prompt enhancement: Employ LLMs to generate an improved prompt that maximizes performance on a given task.

3. Evaluation: Evaluate the responses from the GenAI model for specific tasks using metrics or feedback

4. Optimization: Apply optimization techniques like quantization and distillation to make models more efficient for deployment.

5. Safety: Implement guardrails and filters. Models like Gemini have inbuilt Safety Filters to prevent harmful responses by the model.

6. Fine-tuning: Adapt pre-trained models to specific domains/tasks through additional tuning on specialized datasets. 

7. Version control: Manage different versions of GenAI models, prompts and dataset versions.

8. Deployment: Serve GenAI models with scaling, containerization and integration. 

9. Monitoring: Track model performance, output quality, latency and resource usage in real-time.

10. Security and governance: Protect models and data from unauthorized access or attacks and ensure compliance with regulations. 

Now that we’ve seen the key components in GenOps, let’s take a look at the classic MLOps pipeline in VertexAI and see how it can be extended to GenOps. This MLOps pipeline automates the process of taking a ML model to deployment, incorporating continuous integration, continuous deployment and continuous training(CI/CD/CT).

Extend MLOps to support GenOps
Now let’s understand the key components of a robust GenOps pipeline on Google Cloud, from initial experimentation with powerful models across multiple modalities to the critical considerations of fine-tuning, safety and deployment. This blog focuses on leveraging pre-trained models that can be fine tuned.

Let's explore how these key components of GenOps can be built in Google Cloud.

1.  Data: The Generative AI journey starts with data.

1. Few shot examples: Few-shot prompts provide examples to guide the model's output format, phrasing, scope, and pattern.

2. Supervised fine-tuning dataset: This labeled dataset is used to fine-tune a pre-trained model to a specific task or domain. 

3. Golden evaluation dataset: This is used to assess the performance of the model for a given task. This labeled dataset can be used for both manual and metric-based evaluation for that specific task.

2. Prompt management: Vertex AI Studio allows collaborative prompt creation, testing, and refinement. Teams can input text, select models, adjust parameters, and save finalized prompts within shared projects.

3. Model fine-tuning: This step involves adjusting the pre-trained Gen AI model to specific tasks or domains using the fine-tuning data. Below are the two methods available in Vertex AI for fine tuning models in Google Cloud:

1. 1. Supervised fine-tuning is a good option when the task is well-defined with available labeled data. It's particularly effective for domain-specific applications where the language or content significantly differs from the data the large model was originally trained on. Here are the configuration recommendations for tuning LLMs on VertexAI using Low-Rank Adaptation of Large Language Models(LoRA) and its more memory-efficient version, QLoRA.

   2. Reinforcement Learning from human feedback(RLHF) uses feedback gathered from humans to tune a model. RLHF is recommended when the output of the model is complex and difficult to describe. If the output from the model isn't difficult to define, Supervised fine-tuning is recommended.

Visualization tools like TensorBoard and embedding projector can be used to detect unrelated prompts which can be further used for RLHF and RLAIF for preparing the answer for them, as shown below.

UI can also be created using open source solutions like Google Mesop that helps human evaluators to evaluate and update the LLMs responses, as shown below.

4. Model evaluation:  The GenAI Evaluation Service in Vertex AI is used to evaluate GenAI models with explainable metrics. It provides two major types of metrics:

1. Model-based metrics: The model-based metrics use a proprietary Google model as a judge. It is used to measure model-based metrics pairwise or pointwise.

2. Computation-based metrics: These metrics(such as ROUGE and BLEU) are computed using mathematical formulas to compare the model's output against a ground truth or reference. 

Automatic side-by-side(AutoSxS) is a pairwise model-based evaluation tool that is used to evaluate the performance of either GenAI models in VertexAI Model Registry or pre-generated predictions. It uses an autorater to decide which model gives the better response to a prompt.

5. Model deployment:

1. Models that have managed APIs and do not need deployment: Google’s foundational Gen AI models like Gemini, have managed APIs and can accept prompts without deployment.

2. Models that need deployment: Other Gen AI models must be deployed to a VertexAI Endpoint before they can accept prompts. To be deployable, the model must be visible in VertexAI Model Registry. There are two types of GenAI models that are deployed:

• Fine-tuned models that are created by tuning a supported foundation model with custom data.

• GenAI models that don't have managed APIs: Many models in VertexAI Model Garden like Gemma2, Mistral, Nemo etc are accessible via Deploy button or Open Notebook.

Some models support deployment to Google Kubernetes Engine for more control. Serving a model with a single GPU in GKE includes online serving frameworks such as NVIDIA Triton Inference Server and vLLM that are high-throughput and memory-efficient inference engines for LLMs.

6. Monitoring: Ongoing monitoring assesses the deployed models' real-world performance, analyzing key metrics, data patterns, and trends. Cloud Monitoring is used to monitor the Google Cloud services.

The adoption of GenOps practices helps organizations fully leverage Gen AI's potential by creating systems that are efficient and aligned with business objectives.