Comparing performance with Model based evaluation
1. Comparing performance with Model based evaluation
Automatic side-by-side or AutoSxS is an evaluation tool that facilitates on-demand A/B testing of LLMs for specific tasks. For instance, in a summarization scenario, the autorater, the judging model, assesses the original article and two generated summaries from different models, providing a win rate that highlights the superior model's performance. However, to understand why the autorater prefers one model over another, auto side-by-side provides explanations at the input level, aligning its preference with human judgment. This transparency allows users to better grasp the strengths and weaknesses of each model, ultimately clarifying the autorater's choices. At the core of auto side-by-side is the autorater, an LLM trained to evaluate the quality of responses from other models based on an initial inference prompt. This unique capability allows auto side-by-side to assess the performance of LLMs on Google Cloud. Vertex AI auto side-by-side is a versatile tool capable of evaluating any model in Vertex AI model registry that supports batch prediction, as well as pre-existing models with generated predictions. Specifically designed for assessing LLMs, auto side-by-side employs predefined evaluation criteria for both summarization and question-answering tasks. For example, the summarization criteria in auto side-by-side encompasses various aspects of model performance, including adherence to instructions and the groundedness of generated summaries. Make sure to check the latest token limit and the list of evaluation criteria for each task in Google Cloud's documentation page. Let's say you want to use auto side-by-side to evaluate an LLM like Gemini Pro on Vertex AI for summarization against another model. Assuming you already have your LLM-generated predictions, the evaluation process involves a few simple steps. First, prepare your evaluation dataset with the necessary prompts, contexts, and generated responses. Next, convert this dataset into JSON format and store it in a Cloud Storage bucket or BigQuery table. Finally, run the model evaluation pipeline job in Vertex AI. Let's explore each of these steps in more detail, starting with the evaluation dataset. Auto side-by-side uses an adaptable schema to accept a single evaluation dataset, typically in JSON format for summarization tasks. Each row represents a unique example containing identification fields, data fields for prompts and context, and pre-generated responses from different models. While the fields may not be present in this case, we have the pre-generated predictions containing the responses generated by the LLM task. Human preferences may also be included for comparison against validated data. For optimal results, include examples reflecting real-world model usage. While a single evaluation example is the minimum requirement, using 400-600 examples is recommended for optimal aggregate metrics. Since you have an understanding of how the evaluation dataset might appear and assuming you've uploaded it to a cloud bucket, we can now proceed to define the model evaluation pipeline parameters. AutoSxS released on Vertex AI facilitates running a model evaluation job through its pipeline. Here are the main supported pipeline parameters. Evaluation dataset parameter. This indicates the location of the evaluation dataset. It's either a BigQuery table or a comma-separated list of Cloud Storage paths to a jsonl dataset containing evaluation examples. ID fields. These fields distinguish unique evaluation examples. Here, we typically include ID and document. Task. This parameter specifies the type of task to evaluate, such as summarization or question-answering. In this instance, we're focusing on summarization. Additionally, AutoSxS enables configuring autorater behavior. You can specify inference instructions to guide task completion and set inference contexts to reference during task execution. Lastly, users must provide the names of columns containing predefined predictions to calculate the evaluation metric. Once these parameters are defined, you're prepared to execute the evaluation job. Google Cloud provides a pipeline template to streamline the evaluation process on Vertex AI. Initiate a pipeline job with the defined parameters and then submit a pipeline run using the Vertex AI Python SDK. Upon completion, examine the evaluation results within the Vertex AI pipeline, which include judgment tables, aggregated and align matrices, if human preferences were provided. The judgment table offers metrics for LLM performance per example, indicating the superior response with a confidence score and explanation. This analysis enables you to confidently identify the better response, backed by clear reasoning and a quantifiable confidence level. Here is an example of AutoSxS judgments output. You can see that the summary from Model B was favored over the summary of Model A due to its higher coherence and better coverage. Auto side-by-side also provides aggregated metrics, specifically win rates, derived from the judgment table. These win rates show the percentage of times the autorater favored one model response over another, providing a quick and easy way to identify the superior model for the evaluated task. Fortunately, auto side-by-side does support human preference. To include human preference in your auto side-by-side evaluation, add a column to your evaluation dataset indicating the preferred LLM-generated response. For example, human preference. Before running the pipeline, update the auto side-by-side parameters to specify this column. The remaining evaluation process, including pipeline execution using the Vertex Python SDK, stays unchanged. The result will show additional measurements that incorporate the human preference data. This inclusion of human preference data in your auto side-by-side evaluation will provide additional metrics alongside familiar ones like accuracy. You'll receive both human preference scores and autorater preference scores, indicating how closely these evaluations align. Utilizing auto side-by-side on Vertex AI for evaluating LLMs, comparing them within specific tasks, and ensuring alignment with human preference is just one of the many applications of auto side-by-side. It can also be used to assess a fine-tuned model against its foundational counterpart. Ultimately, auto side-by-side offers a flexible and powerful tool for comprehensive LLM evaluation, empowering you to make informed decisions and optimize your applications.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.