CosineSimilarityMetric

Overview

The CosineSimilarityMetric class provides a way to compute the cosine similarity between two text inputs. It uses embeddings generated by a specified model to calculate the similarity score.

Attributes

  • model (str): The model used for generating embeddings. Defaults to “text-embedding-3-large”.

Methods

compute

Compute the cosine similarity score between the gold standard and prediction.

Parameters:

  • inputs (Dict[str, Any]): Additional input information.
  • gold (Any): The gold standard data item.
  • pred (Any): The prediction to compare against the gold standard.
  • trace (Optional[Dict]): Additional trace information.
  • metadata (Optional[Dict]): Additional metadata.

Returns: MetricResult: The computed cosine similarity score and any intermediate values.

Note: This method converts the inputs to strings if they are not already, generates embeddings for both the gold standard and prediction, and calculates the cosine similarity between the embeddings. If an error occurs, it returns a score of 0.0 and includes the error message in the intermediate values.

Usage

To use the CosineSimilarityMetric class, create an instance and call the compute method with the prediction and gold standard data items.

Example:

from ape.common.metrics import CosineSimilarityMetric
 
metric = CosineSimilarityMetric()
 
result = await metric.compute(inputs={"text1": "Hello, world!", "text2": "Greetings from the cosmos!"}, gold="Hello, world!", pred="Greetings from the cosmos!")
print(result)
BaseMetricJsonMatchMetric