Trainer

Overview

Trainer is an abstract class for implementing prompt optimization methods. Every prompt optimization method should be implemented by subclassing this class.

Methods

__init__

parameters:

• generator: Generator class for generation tasks.
• metric: Metric class for evaluation tasks.
• global_metric: Global metric class for evaluation tasks, Optional.
• testmode: Whether to run in test mode, Optional. If testmode is True, the trainer will evaluate the prompt for each optimization step with valset and save results to the Report object.

When you initialize any kinds of trainer class, you should pass generator and metric parameters.

train

parameters:

• prompt: Prompt object to be optimized.
• trainset: Training dataset. List of DatasetItem TypedDict.
• valset: Validation dataset. List of DatasetItem TypedDict.

returns:

• Tuple[Prompt, BaseReport]: Optimized prompt and report.

_evaluate

parameters:

• prompt: Prompt object to be evaluated.
• dataset: Dataset to be evaluated. List of DatasetItem TypedDict.

returns:

• Tuple[List[Any], List[MetricResult], GlobalMetricResult]: Predictions, Metric results, and Global metric result.

_evaluate method is used to evaluate the prompt for given dataset.
It is used internally, but you can also use it to evaluate the prompt for any dataset.
It returns list of predictions (return value from Generator.generate), list of metric results (return value from Metric.compute), and global metric result (return value from GlobalMetric.compute).