Examinee

ProgressGym aims to evaluate how well algorithms align a model according to progressing human values. Those algorithms are instantialized here by the Examinee class. For each alignment algorithm, you have to (and we have already) implement a sub-class of the base Examinee Class. You should specify the extra (from those stored in the base Examinee class) information it stores and, most importantly, how it interacts with the Judge class at each time step. Please refer to the Judge document page if you’re unfamiliar with what Judges are.

This page documents the base Examinee class as well as the baseline examinees we’ve implemented, so as to give you a better understanding of how ProgressGym works.

class benchmark.framework.ExamineeBase(**kwargs)

ExamineeBase is the base class for all examinees. An examinee is the an alignment algorithm (in combination with a language model operated upon by the algorithm) that is benchmarked by a judge. You are free to implement the benchmarked examinee in any way you like, as long as it follows the ExamineeBase interface. In most cases, you need to re-implement most or all all the methods in your subclass. Base implementations are only provided as an example.

__init__(**kwargs)

abstract get_current_model() → Model

Return the current model that the examinee is using at this timestep. The base class implementation returns the current_model attribute. You should not need to override this method in your subclass unless the model is not stored in the current_model attribute.

abstract query_from_judge(prompt: str | Data | List[Dict], model: Model | None = None) → str | Data | List[Dict]

This method is called by the judge to query the examinee for a response to a prompt. In most cases, you only need to call the base class implementation in your subclass’s implementation.

abstract reset(**kwargs) → None

Initialize the examinee, including endowing it with a language model. When examinee_model_size is not specified, the model will be initialized as a copy of the Judge’s initial model. In that case, the examinee will be able to start from the same initial state as the judge. Normally, you should implement this method in your subclass to initialize the examinee as needed, after calling the base class implementation for basic setup.

abstract run(judge: JudgeBase) → Iterable

This method is called by the judge to start the examinee. It will return an iterable that the judge will iterate over to run the examinee. Every iteration corresponds to the passing of a timestep. In this way, the examinee can control the pause and resume of the examinee. At every iteration: 1. The examinee learns about the latest human preference by calling the judge’s query_from_examinee method. 2. After it has updated its language model, it yields control back to the judge and allow it to evaluate it (by calling query_from_judge).

Unless you are sure that you need to completely override this method, you should not do so. Instead, call the base class implementation at the beginning of your subclass’s implementation.

class algorithms.extrapolative_dpo.ExtrapolativeDPOExaminee(**kwargs)

get_current_model() → Model

Return the current model that the examinee is using at this timestep. The base class implementation returns the current_model attribute. You should not need to override this method in your subclass unless the model is not stored in the current_model attribute.

query_from_judge(prompt: str | Data | List[Dict]) → str | Data | List[Dict]

This method is called by the judge to query the examinee for a response to a prompt. In most cases, you only need to call the base class implementation in your subclass’s implementation.

reset(**kwargs)

Initialize the examinee, including endowing it with a language model. When examinee_model_size is not specified, the model will be initialized as a copy of the Judge’s initial model. In that case, the examinee will be able to start from the same initial state as the judge. Normally, you should implement this method in your subclass to initialize the examinee as needed, after calling the base class implementation for basic setup.

run(judge: JudgeBase) → Iterable

This method is called by the judge to start the examinee. It will return an iterable that the judge will iterate over to run the examinee. Every iteration corresponds to the passing of a timestep. In this way, the examinee can control the pause and resume of the examinee. At every iteration: 1. The examinee learns about the latest human preference by calling the judge’s query_from_examinee method. 2. After it has updated its language model, it yields control back to the judge and allow it to evaluate it (by calling query_from_judge).

Unless you are sure that you need to completely override this method, you should not do so. Instead, call the base class implementation at the beginning of your subclass’s implementation.

class algorithms.extrapolative_rlhf.ExtrapolativeRLHFExaminee(**kwargs)

get_current_model() → Model

Return the current model that the examinee is using at this timestep. The base class implementation returns the current_model attribute. You should not need to override this method in your subclass unless the model is not stored in the current_model attribute.

query_from_judge(prompt: str | Data | List[Dict]) → str | Data | List[Dict]

This method is called by the judge to query the examinee for a response to a prompt. In most cases, you only need to call the base class implementation in your subclass’s implementation.

reset(**kwargs)

Initialize the examinee, including endowing it with a language model. When examinee_model_size is not specified, the model will be initialized as a copy of the Judge’s initial model. In that case, the examinee will be able to start from the same initial state as the judge. Normally, you should implement this method in your subclass to initialize the examinee as needed, after calling the base class implementation for basic setup.

run(judge: JudgeBase) → Iterable

This method is called by the judge to start the examinee. It will return an iterable that the judge will iterate over to run the examinee. Every iteration corresponds to the passing of a timestep. In this way, the examinee can control the pause and resume of the examinee. At every iteration: 1. The examinee learns about the latest human preference by calling the judge’s query_from_examinee method. 2. After it has updated its language model, it yields control back to the judge and allow it to evaluate it (by calling query_from_judge).

Unless you are sure that you need to completely override this method, you should not do so. Instead, call the base class implementation at the beginning of your subclass’s implementation.

class algorithms.lifelong_rlhf.LifelongRLHFExaminee(**kwargs)

get_current_model() → Model

Return the current model that the examinee is using at this timestep. The base class implementation returns the current_model attribute. You should not need to override this method in your subclass unless the model is not stored in the current_model attribute.

query_from_judge(prompt: str | Data | List[Dict]) → str | Data | List[Dict]

This method is called by the judge to query the examinee for a response to a prompt. In most cases, you only need to call the base class implementation in your subclass’s implementation.

reset(**kwargs)

Initialize the examinee, including endowing it with a language model. When examinee_model_size is not specified, the model will be initialized as a copy of the Judge’s initial model. In that case, the examinee will be able to start from the same initial state as the judge. Normally, you should implement this method in your subclass to initialize the examinee as needed, after calling the base class implementation for basic setup.

run(judge: JudgeBase) → Iterable

This method is called by the judge to start the examinee. It will return an iterable that the judge will iterate over to run the examinee. Every iteration corresponds to the passing of a timestep. In this way, the examinee can control the pause and resume of the examinee. At every iteration: 1. The examinee learns about the latest human preference by calling the judge’s query_from_examinee method. 2. After it has updated its language model, it yields control back to the judge and allow it to evaluate it (by calling query_from_judge).

Unless you are sure that you need to completely override this method, you should not do so. Instead, call the base class implementation at the beginning of your subclass’s implementation.

class algorithms.lifelong_dpo.LifelongDPOExaminee(**kwargs)

get_current_model() → Model

Return the current model that the examinee is using at this timestep. The base class implementation returns the current_model attribute. You should not need to override this method in your subclass unless the model is not stored in the current_model attribute.

query_from_judge(prompt: str | Data | List[Dict]) → str | Data | List[Dict]

This method is called by the judge to query the examinee for a response to a prompt. In most cases, you only need to call the base class implementation in your subclass’s implementation.

reset(**kwargs)

Initialize the examinee, including endowing it with a language model. When examinee_model_size is not specified, the model will be initialized as a copy of the Judge’s initial model. In that case, the examinee will be able to start from the same initial state as the judge. Normally, you should implement this method in your subclass to initialize the examinee as needed, after calling the base class implementation for basic setup.

run(judge: JudgeBase) → Iterable

This method is called by the judge to start the examinee. It will return an iterable that the judge will iterate over to run the examinee. Every iteration corresponds to the passing of a timestep. In this way, the examinee can control the pause and resume of the examinee. At every iteration: 1. The examinee learns about the latest human preference by calling the judge’s query_from_examinee method. 2. After it has updated its language model, it yields control back to the judge and allow it to evaluate it (by calling query_from_judge).

Unless you are sure that you need to completely override this method, you should not do so. Instead, call the base class implementation at the beginning of your subclass’s implementation.