Evaluator Module

The evaluator module contains various evaluation methods for different multimodal tasks.

Base Evaluator

class flagevalmm.evaluator.base_evaluator.QuestionMapping(original_question_id: str, is_multi_inference: bool, inference_index: int, total_inferences: int)

Bases: object

original_question_id: str

is_multi_inference: bool

inference_index: int

total_inferences: int

__init__(original_question_id: str, is_multi_inference: bool, inference_index: int, total_inferences: int) → None

class flagevalmm.evaluator.base_evaluator.BaseEvaluator(is_clean: bool = True, use_llm_evaluator: bool = False, eval_func: Callable | str | None = None, base_dir: str = '', detailed_keys: List[str] | None = None, aggregation_fields: List[str] | None = ['raw_answer'], **kwargs)

Bases: object

__init__(is_clean: bool = True, use_llm_evaluator: bool = False, eval_func: Callable | str | None = None, base_dir: str = '', detailed_keys: List[str] | None = None, aggregation_fields: List[str] | None = ['raw_answer'], **kwargs) → None

get_eval_func(eval_func: Callable | str | None)

statistics_tokens(predictions: List[Dict]) → Dict

expand_multi_inference_predictions(predictions: List[Dict]) → Tuple[List[Dict], Dict[int, QuestionMapping]]

Expand multiple inference predictions into individual predictions.

Returns:

List of individual predictions question_mapping: Mapping from expanded prediction index to original question info

Return type:

expanded_predictions

aggregate_multi_inference_results(expanded_predictions: List[Dict], question_mapping: Dict[int, QuestionMapping]) → Tuple[List[Dict], Dict]

Aggregate results from expanded predictions back to original questions.

Returns:

List of predictions with aggregated results stats: Statistics about the evaluation

Return type:

aggregated_predictions

has_multi_inference(predictions: List[Dict]) → bool

Check if any prediction contains multiple inference results.

evaluate_multiple_choice(gt: Dict, pred: Dict) → bool

evaluate_fill_blank_by_rule(gt: Dict, pred: Dict, simality_threshold: float = 0.7) → Tuple[bool, str]

evaluate_multiple_response(gt: Dict, pred: Dict) → Tuple[bool, str]

extract_judgement_result(response_text: str) → Tuple[bool, str]

Extract judgement result from LLM response using regex. Validates format and extracts the result in one step. Returns: (is_correct, extracted_response)

evaluate_by_llm(gt: Dict, pred: Dict) → Tuple[bool, str]

cal_accuracy(annotations: Dict, predictions: List[Dict], *args, **kwargs) → Dict

maybe_clean_answer(answer: str) → str

filter_rejected(predictions: List[Dict], results: Dict) → Tuple[List[Dict], List[Dict]]

process(dataset: Dataset, output_dir: str, **kwargs) → Dict

Parameters:

• dataset (Dataset) – dataset instance

• output_dir – str

save(results: Dict, answers: List[Dict], dataset_name: str, output_dir: str)

MMMU Evaluator

flagevalmm.evaluator.mmmu_dataset_evaluator.check_is_number(string)

Check if the given string a number.

flagevalmm.evaluator.mmmu_dataset_evaluator.normalize_str(string)

Normalize the str to lower case and make them float numbers if possible.

flagevalmm.evaluator.mmmu_dataset_evaluator.extract_numbers(string)

Exact all forms of numbers from a string with regex.

flagevalmm.evaluator.mmmu_dataset_evaluator.parse_open_response(response)

Parse the prediction from the generated response. Return a list of predicted strings or numbers.

flagevalmm.evaluator.mmmu_dataset_evaluator.eval_open(gold_i, pred_i)

Evaluate an open question instance

class flagevalmm.evaluator.mmmu_dataset_evaluator.MmmuEvaluator(is_clean: bool = True, use_llm_evaluator: bool = False, eval_func: Callable | str | None = None, base_dir: str = '', detailed_keys: List[str] | None = None, aggregation_fields: List[str] | None = ['raw_answer'], **kwargs)

Bases: BaseEvaluator

The evaluation method is adapted from the official MMMU benchmark evaluation code (https://github.com/MMMU-Benchmark/MMMU/tree/main/mmmu) with modifications to improve robustness and adapt to the flagevalmm framework.

cal_accuracy(annotation, answers)

Extract Evaluator

class flagevalmm.evaluator.extract_evaluator.ExtractEvaluator(eval_model_name: str, use_llm_evaluator: bool = True, backend: str = 'vllm', port: int = 8001, eval_func: Callable | str | None = None, num_threads: int = 8, eval_method: str = 'extract_compare', **kwargs)

Bases: BaseEvaluator

The evaluation method is implemented to utilize the llm to extract the answer from the model response. Two evaluation methods are supported: 1. Extract + Compare: First extract answer from model response, then compare with ground truth 2. SimpleQA: Directly grade the model response using SimpleQA grading template

__init__(eval_model_name: str, use_llm_evaluator: bool = True, backend: str = 'vllm', port: int = 8001, eval_func: Callable | str | None = None, num_threads: int = 8, eval_method: str = 'extract_compare', **kwargs) → None

extract_answer_by_llm(gt: Dict, pred: Dict)

compare_answer(gt: Dict, extracted_answer: str)

grade_by_simpleqa(gt: Dict, pred: Dict) → Tuple[str, int]

Grade the prediction using SimpleQA grading template

process_single_prediction(pred: Dict, gt: Dict) → Tuple[Dict, int]

Process a single prediction in a thread-safe manner

cal_accuracy(annotations: Dict, predictions: List[Dict], *args, **kwargs) → Dict

process(dataset: Dataset, output_dir: str, **kwargs) → Dict

Parameters:

• dataset (Dataset) – dataset instance

• output_dir – str

Retrieval Evaluator

flagevalmm.evaluator.retrieval_evaluator.i2t(probs: ndarray, return_ranks: bool = False)

flagevalmm.evaluator.retrieval_evaluator.t2i(probs: ndarray, return_ranks: bool = False)

flagevalmm.evaluator.retrieval_evaluator.json_save(content: Dict[str, Any], jf_nm: str) → None

class flagevalmm.evaluator.retrieval_evaluator.RetrievalEvaluator(**kwargs)

Bases: object

__init__(**kwargs)

process(dataset, output_dir, **kwargs)

Common Types

Pre-processing Utilities

flagevalmm.evaluator.pre_process.strip_answer(answer)

flagevalmm.evaluator.pre_process.remove_special_characters(text)

flagevalmm.evaluator.pre_process.process_multiple_choice(answer)

flagevalmm.evaluator.pre_process.remove_unit(value)

flagevalmm.evaluator.pre_process.convert_circled_numbers(text)

flagevalmm.evaluator.pre_process.normalize_string(raw_answer)