Common Module

The common module provides shared utilities and server components.

Server Components

class flagevalmm.server.ServerDataset(task_name: str, task_manager, task_type: str = 'vqa')

Bases: Dataset

Get data from the server

__init__(task_name: str, task_manager, task_type: str = 'vqa') → None

get_data(index: int)

class flagevalmm.server.ModelServer(model_name: str, port: int = 8000, backend: str = 'vllm', extra_args: str | None = None)

Bases: object

Currently, it is only used for vllm server, it will be support SGLang etc. in the future.

__init__(model_name: str, port: int = 8000, backend: str = 'vllm', extra_args: str | None = None)

get_vllm_cmd(args: List)

get_lmdeploy_cmd(args: List)

get_flagscale_cmd(args: List)

get_sglang_cmd(args: List)

launch_server(args: List)

stop()

cleanup()

class flagevalmm.server.server_dataset.ServerDataset(task_name: str, task_manager, task_type: str = 'vqa')

Bases: Dataset

Get data from the server

__init__(task_name: str, task_manager, task_type: str = 'vqa') → None

get_data(index: int)

class flagevalmm.server.model_server.ModelServer(model_name: str, port: int = 8000, backend: str = 'vllm', extra_args: str | None = None)

Bases: object

Currently, it is only used for vllm server, it will be support SGLang etc. in the future.

__init__(model_name: str, port: int = 8000, backend: str = 'vllm', extra_args: str | None = None)

get_vllm_cmd(args: List)

get_lmdeploy_cmd(args: List)

get_flagscale_cmd(args: List)

get_sglang_cmd(args: List)

launch_server(args: List)

stop()

cleanup()

flagevalmm.server.utils.get_meta(task_name: str, server_ip: str, server_port: int, timeout: int = 1000)

flagevalmm.server.utils.get_task_info(server_ip: str, server_port: int, timeout: int = 1000)

flagevalmm.server.utils.submit(task_name: str, model_name: str, server_ip: str, server_port: int, timeout: int = 1000, output_dir: str = '') → Any

flagevalmm.server.utils.get_data(index: int, task_name: str, server_ip: str, server_port: int, timeout: int = 1000) → Any

flagevalmm.server.utils.get_retrieval_data(index: int, task_name: str, data_type: str, server_ip: str, server_port: int, timeout: int = 1000)

flagevalmm.server.utils.parse_args()

flagevalmm.server.utils.process_images_symbol(text: str, dst_pattern: str | None = None) → Tuple[str, List[int]]

flagevalmm.server.utils.load_pil_image(img_paths: List[str], img_idx: List[int], reduplicate: bool = False, reqiures_img: bool = False) → Tuple[List[Image], List[int]]

flagevalmm.server.utils.default_collate_fn(batch: List[Tuple[Any, Any, Any]]) → Tuple[Any, Any, Any]

flagevalmm.server.utils.is_port_occupied(port: int) → bool

flagevalmm.server.utils.get_random_port() → int

flagevalmm.server.utils.merge_args(cfg: Config, task_config_file: str, args: Namespace) → Config

flagevalmm.server.utils.maybe_register_class(cfg: Config, task_config_file: str) → None

Register custom dataset and evaluator classes from config file. :param cfg: Config object containing registration info :param task_config_file: Path to task config file

Common Utilities

This section will be expanded as more common utilities are documented.