Installation¶

This guide will help you install FlagEvalMM and its dependencies.

Basic Installation¶

The easiest way to install FlagEvalMM is from the source repository:

git clone https://github.com/flageval-baai/FlagEvalMM.git
cd FlagEvalMM
pip install -e .

FlagEvalMM supports multiple backend engines for inference. Choose and install the ones you plan to use:

Currently (July 28, 2025), we recommend using vllm==0.8.5.post1 and torch==2.6.0 for optimal inference performance and stability.

pip install vllm==0.8.5.post1

Warning

Make sure to use the recommended versions for the best compatibility and performance.

pip install --upgrade pip
pip install "sglang[all]"
pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.4/

For detailed installation instructions, please refer to the official SGLang documentation.

pip install lmdeploy

For detailed installation instructions, please refer to the official LMDeploy documentation.

git clone https://github.com/FlagOpen/FlagScale.git
cd FlagScale/install
./install-requirements.sh --env inference
cd vllm
pip install .

For detailed installation instructions, please refer to the official FlagScale documentation.

For optimal performance with transformers, we recommend installing flash-attention:

pip install flash-attn --no-build-isolation

If you want to evaluate tasks using GPT models (like charxiv, math_verse, etc.), you need to set the following environment variables:

export FLAGEVAL_API_KEY=$YOUR_OPENAI_API_KEY
export FLAGEVAL_BASE_URL="https://api.openai.com/v1"

For other API providers, adjust the base URL accordingly.

To verify your installation, run:

flagevalmm --help

You should see the help message with available command-line options.

Import Error: If you encounter import errors, make sure all dependencies are installed correctly and your Python environment is activated.

CUDA Issues: For GPU-related problems, verify that your CUDA version is compatible with your PyTorch installation.

Memory Issues: For large models, ensure you have sufficient GPU memory. Consider using model sharding or reducing batch sizes.

If you encounter issues during installation: