Running on AMD GPU
Direction from the BigScience team
Last updated
Direction from the BigScience team
Last updated
Following the great instructions from August and using the docker image, this runs on the 7900 XTX with a few changes, most notably
The rest of the steps are the same
Due to the great work of Odonata (Discord, github ), the hardware of oceanmasterza (Discord), and the help of epicx (Discord, GitHub ), we have the below AMD instructions.
According the the author of the port , using a Docker image is recommended (both and should work). See port discussion .
On host machine, run:
In the running image, run:
Running the model in bfloat16
is also supported but slower than in float16
.
Multi-GPU process (--tensor_parallel_devices
) is still not tested (docker --gpu
flag may not function at this time and other virtualization tools may be necessary).
Tested on:
AMD 6600 XT tested July 24th, 2023 on Arch Linux with Rocm 5.6.0, mesa 22.1.4
AMD 6900 XT tested April 18th, 2023 on bare metal Ubuntu 22.04 (no docker/anaconda/container). Tested with ROCM 5.4.2
Untested on 7000 series, however 7000s may have much better performance as AMD added machine learning tensor library and better hardware support (vs ray tracing only on 6000 series)
Guide:
use the mesa-clover and mesa-rusticl opencl variants
add export HSA_OVERRIDE_GFX_VERSION=10.3.0
to your environment (put it to /home/user/.bashrc
on ubuntu - this tricks ROCM to work on more consumer based cards like the 6000 series)
create and activate a venv for the subnet using python 3.11
python -m venv <yourvenvpath>
cd <yourvenvpath>
source bin/activate
install the subnet LLM template version with AMD GPU support:
Tip: You can set your fans to full speed or close to it before starting the subnet (the default Linux fan profile for AMD GPUs is not good on some cards): rocm-smi --setfan 99%
run the subnet using:
Tip: You can monitor temperature and voltage by running this: rocm-smi && rocm-smi -t
Contributed by: ,
install ROCM. Use this tutorial for Arch Linux:
in the venv install pytorch, nightly version, with the command generated on by the website:
This branch uses an older version of bitsandbytes
to have AMD GPU support (developed by and ). This means that you won't be able to use the 4-bit qunatization (--quant_type nf4
) and LoRA adapters (the --adapters
argument). The server will use 8-bit quantization (int8) for all models by default.