SakanaAI/text-to-lora: Hypernetworks that adapt LLMs for specific benchmark tasks using only textual task description as the input

Install uv if you don’t have uv (see https://docs.astral.sh/uv/getting-started/installation/)

With uv installed, run the following to install the dependencies.

git clone https://github.com/SakanaAI/text-to-lora.git
cd text-to-lora
# make sure you have `uv` installed
# (see https://docs.astral.sh/uv/getting-started/installation/)
uv self update
uv venv --python 3.10 --seed
uv sync
# we use the following wheel for installation
# you might have to change the wheel to be compatible with your hardware
uv pip install https://github.com/Dao-AILab/flash-attention/releases/download/v2.6.3/flash_attn-2.6.3+cu123torch2.3cxx11abiFALSE-cp310-cp310-linux_x86_64.whl
uv pip install src/fishfarm

Downloading trained T2L

⚠️ You need to download the checkpoints before running any of the demos. ⚠️

⚠️ You need a >16GB GPU to handle both models simultaneously to run any of these demos. ⚠️

uv run huggingface-cli login
uv run huggingface-cli download SakanaAI/text-to-lora --local-dir . --include "trained_t2l/*"

Web UI

This demo runs Mistral-7B-Instruct-v0.2 locally alongside a T2L model.

# start webui locally
uv run python webui/app.py

LoRA generation from CLI

This script allows us to generate a LoRA based on a task description. Running for the first time would take longer as the base model will be downloaded and cached.

# uv run generate_lora.py {T2l_DIRECTORY} {TASK_DESCRIPTION}
# e.g.,
uv run python scripts/generate_lora.py \
trained_t2l/llama_8b_t2l \
"This task challenges your problem-solving abilities through mathematical reasoning. You must carefully read each scenario and systematically work through the data to compute the final outcome."

# You might be able to run T2L w/ `gemma-2-2b-it` with a smaller GPU.
uv run python scripts/generate_lora.py \
trained_t2l/gemma_2b_t2l \
"This task challenges your problem-solving abilities through mathematical reasoning. You must carefully read each scenario and systematically work through the data to compute the final outcome."

Evaluating generated LoRA

We can evaluate the generated LoRA by using the path printed at the end of the above script.

# uv run python scripts/run_eval.py --model-dir {base_model_dir} \
# --lora-dirs {lora_dirs} --save-results --tasks {tasks}
# e.g.,
uv run python scripts/run_eval.py \
--model-dir meta-llama/Llama-3.1-8B-Instruct \
--lora-dirs {PATH_TO_GENERATED_LORA} \
--save-results --tasks gsm8k

# You might be able to run T2L w/ `gemma-2-2b-it` with a smaller GPU.
uv run python scripts/run_eval.py \
--model-dir google/gemma-2-2b-it \
--lora-dirs {PATH_TO_GENERATED_LORA} \
--save-results --tasks gsm8k

Disclaimer ❕

Even with random descriptions, SFT-trained T2L still generates reasonable LoRAs.
This result with SFT-trained T2L differs from Table 5 in the paper that uses reconstruction-trained T2L.
Nonetheless, comparing the results obtained between aligned and unaligned descriptions, aligned LoRAs perform noticeably better overall. The training descriptions can be found in tasks/ folder, while the unseen ones used for evaluation can be found in configs/hyper_lora_decontam_lol_tasks.yaml

For asynchronous validation evaluation, we need a separate evaluator script.
The `watcher.py` checks for new checkpoints and evaluates them as they get saved.
The script also keeps track of which one is the best checkpoint so far.

# start a watcher process for async eval
uv run watcher.py

Then run one of the following scripts for each GPU you have.
Each takes around 5 days on a single H100 GPU.

# T2L training
./scripts/train_t2l_mistral.sh
./scripts/train_t2l_llama.sh
./scripts/train_t2l_gemma.sh

First, we need to train the “oracle” adapters for all tasks.
This script trains hundreds of LoRAs and might run for many hours.

./scripts/train_lora_baselines.sh

Then, we train T2L to reconstruct the oracle adapters.

# train T2L via reconstruction training
WANDB_MODE=disabled uv run python scripts/train_hyper_recon.py configs/hyper_lora_decontam_lol_tasks.yaml \
--model_dir=mistralai/Mistral-7B-Instruct-v0.2/ \
--emb_model=Alibaba-NLP/gte-large-en-v1.5 \
--warmup_frac=0.1 --lr=1e-3 --epochs=10000 \
--n_train_ds=479 --exp_setup=hyper_lora --encoder_type=linear \
--pred_z_score=True --n_descs_per_ds=128 --n_embs_per_sampled_task=1 \
--n_tasks_per_batch=4 --factorized=False --delta_w_scaling=10000 --shared_AB_head=True

Base model

./scripts/eval_base_models.sh

T2L

# example for T2L trained for gemma-2-2b-it
WANDB_MODE=disabled uv run python scripts/eval_hypermod_checkpoint.py --checkpoint_path trained_t2l/gemma_2b_t2l/hypermod.pt --full_eval --use-icl

--use-icl includes 3-shot in-context examples into evaluation queries.

Disclaimer ❕

We have re-trained a new set of baselines and T2L models due to a small mismatch
between the specific package version combinations used in the original submission.

Furthermore, vLLM is inherently non-deterministic when applying LoRA (in the version-configuration combinations we tested).
Thus, we have observed some small variance between evaluation runs even with a fixed initial seed.

However, after re-retraining and collecting new results, T2L still consistently outperforms baselines across model families, confirming the validity of our original empirical analysis. We present this new set of results with the updated package versions in the following tables.
(eval #1 and eval #2 indicate that the results are gathered from two different evaluation calls with the same random seed.)

Mistral-7B-Instruct-v0.2

Llama-3.1-8B-Instruct

Gemma-2-2b-it

Huggingface datasets server might reject connections sometimes, which can be problematic if you haven’t locally cached the relevant datasets.
The cause could be that we’re downloading a lot of datasets (~500 datasets).
If this happens, keep retrying the SFT training script until all datasets are cached locally.
Once the datasets are cached, you should be able to run the SFT training script.

@inproceedings{
    charakorn2025texttolora,
    title={Text-to-Lo{RA}: Instant Transformer Adaption},
    author={Rujikorn Charakorn and Edoardo Cetin and Yujin Tang and Robert Tjarko Lange},
    booktitle={Forty-second International Conference on Machine Learning},
    year={2025},
    url={https://openreview.net/forum?id=zWskCdu3QA}
}

Keep your files stored safely and securely with the SanDisk 2TB Extreme Portable SSD. With over 69,505 ratings and an impressive 4.6 out of 5 stars, this product has been purchased over 8K+ times in the past month. At only $129.99, this Amazon’s Choice product is a must-have for secure file storage.

Help keep private content private with the included password protection featuring 256-bit AES hardware encryption. Order now for just $129.99 on Amazon!

Help Power Techcratic’s Future – Scan To Support

If Techcratic’s content and insights have helped you, consider giving back by supporting the platform with crypto. Every contribution makes a difference, whether it’s for high-quality content, server maintenance, or future updates. Techcratic is constantly evolving, and your support helps drive that progress.

As a solo operator who wears all the hats, creating content, managing the tech, and running the site, your support allows me to stay focused on delivering valuable resources. Your support keeps everything running smoothly and enables me to continue creating the content you love. I’m deeply grateful for your support, it truly means the world to me! Thank you!