Note: I've been running some automated LoRA loading tests here Nerogar/OneTrainer#1544
Many issues on OneTrainer's side to fix, but where I think it's a diffusers issue I'm reporting it upstream, but these can be about old or even obsolete features such as support Kohya-format LoRAs for SD 1.5 as a result.
Feel free to close if you think an issue isn't worth reporting anymore.
Describe the bug
Loading a Kohya-format UNet LoRA that touches conv_in, conv_out, or time_embedding.linear_1/linear_2 produces unexpected keys and those layers' weights aren't applied. Reproduced on SD 1.5, SD 2.1, and SDXL.
_convert_unet_lora_key() (diffusers/loaders/lora_conversion_utils.py) converts Kohya-style keys by replacing every _ with . and then patching specific substrings back (input.blocks→down_blocks, proj.in→proj_in, conv.shortcut→conv_shortcut, etc.):
|
def _convert_unet_lora_key(key): |
|
""" |
|
Converts a U-Net LoRA key to a Diffusers compatible key. |
|
""" |
|
diffusers_name = key.replace("lora_unet_", "").replace("_", ".") |
|
|
|
# Replace common U-Net naming patterns. |
|
diffusers_name = diffusers_name.replace("input.blocks", "down_blocks") |
|
diffusers_name = diffusers_name.replace("down.blocks", "down_blocks") |
|
diffusers_name = diffusers_name.replace("middle.block", "mid_block") |
|
diffusers_name = diffusers_name.replace("mid.block", "mid_block") |
|
diffusers_name = diffusers_name.replace("output.blocks", "up_blocks") |
|
diffusers_name = diffusers_name.replace("up.blocks", "up_blocks") |
|
diffusers_name = diffusers_name.replace("transformer.blocks", "transformer_blocks") |
|
diffusers_name = diffusers_name.replace("to.q.lora", "to_q_lora") |
|
diffusers_name = diffusers_name.replace("to.k.lora", "to_k_lora") |
|
diffusers_name = diffusers_name.replace("to.v.lora", "to_v_lora") |
|
diffusers_name = diffusers_name.replace("to.out.0.lora", "to_out_lora") |
|
diffusers_name = diffusers_name.replace("proj.in", "proj_in") |
|
diffusers_name = diffusers_name.replace("proj.out", "proj_out") |
|
diffusers_name = diffusers_name.replace("emb.layers", "time_emb_proj") |
|
|
|
# SDXL specific conversions. |
|
if "emb" in diffusers_name and "time.emb.proj" not in diffusers_name: |
|
pattern = r"\.\d+(?=\D*$)" |
|
diffusers_name = re.sub(pattern, "", diffusers_name, count=1) |
|
if ".in." in diffusers_name: |
|
diffusers_name = diffusers_name.replace("in.layers.2", "conv1") |
|
if ".out." in diffusers_name: |
|
diffusers_name = diffusers_name.replace("out.layers.3", "conv2") |
|
if "downsamplers" in diffusers_name or "upsamplers" in diffusers_name: |
|
diffusers_name = diffusers_name.replace("op", "conv") |
|
if "skip" in diffusers_name: |
|
diffusers_name = diffusers_name.replace("skip.connection", "conv_shortcut") |
|
|
|
# LyCORIS specific conversions. |
|
if "time.emb.proj" in diffusers_name: |
|
diffusers_name = diffusers_name.replace("time.emb.proj", "time_emb_proj") |
|
if "conv.shortcut" in diffusers_name: |
|
diffusers_name = diffusers_name.replace("conv.shortcut", "conv_shortcut") |
|
|
|
# General conversions. |
|
if "transformer_blocks" in diffusers_name: |
|
if "attn1" in diffusers_name or "attn2" in diffusers_name: |
|
diffusers_name = diffusers_name.replace("attn1", "attn1.processor") |
|
diffusers_name = diffusers_name.replace("attn2", "attn2.processor") |
|
elif "ff" in diffusers_name: |
|
pass |
|
elif any(key in diffusers_name for key in ("proj_in", "proj_out")): |
|
pass |
|
else: |
|
pass |
|
|
|
return diffusers_name |
There's no patch for:
conv.in → conv_inconv.out → conv_outtime.embed.0 / time.embed.2 → time_embedding.linear_1 / time_embedding.linear_2
So a Kohya key like lora_unet_conv_in.lora_A.weight comes out as conv.in.lora_A.weight instead of conv_in.lora_A.weight, and the PEFT adapter loader reports it as an unexpected key instead of applying it to the model's conv_in module.
Reproduction
import torch
from diffusers import DiffusionPipeline
pipe = DiffusionPipeline.from_pretrained("stable-diffusion-v1-5/stable-diffusion-v1-5", torch_dtype=torch.bfloat16)
pipe.load_lora_weights("path/to/kohya_format_lora.safetensors")Loading adapter weights from state_dict led to unexpected keys found in the model: conv.in.lora_A.default_0.weight, conv.in.lora_B.default_0.weight, conv.out.lora_A.default_0.weight, conv.out.lora_B.default_0.weight, time.embedding.linear.lora_A.default_0.weight, time.embedding.linear.lora_B.default_0.weight.
Same result on sd2-community/stable-diffusion-2-1 and stabilityai/stable-diffusion-xl-base-1.0.
Logs
Loading adapter weights from state_dict led to unexpected keys found in the model: conv.in.lora_A.default_0.weight, conv.in.lora_B.default_0.weight, conv.out.lora_A.default_0.weight, conv.out.lora_B.default_0.weight, time.embedding.linear.lora_A.default_0.weight, time.embedding.linear.lora_B.default_0.weight.
System Info
- diffusers commit: 0f1abc4 (main)
- Python: 3.12
- Platform: Linux
Who can help?
@yiyixuxu @sayakpaul
Note: I've been running some automated LoRA loading tests here Nerogar/OneTrainer#1544
Many issues on OneTrainer's side to fix, but where I think it's a diffusers issue I'm reporting it upstream, but these can be about old or even obsolete features such as support Kohya-format LoRAs for SD 1.5 as a result.
Feel free to close if you think an issue isn't worth reporting anymore.
Describe the bug
Loading a Kohya-format UNet LoRA that touches
conv_in,conv_out, ortime_embedding.linear_1/linear_2produces unexpected keys and those layers' weights aren't applied. Reproduced on SD 1.5, SD 2.1, and SDXL._convert_unet_lora_key()(diffusers/loaders/lora_conversion_utils.py) converts Kohya-style keys by replacing every_with.and then patching specific substrings back (input.blocks→down_blocks,proj.in→proj_in,conv.shortcut→conv_shortcut, etc.):diffusers/src/diffusers/loaders/lora_conversion_utils.py
Lines 260 to 313 in 0f1abc4
There's no patch for:
conv.in→conv_inconv.out→conv_outtime.embed.0/time.embed.2→time_embedding.linear_1/time_embedding.linear_2So a Kohya key like
lora_unet_conv_in.lora_A.weightcomes out asconv.in.lora_A.weightinstead ofconv_in.lora_A.weight, and the PEFT adapter loader reports it as an unexpected key instead of applying it to the model'sconv_inmodule.Reproduction
Same result on
sd2-community/stable-diffusion-2-1andstabilityai/stable-diffusion-xl-base-1.0.Logs
System Info
Who can help?
@yiyixuxu @sayakpaul