Train Your Own YuE2 LoRAs Directly in ComfyUI

What it trains and how

YuE2-3B generates complete songs from a style prompt and lyrics, and internally it has three parts: a 2.2B AR language model that plans the song and writes semantic tokens, a 1.5B NAR flow-matching branch that renders 64-channel VAE latents into sound, and a 48 kHz stereo VAE. The trainer freezes the AR "composer" branch (m-a-p has not released an audio-to-token encoder or its training code) and trains LoRA adapters on the NAR branch only, using the released flow-matching objective.

That makes the result a style, instrumentation and vocal timbre LoRA rather than a voice clone. Training is caption-free: audio files are encoded into VAE latents once and cached to disk, and the trigger word is injected through YuE2's checkpoint-native text prefix ([Tags] your_trigger_word, caption ...). An optional same-named .txt caption per file is supported if you want finer control.

The saved LoRA is a standard *.safetensors in native ComfyUI format, dropped into models/loras/ and loaded with the stock LoraLoaderModelOnly node on the native YuE2 checkpoint. No conversion step.

Requirements

• The single-file native YuE2 checkpoint: yue2_3b_bf16.safetensors (~7.8 GB) from Comfy-Org/YuE2, the same file the native generation nodes use. Use the bf16 file; the INT8 repack cannot be trained.
• 24 GB VRAM recommended (tested on an RTX 5090 Laptop 24 GB).
• The node pack is standalone: the official YuE2 model code (m-a-p's yue2_infer, Apache-2.0, unmodified) is bundled, so no other custom nodes are needed.

Author settings and early feedback

The author's recommended recipe is 5000 steps with weight 2.0, and the LoRA works better with the FP16 model than the convrot variant. Early community feedback is mixed: the first issue reports a trained LoRA having no audible effect, and voice cloning is explicitly listed as not working, so treat this as an experimental tool that is still being worked out in the open. The project already has forks exploring wider training coverage, including one training the acoustic model plus the planner CLIP.

Availability

The trainer is free on GitHub at Starnodes2024/ComfyUI-YuE2-Trainer (51 stars at the time of writing). Install through ComfyUI-Manager or git clone into custom_nodes.