YuE2 Real-Audio Encoder: Bring Your Own Music Into YuE2

The missing piece YuE2 never shipped

YuE2-3B generates complete songs from a style prompt and lyrics. Internally it writes "semantic tokens" that decode into audio, but the encoder that maps an existing recording back into those tokens was never released. Without it, there was no way to bring your own music into the model for fine-tuning.

The trick behind the new yue2-mothersuperior-realaudio-tokenizer-v4 release: the model teaches itself. Every song YuE2 generates comes with the exact tokens that produced it, which is a labeled example nobody had to hand-label. The author generated a few thousand songs across many genres, trained a small encoder on those pairs, then adapted it to real recordings by letting YuE2's own decoder grade the encoder: if the tokens rebuilt the real audio, they were right. No token labels for real music were ever needed.

What is in the release

The tokenizer repo ships three pieces:

• Tokenizer head: an 8-layer transformer (d=512) that maps MERT-v2-FullSong layer-20 features to YuE2's 32,768 semantic codes at 25 Hz. Held-out exact match on YuE2's own songs is 16.1% top-1, with near-miss codes rendering almost identically, and NAR round-trip ear tests sitting around 95%.
• NAR-branch LoRA (nar_lora_joint_v4): a rank-32 LoRA on the NAR self-attention and MLP projections, trained jointly with the head on real audio so the decoder renders real-production latents.
• Training and inference scripts so anyone can repeat or extend the pipeline.

The instrumental LoRA

A follow-up instrumental LoRA (ar_lora_inst_v3abc) was trained on roughly 2,700 instrumental tracks across more than 100 genres, paired with chord-annotated ABC scores. It makes YuE2 write instrumental music with a section plan, and it is meant to run with chain-of-thought on (cot="full"), where the model writes its own ABC score first and then the music tokens conditioned on that score.

The lyrics field accepts three styles of structural prompting, trained in equal thirds:

1. Bare: just [instrumental], letting the model choose structure and length.
2. Untimed tags: you pick the section order, the model picks the timing, for example [intro] / [verse] / [chorus] / [bridge] / [chorus] / [outro].
3. Timed tags: each section also gets a start and end time like [verse 0:15-0:45], the strongest structural steer, though the model follows section order and proportions better than absolute end times.

Using it in ComfyUI

Both LoRAs ship a *_comfyui.safetensors variant packed for ComfyUI's native YuE2 layout (fused qkv / gate_up keys):

• nar_lora_joint_v4_comfyui.safetensors loads into a standard LoraLoader on the MODEL output of the YuE2 checkpoint loader.
• ar_lora_inst_v3abc_comfyui.safetensors loads on the CLIP output, because the AR planner lives in the CLIP slot, and requires the YuE2 Generate nodes' mode set to full with a connected ABC node.

The first Reddit version of the instrumental LoRA did not fire in ComfyUI because of ComfyUI's layer naming conventions; the author published the compatible file within minutes, and users in the thread confirmed the fix with reactions like "It's so, so good. You're a wizard".

The ecosystem effect

The encoder unlocks a wave of follow-on work that was previously blocked: the author's own Hum-to-Song adapter (hum a melody, YuE2 produces the full song), a community LoRA trainer node pack, and genre or artist LoRAs trained on real recordings. The author notes LoRA training for YuE2 "will begin to show up everywhere now", since anyone can train their own music into a LoRA using these scripts.

Availability

The weights are free on Hugging Face: the real-audio tokenizer v4, the instrumental LoRA, and the hum-to-song adapter. In ComfyUI, pair them with the native YuE2 pipeline nodes; the ComfyUI-compatible LoRA files load directly with stock LoRA loaders, no conversion needed.