Fizgig LoRA Isolation Keeps One Face From Spreading to Everyone

Fizgig v8.4.0 adds LoRA Isolation, a training mode for character LoRAs that stops the subject's face from spreading onto everyone else in the picture (release notes | technical write-up | demo video). A normal LoRA of a person pulls that face onto strangers in the background, a friend standing next to them, or a whole crowd. With LoRA Isolation the trained subjects can stand together in one image, or next to someone the model already knows, and each face stays its own. Without the trigger word you get the base model's own people, as before training.

Fizgig LoRA Isolation, demo video by the Fizgig author. Source: Fizgig v8.4.0 release notes

Results

Two LoRAs were trained on Krea 2 with the same photos, settings, seed and epoch, differing only in LoRA Isolation. Both learned two people, the subject plus a friend as a second character, and a paper-cut style. Each LoRA was then rendered on 25 prompts at two seeds, 50 pictures in total. Every picture is a split frame: one side holds a different, carefully described white man, the other side the trained subject, also a white man. That pairing is deliberately the hardest case for leakage. Each half was scored with ArcFace (insightface buffalo_l) against the subject's mean face from 35 training photos, where the same person typically scores 0.45 or more and unrelated people score 0.0 to 0.2.

Likeness does not drop: the subject's own real photos score 0.81 against each other, and these renders reach 0.78 while the LoRA is shared with a second person. A second person in a LoRA is usually the problem; here it is part of the fix.

How it works

Fizgig applies six pieces when Fizgig LoRA Isolation is selected on the Subjects card.

DOP: Differential Output Preservation

Every training picture also trains with its trigger removed from the caption, and on that pass the LoRA is held to the base model's own prediction for the same noised input. Without the trigger the LoRA learns to change nothing, so the subject stays out of prompts that do not name them. Fizgig credits DOP to Ostris and AI-Toolkit, and builds its own layers on top.

Multi-subject identity anchoring

Two or more identities train together, each from its own folder with its own trigger. Two faces learned side by side teach the LoRA that a face belongs to one trigger rather than to the class in general, which is why a second person helps instead of hindering. The test above used a woman as the second subject while every stranger was a man, so the effect is not gender-specific. Even for a LoRA of a single person, training a second person alongside is recommended.

Fused-token class decoupling

The trigger and its class word are merged into one word in every caption, so gwelt, a man trains as gweltman. The plain class token ("man") is what every other man in the model shares, and it is the bridge a subject's features cross to reach other people. Joined onto the trigger, the class word becomes a different token, and the subject's look never attaches to the shared one. Fizgig applies the join while captions are read and leaves the caption files on disk untouched. A trigger has to join cleanly: fyxo plus woman reads as f | yx | ow | oman, which breaks the fused word, so the Subjects card ships a Check a trigger word button that tests the join with the training model's own tokenizer.

Negative-pair compositing in one frame

Each person gets 16 side-by-side pictures: one of their own training photos next to a generated stranger in a single 2:1 frame, with the subject's side chosen at random. These teach "this person is the subject, that person stays a stranger" inside one image, which is exactly where a normal LoRA leaks. The captions describe the stranger first and the subject second, because the text encoders these models use are causal and every word after the trigger is encoded with the trigger already seen.

The strangers are a fixed set of 32 generated people across 8 ethnicities, men and women aged 20s to 50s. The second version of the set is rendered on Qwen Image 2.1 for natural skin texture, so it does not pull a LoRA toward poor skin, and no real person's photo is used. The set downloads with Fizgig's models and takes about 21 MB.

Learning rate scaling and balanced sampling

The side-by-side pictures train at twice the learning rate on their own steps, DOP runs every second step at double weight, and Balance the subjects repeats smaller folders so each subject trains about as often as the largest one.

Class-aware training paths

Each subject is read as a person, an animal or thing, or a style or look. A person gets the fused trigger and the side-by-sides. An animal or thing gets the fused trigger only. A style or look keeps its trigger as written and gets no side-by-sides, since a style should change the whole picture.

The recipe behind the numbers

Training takes about 1.5× as long as a normal LoRA with the same settings.

What is known so far

• Up to 4 subjects train well. Two trained people in one picture plus a person the model already knows works almost every time; three or more trained people in one picture is hit and miss.
• Two subjects of the same class, such as two women, need about 1.6× the epochs, because both feed the same "a woman" preservation pass and that class is held roughly twice as hard.
• Prompt with the joined word, for example gweltman man at a cafe, and describe the other people in the picture. The LoRA keeps them apart on its own; the description is a safety net.
• The method has only been tested on Krea 2 so far. It runs on every model in Fizgig except ACE-Step, and on MiniMax H3 the subjects train from their photos, with clips and voice files left out of the folder.

Using it

The mode lives on the Training tab's Subjects card: choose how many subjects, point each one at a folder, say what it is, then set Your subject and other people to Fizgig LoRA Isolation. Under each subject the card shows the word it trains as, one of its captions, and the side-by-side pictures training will use, so captioning problems show up before the run rather than after it.

Setting up the Subjects card: walkthrough video by the Fizgig author. Source: Fizgig v8.4.0 release notes

Captioning guide: MULTI_PERSON.md. The same release makes multi-concept training, 2 to 4 subject folders, available on every model's Standard LoRA and fine-tune rather than only on H3.

Availability

Fizgig is an open-source LoRA and fine-tune studio for Klein 9B, Krea 2, MiniMax H3, Qwen Image 2.1, Anima, Z-Image Turbo and ACE-Step XL (GitHub), and v8.4.0 is the current release. LoRA Isolation is a training-side feature, so the resulting LoRAs load in ComfyUI like any other LoRA trained in Fizgig.