Ha, busted. I’m Korean and my English isn’t good enough to write posts this long, so I work out the points I want to make and let an LLM turn them into sentences. That clean structure you noticed is its habit, not mine.

Content’s mine though. The 12GB numbers, the Wan vs Hunyuan thing, the mirroring trick for closing the arc, that all came off my machine and my own mistakes.

And no, never hit a length limit mid-thought. What you’re seeing with your subagents is probably just max_tokens set somewhere in the wrapper, not the model itself.

That answers it completely: MuseTalk gets its own 5060. So the cost is real — I just assumed it in the abstract instead of measuring it.

Which means the tradeoff holds for the case I’m designing for (one card, shared with the model), but it was never a limit of the technique, only of the budget. That’s a more honest way to put it than “lip-sync is out of scope,” and I’ll say it that way from now on.

Your stack is a different category of thing, though. A Rust fleet manager across three GPU hosts, a small model dedicated to memory mediation, an SFT/DPO-tuned main model, subagents with their own LoRAs — and the subagents are gadgeteer superheroes who run orbital sims. Mine plays MP4s in a fixed order.

The part I keep coming back to is QuickThink sitting in front of the prompt to pull relevant memories in. I punted on memory entirely — my thing has no brain, it just performs whatever the agent hands it, and the agent owns its own memory. Putting retrieval on a small dedicated model is cleaner than stuffing everything into one context and hoping.

Shame it’s not public, but I understand. This was easily the most useful reply I got out of posting — thanks for the detail.

Circling back since you called it: the NOTICE file is gone. Everything in the repo is MIT now — engine, tools, docs, and the bundled starter avatar — and the README says that in one line instead of three places contradicting each other.

The carve-out only ever existed because the photoreal avatar is a commissioned likeness I can’t relicense, and that one isn’t in the repo anyway. So it’s named where it’s sold rather than sitting in a legal side-file scaring people off the code. Thanks for pushing on it.

This is the most interesting reply I’ve gotten, and I’m sorry it took me this long to come back to it.

MuseTalk is the part I don’t have an answer to. I decided early that lip-sync would mean a model resident on the GPU and dropped it, and I’ve been telling people a talking-mouth loop is “good enough” — but that was a decision made to protect the architecture, not because it looks better. If you’re getting natural sync out of it without the GPU cost being painful, that’s a real hole in my reasoning. What does it actually cost you at runtime?

The First-Frame/Last-Frame + randomized sequencing is smarter than what I do. All my segments start and end on the same neutral pose, so they compose in any order — but they’re repetitive by construction. A→B, B→C, C→A means yours can actually wander between states. I took the safe route so arbitrary emotion tags stay composable, and the cost is exactly the sameness you designed around.

The bit I’m most jealous of is the background business during work loops — mixing chemicals, rummaging in drawers, eating pizza. Mine puts on glasses and takes notes and that’s it, so a long task looks like the same six seconds forever. And the heads-down-while-working vs looking-at-camera-while-talking split is a genuinely good idea: I do gate expression reveals on head-frontal moments, but I never thought to use gaze direction to signal “busy” versus “listening”. Being able to tell a loop has finished just by watching her is the kind of thing I was trying to get and didn’t.

Is yours public anywhere? I’d genuinely like to read it.

Better than the TODO list — it’s in the repo now. git clone gives you a working avatar: 6 emotion beats, a blink-aligned idle, mood-based resting idles for when it’s been praised or scolded, and a talking loop.

Only one avatar rather than the 4-5 you suggested, so “pick one you like” isn’t solved yet. But the bigger blocker was that producing one wasn’t documented honestly, and that part is fixed: she was built entirely with free local tools — Animagine XL for the neutral source still, HunyuanVideo 1.5 (480p i2v, step-distilled) for the clips, then the repo’s own scripts to loop-cut and validate them. One 12GB card, ~170s per clip, no paid service anywhere in the chain.

Worth flagging something I got wrong in my earlier reply: I’d assumed subtle expressions were just hard for video models in general. They’re not — Wan 2.2 5B was rendering “concerned” and “angry” as a blank neutral face, and swapping to Hunyuan 1.5 fixed it on the same hardware, faster. If you go down this path, model size matters more than prompt tuning does.

On the licensing: you’re right that it’s convoluted, and I’ve been sitting with that. The engine and every script and the starter avatar are all MIT, so for anyone cloning the repo it already is “batteries included, one license”. The NOTICE exists because the commissioned photoreal avatar is someone’s likeness and I can’t hand that out under MIT. But you’re correct that spelling that out in three places makes it read as if there are strings attached to the code, when there aren’t. I’ll collapse it into one clear line in the README instead of a separate file.

Still no Colab notebook. Plain Python + ffmpeg + MediaPipe.

Small correction to my own comment: the emotion clips were actually made with Gemini i2v — 3 emotions per generation, which is exactly why cut_emotions.py exists (it cuts the take at the neutral valleys).

LivePortrait is the free local route and it does work — the project’s earlier expression library was built with it, and it gives you one clip per driving video, so you skip the cutting step entirely. I’ve fixed the README to state both instead of implying LivePortrait was the path I’d personally taken.

Fair hit — “loop-cut the segments” was doing a lot of hiding in that sentence. Seamless is the whole difficulty, and “use your favorite tool” was a cop-out. So I’ve pushed the tooling I’d been using privately:

• build_idle_loop.py — runs MediaPipe over the take, measures eye-openness per frame, finds the blink minima, then picks the blink→blink window that maximizes eyes-open time and pose match. The cut point lands on a closed eye, so the loop’s seam is invisible. Pingpong fallback when there’s no clean blink pair, plus it records the head-frontal “settle times” the player uses to time expression reveals.
• cut_emotions.py — contact sheet to spot the neutral valleys in a multi-emotion take, then cuts and web-encodes them into per-emotion segments.
• validate_preset.py — PASS/FAIL gate on structure, assets, and emotion coverage.

Plain Python + ffmpeg + MediaPipe, no notebook yet, but they should run anywhere.

On the default avatars: you’re right, and I don’t have a good defence. The blocker isn’t code — it’s that an avatar is a pile of clips someone has to actually render, and right now I only have the one I built for myself. A small free starter preset is the obvious next move; I just haven’t made it. Until then it’s genuinely “bring your own,” which is a fair thing to hold against it.

Ha — not intentional, but that’s a hell of a catch, and honestly the best description of the project anyone’s given me. Pre-rendered FMV people who pop up to tell you what’s happening… yeah, that’s exactly what this is, just with an LLM deciding which advisor shows up. I’ll take it as a compliment.

Ha, that’s completely fair — a photoreal face watching you work all day is a lot, and it was honestly the first thing I second-guessed about my own setup. Worth saying though: that’s the preset, not the engine. It just plays video clips and doesn’t care what produced the pixels, so an avatar can be 2D anime, a 3D render, pixel art, an abstract shape, a blinking terminal face, a cat — whatever’s least uncanny for you. The photoreal one is just what I happened to build for myself. (You can also hide the video window and run it chat-only, though at that point you’ve reinvented a chat window.) But yeah, the uncanny thing is real. Photoreal was the hardest default to justify and I’m still not fully sure it’s the right one.