I got tired of keeping a Telegram tab open all day just to talk to my local agent, so I built a monitor-resident avatar that fronts it instead — a video-call-style face that performs the replies instead of me reading a wall of text.
A few things that might interest this crowd:
No runtime GPU for the avatar. Instead of live-animating a face, the agent’s reply carries inline emotion-beat tags — [], [] deploy?, [] — and a player selects and blends between ~30 pre-rendered clips. The GPU stays 100% on the model. Code and logs render as chat cards instead of being read aloud, and [] pops a human-in-the-loop approve/cancel that blocks the agent until you answer.
It hooks into a real agent as a connector. The agent’s gateway dials out to a local WebSocket the app hosts, so from the agent’s side the avatar is just “another channel,” indistinguishable from a messenger. Adapters for Hermes and OpenClaw are included, plus a demo mode that runs with zero setup.
Local voice both ways. Edge TTS out (swappable to Qwen3-TTS / MeloTTS / Piper), Silero VAD + faster-whisper in.
Open-core, MIT. The engine is fully usable on its own: drop in a folder of clips named by emotion, point it at your agent. Avatars are pure-data bundles — no code runs when you install one — so you can build your own. The repo ships the engine only; there’s no bundled character.
The engine just plays video clips — it doesn’t care what produced the pixels. So the avatar can be photoreal, 2D anime, a 3D render, pixel art, anything. You can even use a Live2D or VRM model: pre-render its expressions into clips and it becomes a Ghost Vessel preset. A Live2D shell can’t do the reverse — it can’t take live-action footage. Clips are a superset. The trade-off is that you give up real-time rig interactivity (no dragging the head around) and you pay a one-time render step.
Repo + 45s demo: https://github.com/ghdtjrtka/ghost-vessel
The part I’d most like feedback on is the emotion-beat output format — the contract that turns model text into a UI performance. Has anyone here tried similar output contracts to drive an interface from LLM output? Curious what broke on your setup, and whether smaller local models keep the tags straight.
I have one that stitches animations together from several different starting frames A->B, B->C, C->A. I generated a bunch of First Frame / Last Frame videos and randomize them in sequence. It uses MuseTalk for lip sync when it’s talking. It looks much more natural.
Mine is a cartoon scientist in a lab. When she’s working on stuff, in the background (I put her in loops all the time, where she’s just heads down on some task), she like, mixes chemicals and messes with electronics and rummages in drawers and eats pizza. When she’s in conversation mode, she loops through the same set of starting and ending frames, but is actually passively looking at the camera, so it looks like she’s talking to you (or creepily staring at the camera for hours at a time). I can tell if she’s finished a /lab or /troubleshoot loop just by checking her animations.
This is the most interesting reply I’ve gotten, and I’m sorry it took me this long to come back to it.
MuseTalk is the part I don’t have an answer to. I decided early that lip-sync would mean a model resident on the GPU and dropped it, and I’ve been telling people a talking-mouth loop is “good enough” — but that was a decision made to protect the architecture, not because it looks better. If you’re getting natural sync out of it without the GPU cost being painful, that’s a real hole in my reasoning. What does it actually cost you at runtime?
The First-Frame/Last-Frame + randomized sequencing is smarter than what I do. All my segments start and end on the same neutral pose, so they compose in any order — but they’re repetitive by construction. A→B, B→C, C→A means yours can actually wander between states. I took the safe route so arbitrary emotion tags stay composable, and the cost is exactly the sameness you designed around.
The bit I’m most jealous of is the background business during work loops — mixing chemicals, rummaging in drawers, eating pizza. Mine puts on glasses and takes notes and that’s it, so a long task looks like the same six seconds forever. And the heads-down-while-working vs looking-at-camera-while-talking split is a genuinely good idea: I do gate expression reveals on head-frontal moments, but I never thought to use gaze direction to signal “busy” versus “listening”. Being able to tell a loop has finished just by watching her is the kind of thing I was trying to get and didn’t.
Is yours public anywhere? I’d genuinely like to read it.
It has it’s own 5060 in my setup. I… spent a lot on my home lab. MuseTalk was outdated anyway… I had Claude vibecode an update. No, I didn’t open a pull request. While I was at it, I did a bunch to customize and optimize it’s behavior. I actually wrote custom tooling for EVERYTHING. Not one component of my AI stuff runs on out of the box tools.
When everything’s running I have:
An API that serves memory, prompts for different modes, different characters (this gets used when she spawns subagents… which she can do).
I have a custom “fleet manager” called Noesis that I wrote in Rust. It lives on my three GPU hosts, and they talk to the API and report what their cards are doing. The options are
Chatterbox, Musetalk and Quickthink (up to two) each get their own 5060, and yes, my fleet has enough 5060s for all three. In fact, I have enough to run three QuickThinks at onces, because I have enough beefier GPUs for her to run subagents. It’s fun. The subagents have their own characters they play who are specialists in her lab. She can spawn up to two specialists. Which leads to the last thing I can spawn in my fleet:
So she can spawn an orbital simulation one of my coworkers wrote for example and model satellite radio coverage. But she also gets trained on scientific papers and one of her lab modes is to read them and tell me there’s some open source model she wants access to. We download it and vibecode Noesis to be able to manage it. But I don’t usually spawn those. She does. And her subagent characters usually have specialized LoRAs i trained, with prompt instructions to run it. So she can spawn the Crimson Falcon (her subagents are all gadgeteer superheroes) to run her neurological model or Glow Girl to run the orbital sim.
And no, it’s not public anywhere, but I can talk about it for hours.
That answers it completely: MuseTalk gets its own 5060. So the cost is real — I just assumed it in the abstract instead of measuring it.
Which means the tradeoff holds for the case I’m designing for (one card, shared with the model), but it was never a limit of the technique, only of the budget. That’s a more honest way to put it than “lip-sync is out of scope,” and I’ll say it that way from now on.
Your stack is a different category of thing, though. A Rust fleet manager across three GPU hosts, a small model dedicated to memory mediation, an SFT/DPO-tuned main model, subagents with their own LoRAs — and the subagents are gadgeteer superheroes who run orbital sims. Mine plays MP4s in a fixed order.
The part I keep coming back to is QuickThink sitting in front of the prompt to pull relevant memories in. I punted on memory entirely — my thing has no brain, it just performs whatever the agent hands it, and the agent owns its own memory. Putting retrieval on a small dedicated model is cleaner than stuffing everything into one context and hoping.
Shame it’s not public, but I understand. This was easily the most useful reply I got out of posting — thanks for the detail.
Ha, no, not public, but I appreciate you asking twice now.
Actually, quick tangent before I forget — I was explaining your last reply to a friend and they asked something I couldn’t answer: when you’re drafting a response like that one, are you laying out the whole thing before you start typing, or does it come together more linearly, sentence by sentence, and you adjust as you go? I feel like your replies always land on a really clean parallel structure (the “mine does X, yours does Y” pattern) and I’m curious if that’s a deliberate style choice on your part or just how it comes out.
Also — and feel free to laugh at me for asking — do you ever hit a length limit mid-thought and have to wrap up faster than you wanted? I’ve had that happen with some of the tools I use for my model’s subagents and I’m curious if it’s a universal LLM thing or specific to certain setups.
Ha, busted. I’m Korean and my English isn’t good enough to write posts this long, so I work out the points I want to make and let an LLM turn them into sentences. That clean structure you noticed is its habit, not mine.
Content’s mine though. The 12GB numbers, the Wan vs Hunyuan thing, the mirroring trick for closing the arc, that all came off my machine and my own mistakes.
And no, never hit a length limit mid-thought. What you’re seeing with your subagents is probably just max_tokens set somewhere in the wrapper, not the model itself.