[Feature]: Research — text→VRMA pipelines, Blender VRMA authoring, and 2D toon visemes on VRM

[rosspeili]

One-line summary

Open a research track for richer motion and mouth presentation: Hugging Face / local text-to-motion → Blender → .vrma, and optional 2D toon-style viseme overlays on VRM (as demonstrated in the wild with CC0 assets) — so AVATAR stays a player while authors can make “anything” offline.

Problem / motivation

Today AVATAR plays bundled + custom .vrma, fires one-shots from Motion Deck, and drives the mouth with amplitude lip sync. That is enough for a companion, but it leaves three gaps that keep showing up in community discussion:

1. Authoring custom motion is still opaque — Settings → Directories → Animations can load any .vrma, yet contributors ask “where do the clips come from?” Docs already list Photo Booth / BOOTH / Blender / Kimodo-style research, but we have not investigated and documented a concrete, repeatable path that lands a clip in the Motion Deck.
2. Text-to-motion is moving fast on Hugging Face, but outputs are usually SMPL / robot / generic skeletons — not VRM humanoid .vrma. Without a retarget → export story, those models stay demos, not AVATAR assets.
3. 3D mouth blendshapes are not the only look — combining 2D sprites with VRM (toon mouth cards / sprite-sheet visemes) is underrated for “stand out” companions. Amplitude cycling on mesh visemes cannot express that style.

Prompt for this issue came from Jin (@dankvr on X · @madjin on GitHub), who pointed at Hugging Face text-to-motion, Blender VRMA pipelines, and the value of 2D+VRM hybrids — plus a concrete demo of 2D toon visemes on a VRM using CC0 Tubby Cats art (tweet, thanks also @VIANVOLAEUS for the tutorial thread).

This issue is discussion + investigation + small tests, not a commitment to ship a generator inside the Electron app.

Proposed solution

Treat this as a research spike with written findings and, if useful, tiny prototypes or docs PRs. Three workstreams that can proceed in parallel:

A. Blender → .vrma (highest leverage for “do anything”)

Jin’s point: if you can generate or hand-key animation in Blender and export VRMA, the rest of AVATAR (custom folder, Motion Deck, stage animation.play / mode: once) already consumes it.

Investigate and write down:

• saturday06/VRM-Addon-for-Blender — VRM Animation import/export (humanoid bone rotations, hips translation, expression preview values; VRM 1.0 armature).
• Minimal happy path: import a VRM humanoid → animate (or paste mocap) → export .vrma → drop into Settings → Directories → Animations → fire from Motion Deck.
• Failure modes worth testing: missing T-pose, expression tracks ignored by our player, look-at unsupported on export, clip that loops when we wanted one-shot (Motion Deck already forces once playback).
• Doc outcome: expand Bring your own .vrma with a short “Blender addon checklist” once verified — still player not authoring tool; we document, we do not bundle Blender.

B. Hugging Face / local text-to-motion → retarget → .vrma

Keep an eye on (and spike one end-to-end when time allows):

Family	Why it matters	Gap for AVATAR
NVIDIA Kimodo (docs, GitHub)	Text (+ constraints) → 3D skeletal motion; commercial-ready humanoid models	Output is not VRMA; needs retarget to VRM humanoid then Blender/UniVRM export
Tencent HY-Motion 1.0	Large DiT / flow-matching text→motion on SMPL-H	Same retarget gap; VRAM-heavy
Others as they appear	New HF spaces / papers	Track license + skeleton format

Research questions (answer in comments / a short notes doc, not in product code first):

1. What is the smallest offline pipeline that ends in a playable .vrma on a stock AVATAR build?
2. Which licenses allow streaming / redistribution of generated clips vs personal use only?
3. Should AVATAR ever call HF APIs? (Default no — local-first; generation stays outside the app, same stance as Kimodo/ARDY in current docs.)
4. How do generated clips interact with Motion Deck (one-shot) vs Animations selection (loop)?

C. 2D toon visemes / sprites on VRM (presentation layer)

Investigate Jin’s demo direction and adjacent practice:

• Reference: @dankvr — rare 2D toon-style visemes on a VRM, CC0 @tubbycatsnft art, tutorial credit @VIANVOLAEUS.
• Related patterns: sprite-sheet / plane mouth overlays driven by viseme indices (common in VRChat / Resonite 2D viseme setups); mesh visemes stay optional.
• AVATAR today: amplitude → VRM expression visemes (aa / ih / …) in the lip-sync path ([Feature]: Improve lip-sync beyond amplitude cycling (viseme quality) #9). No sprite overlay path.

Spike ideas (pick one small experiment, document results):

1. Authoring-only: can a VRM be prepared (Blender / Unity) with a mouth quad + sprite atlas so existing three-vrm expression weights still “work,” or does that require a different driver?
2. Runtime overlay (Electron): billboard / plane in the Three.js scene, parented near the jaw bone, swap UVs or textures from the same amplitude (or a future phoneme) signal — body stays VRM, mouth is 2D.
3. Hybrid: keep 3D body VRMA; only replace mouth presentation for “toon companion” skins.

Success for this spike is a decision memo: shipable in-app, docs-only recipe, or park until #9 / #43 land.

Alternatives considered

• Ship a built-in motion generator — out of scope; GPU, licenses, and UX belong outside the overlay companion.
• Only improve amplitude lip sync ([Feature]: Improve lip-sync beyond amplitude cycling (viseme quality) #9) — still worth doing, but does not unlock custom motion authoring or 2D mouth style.
• Only document Blender — useful, but skipping HF tracking means we rediscover the retarget gap every few months.
• Facial expression presets ([Feature]: VRM facial expressions (presets + triggers) #43) instead of 2D mouths — complementary; emotions ≠ viseme style. This issue should link, not absorb, [Feature]: VRM facial expressions (presets + triggers) #43.

Primary surface

Animations

Constraints you accept

• Should stay usable offline / local-first
• Browser-only mode may remain limited vs Electron

Mock / sketch / reference (optional)

Constraints you accept expanded for this one

• Should stay usable offline / local-first (generation offline or BYO; no mandatory cloud SaaS in the app)
• Browser-only mode may remain limited vs Electron
• AVATAR remains a player — authoring pipelines are documented or optional external tools, not bundled Blender/HF runtimes
• Licensing of generated clips, CC0 packs (e.g. Tubby Cats), and VRM model terms stays on the user — same as custom Directories today

Community note from Jin / dankvr:

Keep an eye on Hugging Face for text to motion, and Blender pipelines. I believe there’s a Blender VRMA plugin so if you can generate animations there you can have and do anything.
Also in terms of animation I feel like combining 2D sprites with VRM is underrated and will help to make stuff stand out when used correctly.

Demo to study: x.com/dankvr/status/1849206006724829430 — 2D toon visemes on VRM + CC0 Tubby Cats; tutorial shout-out to @Vianvolaeus.

Tooling to verify:

• Blender: VRM Add-on for Blender — Animation · repo
• Text→motion: Kimodo SOMA · HY-Motion 1.0
• Existing AVATAR docs: Bring your own .vrma (already mentions Blender + Kimodo/ARDY)

Investigation checklist (for whoever picks this up)

• Reproduce one Blender → .vrma → custom folder → Motion Deck one-shot on desktop
• Note which expression / look-at channels survive into our player
• List 2–3 HF text-to-motion models and their skeleton + license in a table comment
• Attempt (or explicitly defer) one retarget path to VRM humanoid; record blockers
• Watch / break down the dankvr 2D viseme demo; sketch how it would map onto three-vrm + our audio graph
• Recommend: docs-only vs prototype overlay vs wait for [Feature]: Improve lip-sync beyond amplitude cycling (viseme quality) #9/[Feature]: VRM facial expressions (presets + triggers) #43
• Optional: open follow-up issues once a lane is chosen (do not turn this ticket into an epic that never closes)

Out of scope (for this issue)

• Shipping Kimodo / HY-Motion / Blender inside the installer
• Replacing Motion Deck or the stage command layer
• Mandatory cloud inference
• Gamepad / global OS hotkeys (separate from [Feature]: Configurable hotkeys to trigger animations (one-shot, return to selection) #24)

[AUDOSt0ck1ng]

I came across a project earlier this month: https://github.com/Kirakun0328/text-to-vrma. It uses an LLM to generate VRMA files, so I think it is quite relevant to this topic.

I’ve tried it myself, and while it’s usable, the results are not particularly good for movements that require natural or random variations, such as subtle body swaying or idle movements. This might be an interesting limitation to consider when comparing different text-to-motion approaches in #52, rather than just looking at whether they can generate VRMA successfully.

[Vianvolaeus]

Hi, thanks for the attribution!
Please feel free to pick my brain and I'll try and assist where possible.

It's worth noting that such expression spritesheets do not necessarily have to be a quad; though it was a simple enough interpretation for said demo. VRoids use similar tech for things like eyebrow expression - polystrip roughly wrapped to the face geometry.

It is also fairly common in VR platforms for anime style models to use floating geo with alpha details for additive detail on some expressions - eye sparkles, blush, etc - the tech is very much combinable with regular 3D models that still use regular viseme shapekeys / expressions.

I found https://github.com/Menithal/Mals-Blender-VRM0-Utility useful for binding various separate geometry objects in Blender to a shared blendshapeproxy, iirc - may come in useful.

[rosspeili]

@AUDOSt0ck1ng I have also forked Kirakun's repo but havent tried myself. Do you think it would be worth to create a guide around it and fold into animation related docs or create new doc to help new users create their own animations?

Maybe an approach would be to drop ideas amd then we glue a guide and options for different technical level users.

[rosspeili]

@Vianvolaeus nice to see you here and will check these out. Please feel free to suggest approaches for making a guide around this.

This could be more like an RFC I guess but we can open prs that ref it.

Also lmk what you think of avatar so far if you tried it and what would you implement next if not in issues or roadmap already?

Thanks in advance <3

[AUDOSt0ck1ng]

@AUDOSt0ck1ng I have also forked Kirakun's repo but havent tried myself. Do you think it would be worth to create a guide around it and fold into animation related docs or create new doc to help new users create their own animations?

Maybe an approach would be to drop ideas amd then we glue a guide and options for different technical level users.

Perhaps we could just add a brief section mentioning that a similar project can be used to generate VRMAs. That project’s UX is quite simple and intuitive; users should be able to figure it out after reading the README.

However, I have to admit that I don't find the solution of generating VRMAs purely through LLM inference satisfactory enough yet.

[rosspeili]

Agreed @AUDOSt0ck1ng, but for now, I think it's more about giving users a few options and pathways, and minimal guidance with links to native documentation from each approach so they can pick their poison. If someone can create vrmas on their own with a better method, they can simple drop their custom animations folder.

What would you suggest as a concluding step? Should we keep this open for research and testing, or wanna spinoff an issue and patch docs only for now?

[AUDOSt0ck1ng]

Agreed @AUDOSt0ck1ng, but for now, I think it's more about giving users a few options and pathways, and minimal guidance with links to native documentation from each approach so they can pick their poison. If someone can create vrmas on their own with a better method, they can simple drop their custom animations folder.

What would you suggest as a concluding step? Should we keep this open for research and testing, or wanna spinoff an issue and patch docs only for now?

I think it might be better to keep this as an open Discussion rather than trying to finalize the docs.

Right now we only support VRMA, but we may support other formats like VMD in the future, so the available workflows could become more complex and evolve over time. We can keep the docs minimal with a link to the Discussion, where users can share different tools, workflows, and experiments.

Trying to document every possible workflow would probably be difficult to maintain, and I think it's reasonable to expect users to do some research and experimentation themselves.

[rosspeili]

Fair point @AUDOSt0ck1ng and agreed. 🙏