Automatic lipsync in Blender

Following by a previous tutorial about making lipsync using Papagayo, here is another one showing how to apply lipsync data to Blender model.

We have made a special version of official Papagayo Lipsync addon, which implements a special approach for adding lipsync animation to your models.

With this approach you use a Pose Library to define a mouth poses for the face rig of your model. Each pose should be named as corresponding phoneme in your lipsync breakdown  – “MBP”, “AA”, “O”, “EHSZ”, “etc”… (just inspect the .dat file produced by Papagayo to figure out which phonemes you need).

screen-8

The addon takes a Papagayo’s .dat file and sets keyframes based on mouth shapes defined in your Pose Library.

This way you can easily apply lipsync animation to the models controlled by Rigify, CGCookie Flex Rig and other similar rigs.

Below you can see a quick tutorial by Terry Hancock, demonstrating the use of the script for Lunatics project.

Liked it? Take a second to support us on Patreon!
Posted in Blender, Blender, Papagayo and tagged , , .

19 Comments

  1. Hey guys, i’m trying to use this addon, but in 2.76x it doesnt seem to appear in the toolshelf, maybe it needs to be assigned to a TAB?, because i believe it was made in an era where tabs didnt exist, can you fix it?, it’d be great!!
    Cheers!

  2. Pingback: New animation: "What is Etiquette 3D" - Morevna Project

  3. The video tutorial is too short and some important parts are missing ? Infact it starts with the pose library full of phoenemes,but I don’t know how to add them into the library. Or better,generally I know how to add a pose,but in this specific case,I don’t know which kind of phoenemes are those and how to create them.

  4. Hello Mario! Please take a look at the text in the blog post right above the video: It says: “use a Pose Library to define a mouth poses for the face rig of your model. Each pose should be named as corresponding phoneme in your lipsync breakdown – “MBP”, “AA”, “O”, “EHSZ”, “etc”… (just inspect the .dat file produced by Papagayo to figure out which phonemes you need).”

    So, just open your .dat file and see which phonemes it is uses. Then open your model file and create Poses for your Pose library. Each pose should be named in the same way as phoneme. I.e. for “O” phoneme you need to set bones of your character’s lips in the way as he says “o”. Then save this pose into the Pose Library under the name “O”. I hope that helps.

  5. Yes. It helped. But If I use the MHX2 framework I don’t need shape-keys and/or the pose library. Instead,If I use the rigify script I need to create one pose for each phoneme defined into the papagajo dat file. I did it and it worked. So,from the point of view of the lip synching If I use the makehuman armature is better than If I use rigify. My doubt was that in the short video of Terry Hancock I don’t understand which kind of armature he used. The video is too short. It starts with the poses already created. Instead,for a better understanding,it was better to explain that with rigify it needs to create the mouth poses,but it is not necessary if you use the makehuman armature…

  6. Blender 2.78c. Papagayo import script. Pose library using a matrix of 2D mouth shapes on a plane with base and target bones, then rendered through an image texture node. Just name your poses properly. Import a papagayo .dat of phonemes to mouth shapes generated from .wav. Bring the wave into the NLA editor. Put your bone to pose mode. Set the timeline audio to sync. And generate keyframes.

    This shit still works.

  7. Скажите пожалуйста как использовать папагайо в блендер без использования ключей форм, или я что-то не понимаю, проблема в том, что я работаю с 2д анимацией в блендер.

    • Здравствуйте, Heenash. В этом посте как раз рассказывается как использовать папагайо без ключей форм. Посмотрите видео – там используется Pose Library для того чтобы задавть положение костей для каждой фонемы.

      Если вы работаете с 2D-анимацией в Blender, то необходимо сделать так, чтобы переключение рта задавалось костью (это, как я понимаю, известный приём с координатами текстуры). После этого вы просто делаете Pose Library, где разные положения кости соответствуют разным фонемам. Ну и после этого используйте плагин чтобы импортировать липсинх из Papagayo.

  8. I keep getting an error upon clicking “Plot Keys to the timeline” in the console that says “skipping phoneme: pose already set”, after each supposed key frame. None of the poses end up in the timeline or dopesheet.

    – Joe

  9. Awesome, work. Is there a way to use this add on to any other Rig? for Bone Pose Library. Lets’ say my own made Rig. What should I do with the Python Script Addon ? . Please respond. Thanx.

Leave a Reply