Automatic lipsync in Blender

Following by a previous tutorial about making lipsync using Papagayo, here is another one showing how to apply lipsync data to Blender model.

We have made a special version of official Papagayo Lipsync addon, which implements a special approach for adding lipsync animation to your models.

With this approach you use a Pose Library to define a mouth poses for the face rig of your model. Each pose should be named as corresponding phoneme in your lipsync breakdown  – “MBP”, “AA”, “O”, “EHSZ”, “etc”… (just inspect the .dat file produced by Papagayo to figure out which phonemes you need).

screen-8

The addon takes a Papagayo’s .dat file and sets keyframes based on mouth shapes defined in your Pose Library.

This way you can easily apply lipsync animation to the models controlled by Rigify, CGCookie Flex Rig and other similar rigs.

Below you can see a quick tutorial by Terry Hancock, demonstrating the use of the script for Lunatics project.

Related Posts

35 Responses

  1. Hey guys, i’m trying to use this addon, but in 2.76x it doesnt seem to appear in the toolshelf, maybe it needs to be assigned to a TAB?, because i believe it was made in an era where tabs didnt exist, can you fix it?, it’d be great!!
    Cheers!

    1. Hello, Luciano! Yes, the plugin was written a long time ago, when everything was done through operator panels. ^__^ Thanks for the fix, will apply it to our version.

  2. The video tutorial is too short and some important parts are missing ? Infact it starts with the pose library full of phoenemes,but I don’t know how to add them into the library. Or better,generally I know how to add a pose,but in this specific case,I don’t know which kind of phoenemes are those and how to create them.

  3. Hello Mario! Please take a look at the text in the blog post right above the video: It says: “use a Pose Library to define a mouth poses for the face rig of your model. Each pose should be named as corresponding phoneme in your lipsync breakdown – “MBP”, “AA”, “O”, “EHSZ”, “etc”… (just inspect the .dat file produced by Papagayo to figure out which phonemes you need).”

    So, just open your .dat file and see which phonemes it is uses. Then open your model file and create Poses for your Pose library. Each pose should be named in the same way as phoneme. I.e. for “O” phoneme you need to set bones of your character’s lips in the way as he says “o”. Then save this pose into the Pose Library under the name “O”. I hope that helps.

  4. Yes. It helped. But If I use the MHX2 framework I don’t need shape-keys and/or the pose library. Instead,If I use the rigify script I need to create one pose for each phoneme defined into the papagajo dat file. I did it and it worked. So,from the point of view of the lip synching If I use the makehuman armature is better than If I use rigify. My doubt was that in the short video of Terry Hancock I don’t understand which kind of armature he used. The video is too short. It starts with the poses already created. Instead,for a better understanding,it was better to explain that with rigify it needs to create the mouth poses,but it is not necessary if you use the makehuman armature…

  5. Blender 2.78c. Papagayo import script. Pose library using a matrix of 2D mouth shapes on a plane with base and target bones, then rendered through an image texture node. Just name your poses properly. Import a papagayo .dat of phonemes to mouth shapes generated from .wav. Bring the wave into the NLA editor. Put your bone to pose mode. Set the timeline audio to sync. And generate keyframes.

    This shit still works.

  6. Скажите пожалуйста как использовать папагайо в блендер без использования ключей форм, или я что-то не понимаю, проблема в том, что я работаю с 2д анимацией в блендер.

    1. Здравствуйте, Heenash. В этом посте как раз рассказывается как использовать папагайо без ключей форм. Посмотрите видео – там используется Pose Library для того чтобы задавть положение костей для каждой фонемы.

      Если вы работаете с 2D-анимацией в Blender, то необходимо сделать так, чтобы переключение рта задавалось костью (это, как я понимаю, известный приём с координатами текстуры). После этого вы просто делаете Pose Library, где разные положения кости соответствуют разным фонемам. Ну и после этого используйте плагин чтобы импортировать липсинх из Papagayo.

  7. I keep getting an error upon clicking “Plot Keys to the timeline” in the console that says “skipping phoneme: pose already set”, after each supposed key frame. None of the poses end up in the timeline or dopesheet.

    – Joe

  8. Awesome, work. Is there a way to use this add on to any other Rig? for Bone Pose Library. Lets’ say my own made Rig. What should I do with the Python Script Addon ? . Please respond. Thanx.

    1. Yes, you can use it with any other rig! ^__^ Just use a Pose Library to define a mouth poses for the face rig of your model. Each pose should be named as corresponding phoneme in your lipsync breakdown – “MBP”, “AA”, “O”, “EHSZ”, “etc”. Also, make sure to call your pose library “Lib_Phonemes” (or edit the corresponding option in the “Misc” tab, right on top of the “Plot keys to the Timeline” button – https://morevnaproject.org/wp-content/uploads/2017/07/lipsync-papagayo.png).

  9. Could you maybe make a paid addon like that for maya? I would buy it heh…

  10. hello good day guys,had been trying this addon on 2.79b but keeps returning an error what do i do please

      1. This only works with shape keys, do you think you guys will make a version for 2.8 that will support the pose library?

  11. This is all great stuff. thank you to all involved!
    Just so people do not pass this up due to misinformation..
    It was said this was not tested with pose libraries, it did not say it does not work with pose libraries. I just finished testing it, took all of 30 seconds to see that is Does work with pose libraries. I seriously almost ignored this update to 2.8 because of a previous comment that said that it would not work.
    So, to say it loudly.. It DOES WORK with POSE LIBRARIES in 2.8

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: