KenSchae
/
Pallaidium
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
Generative AI for the Blender VSE: Text, video or image to video, image and audio in Blender Video Sequence Editor.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
122 Commits
9 Branches
0 Tags
33 MiB
 
Tree: aba2130e28
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'aba2130e28'
${ noResults }
Pallaidium/README.md

3.8 KiB
Raw Blame History

Generative AI - Text to Video, Image and Audio in Blender

AI generate video, image and audio from text prompts or strips.

Video Examples

Modelscope & Bark

Watch the video Watch the video

Features

  • Text to video
  • Text to audio
  • Text to image
  • Seed, Quality Steps, Frames, Word Power.
  • Batch operation of text strips into videos, audios or images.
  • Model card selector
  • Render finished notification
  • Model Cards: Modelscope, Animov, Stable Diffusion 1.5 & 2, AudioLMD and Bark

How to install (on Win)

Download the add-on: https://github.com/tin2tin/text_to_video/archive/refs/heads/main.zip

Right click on the Blender icon and "Run Blender as Administrator".

Install the add-on as usual: Preferences > Add-ons > Install > select file > enable the add-on.

In the add-on preferences, hit the "Install all Dependencies" button.

The first time a model is executed and many GB will have to be downloaded, so go grab lots of coffee.

If it says git is missing, you can find it here: https://git-scm.com/download/win

As for Linux and MacOS, if anything differs in installation, then please share instructions.

Location

Video Sequence Editor > Sidebar > Generative AI

image image image

Converting Text strips into GeneratorAI strips:

image

Install Dependencies, set Movie Model Card and set Sound Notification in the add-on preferences:

image

Tip

If the image of your renders breaks, then use the resolution from the Model Card in the Preferences.

Text to Audio

AudioLDM

Find AudioLDM documentation here: https://github.com/haoheliu/AudioLDM Try prompts like: Bag pipes playing a funeral dirge, punk rock band playing hardcore song, techno dj playing deep bass house music, and acid house loop with jazz. Or: Voice of God judging mankind, woman talking about celestial beings, hammer on wood.

Bark

Find Bark documentation here: https://github.com/suno-ai/bark

  • [laughter]
  • [laughs]
  • [sighs]
  • [music]
  • [gasps]
  • [clears throat]
  • — or ... for hesitations
  • ♪ for song lyrics
  • capitalization for emphasis of a word
  • MAN/WOMAN: for bias towards speaker

Tip

If the audio breaks up, try processing longer sentences.

Modules

Diffusers: https://github.com/huggingface/diffusers

ModelScope: https://modelscope.cn/models/damo/text-to-video-synthesis/summary

AudioLDM:

https://huggingface.co/cvssp/audioldm-s-full-v2

https://github.com/haoheliu/AudioLDM

Bark: https://github.com/suno-ai/bark

If some additional python modules are missing, write about it here(so I can add them), and use the Blender PIP add-on to manually install the missing modules:

https://github.com/amb/blender_pip

Disclaimer for using the Modelscope model:

  • The model can only be used for non-commercial purposes. The model is meant for research purposes.
  • The model was not trained to realistically represent people or events, so using it to generate such content is beyond the model's capabilities.
  • It is prohibited to generate content that is demeaning or harmful to people or their environment, culture, religion, etc.
  • Prohibited for pornographic, violent and bloody content generation.
  • Prohibited for error and false information generation.