What are Stable Diffusion Hypernetworks? (usage and training tutorial)

Hypernetworks are a fine-tuning technique that enhance the results of your Stable Diffusion generations.

You use hypernetwork files in addition to checkpoint models to push your results towards a theme or aesthetic.

For example, you could use the MJV4 hypernetwork in addition to any checkpoint model to make your results look more like Midjourney.

The alternatives to hypernetworks are:

• LoRAs - more popular, users find they produce better results, file size 100-200MB
• Textual Inversion Embeddings - very small file size,

All of these function similarly, but the underlying training metholody is different.

Using Hypernetworks

You should already have installed AUTOMATIC1111, which can run on Windows, Mac or Google Colab.

The best place to find hypernetworks is civitai.com.

After downloading

To use a hypernetwork, add the following to your prompt:

<hypernet:FILENAME:WEIGHT>

Where FILENAME is the file name of the hypernetwork, excluding the file extension.

AND WEIGHT is the strength of the hypernetwork from 0-1. 1 is the strongest, and 0 is the same as disabling it.

As a shortcut, instead of writing this phrase, you should click on the pink model button under the big “Generate” button.

Click on the Hypernetworks tab, and you will see all the hypernetworks in your stable-diffusion-webui/models/hypernetworks folder.

Click on any and the tag will automatically be inserted into the prompt.

Be aware that the hypernet phrase is not treated as part of the prompt. It merely directs which hypernetworks to use. It will be removed after the hypernetwork is applied. So you cannot use any prompt syntax like [keyword1:keyword2:0.5] with them.

Training your own

Before starting, make sure you have:

• Installed and working Automatic1111
• At least 10 images of the subject or aesthetic you want to train on

Step 1: Crop images

Training images must have a resolution of 512x512. Anything else will add distortion to the end results.

For this, you can use Birme.net - a website that'll automatically resize your images. I'll be training a golumn hypernetwork.

For demonstration purposes, I'm using Gollum from Lord of the Rings.

Step 2: Create a Hypernetworks Sub-Folder

In your stable-diffusion-webui folder, create a sub-folder called hypernetworks.

In the hypernetworks folder, create another folder for you subject and name it accordingly.

Mine will be called gollum.

Inside your subject folder, create yet another subfolder and call it output.

So fa, my file path looks like:

stable-diffusion-webui\hypernetworks\gollum\output

Step 3: Add Your Images

Now, add your resized images to your subject folder:

Step 4: Download Stable Diffusion Checkpoint Model

Go to the Stable Diffusion 2.1 Hugging Face.

Then, you simply download the CKPT file.

B. Move To The Correct Filepath

Once it's downloaded, move it to the following filepath:

C:\Users\Your+Name\Stable Diffusion\stable-diffusion-webui\models\Stable-diffusion

When you run the AUTOMATIC1111 Web UI, it will automatically run the latest model of Stable Diffusion.

Step 5: Run Stable Diffusion with AUTOMATIC1111 Web GUI

Click on webui-user.bat to run Stable Diffusion AUTOMATIC 1111 Web GUI.

Enter the following URL in Google Chrome. This is the provided local URL you use to access Stable Diffusion.

You may copy and paste it from below:

http://127.0.0.1:7860

Step 6: Training

A. Creating a Hypernetwork

Firstly, click on the Training Tab on Stable Diffusion's UI.

Then, click on Create Hypernetwork.

Name your hypernetwork. Then, click on Create Hypernetwork.

B. Pre-Processing Images

Then, click on Preprocess Images.

On Preprocess Images, enter the source directory of your hypernet. Then, your destination directory.

This demonstration, it's the sub-folders we created earlier.

Simply, copy and paste their directories from Windows File Explorer to the appropriate text fields.

In our case, the file paths are:

Source Directory

Ours: C:\Users\Joe\Documents\Stable Diffusion\stable-diffusion-webui\Hypernetworks\Gollum

C:\Users\Your_Name\Stable Diffusion\stable-diffusion-webui\Hypernetworks\Model_Name

Destination Directory

Ours: C:\Users\Joe\Documents\Stable Diffusion\stable-diffusion-webui\Hypernetworks\Gollum\Processed

C:\Users\Your_Name\Stable Diffusion\stable-diffusion-webui\Hypernetworks\Model_Name\Processed

Note: Mine is in the documents folder. For some reason, Stable Diffusion fails to run when moved to another library. Avoid making this mistake.

C. Using BLIP for Captioning

Now, click the checkbox for using BLIP to caption your images. This will add a unique caption to each of your images.

If you're using anime images, opt for Deepbooru, which is specialist for anime images.

Then, click on preprocess images.

Once you click preprocess, you may need to wait around 1 minute.

After, go to your processed file. You'll see a text file right next to each of your images.

Your generated text files are descriptions of each image.

D. Select your Hypernetwork

Now, that you've created a Hypernetwork, it's time to use it.

Go back to Stable Diffusion. Then, click on the Train Tab. You need to click on the Train tab next to the Preprocess Images Tab.

Then, select your Hypetnetwork on the dropdown under Hypernetwork.

E. Setting Up The Training

Set the Hypernetwork Learning Rate to 0.0005.

Then, set the max steps to 2,000.

After, set the numbers under 'Save an image to log directory every N steps, 0 to disable' and 'Save a copy of embedding to log directory every N steps, 0 to disable' to 100.

In the dataset directory, copy and paste the filepath of your set of images.

Also, set the prompt template to hypernetwork.

Check the 'Read Parameters (Prompts, etc...) from Text2Image When Generating Previews' checkbox.

Go onto the text2image tab, and enter a prompt of how you want your hypernetwork to be.

Feel free to feature art styles, as I have below:

Then, go back to the hypernetwork tab, and click on Train Hypernetwork.

F. Overcoming the Low VRAM Error

When you try to train your hypernetwork, you may come across a VRAM error - despite having enough RAM to begin with.

To resolve the Low RAM error, first you can try lowering the resolution of your images to 256x256.

Alternatively, switch from lowvram to medvram in the web-user.bat file - if you have 8GB or less.

If you have 4GB or less, stick with lowvram.

G. The Training

Once you've clicked Train Hypernetwork, the training will begin.

It will take up to an hour, so be patient.

After around 60-70 minutes, you'll find 50-100 images in your dated folder - created by the AI. Each image should be better than the others.

The following filepath should appear as follows:

C:\Users\Your_Name\Stable Diffusion\stable-diffusion-webui\Hypernetworks\Output\2023-03-30\Gollum\

In other words, it'll be in a new sub-folder with today's date in your Hyper Network's sub-folder.

You can increase the number of steps to around 2,500. Although, you may find distortion in the final results.

If you get distortion, simply copy and paste the image generated right before distortion occurs. Then, copy it to the hypernetwork's sub-folder.

Then, make the hyper network's sub-folder file-path to the dataset entry.

H. Tackling the CUDA Error

About 1 minute into your training, you may across these errors, which abort the entire process:

To solve this issue, go into your webui.user.bat file, and enter --deepbanbooru -- xformers --precision full --no-half. Then, save the file.

set COMMANDLINE_ARGS= --medram --deepdanbooru --xformers --precision full --no-half

Also, resize the output to 256x256.

After, the model should start training.

Some people suggest overclocking the GPU clock and RAM, but I don't recommend risking your hardware for this. I certainly won't be. If you're willing to take a risk, go for it.

Tips for Training

Conclusion

In conclusion, training a hypernetwork isn't a straightforward task.

Although, if you have the right hardware, you won't come across the 2 RAM errors listed above. Thus, it should work well for you.

Learn More About AI

AI is always changing, at a rapid rate. We get it... It's not always easy to stay up-to-date.

Our newsletter includes the latest updates on Stable Diffusion models, Midjourney, Novel AI, Bing Image Creator, among others. We don't only provide insights into AI advancements, but we also tell you how we feel about them.

If our newsletter takes your fancy, don't hesitate to give it a try.