ArturoWolff

A step-by-step guide on how to train a LoRA; part 3!Warning: This guide is based on Kohya_SSThis guide REQUIRES that you read "How I LoRA: A beginners guide to LoRA training | Part 1: Dataset Prep." and "How I LoRA: A beginners guide to LoRA training | Part 2: Training Basics"This guide CAN be ported to Tensor.art's trainer; if you know what you are doing.This guide is an (almost) 1:1 of the following guide: https://civitai.com/articles/3522/valstrixs-crash-course-guide-to-lora-and-lycoris-trainingEdits were made to keep it short and only dive into the crucial details. It also removes a lot of recommendations I DO NOT follow.; for more advanced information, please support the original guide. If you want to do things MY way, keep reading.THE SETTINGS USED ARE BASED ON SDXL, DO NOT FOLLOW IF YOU ARE TRAINING ON V-PRED OR 1.5Testing your LoRAThere are two ways to test a LoRA. During training and after training.During:While on your Kohya_ss, there is a section for a "test" prompt. Use it. If you followed the guide you should have set "save every N epoch" as 1. Meaning that every epoch it will save a model and by proxy, test it with the given prompt.Look at each image, and judge its quality.After (The right way):After training is done, move all your safetensors files to your lora folder on your WebUI instalation. I will asume you have A1111, A1111 Forge or A1111 Re-Forge (the best one).On your WebUI, set yourself up with all the settings you would normally use; checkpoint, scheduler, etc.Copy/paste one of your dataset prompts to the prompt area (This will test overfitting).Navigate to the LoRA subtab and add the first file; ie.: Shondo_Noob-000001.safetensor, this will add the LoRA to the prompt as: <lora:Shondo_Noob-000001:1>; change the :1 to :0.1Set a fixed seed; ie.: 1234567890Scroll down to the "script" area of your WebUI and select X/Y/ZSet your X, Y and Z as "Prompt S/R"On X; write all of your LoRA's filenames; ie.: Shondo_Noob-000001, Shondo_Noob-000002, Shondo_Noob-000003, Shondo_Noob-000004, etc. Depending on how many files you saved, their names, etc. ALWAYS SEPARATE WITH A COMMA.On Y; write all the strength variables from 0.1 to 1, ie.: 0.1, 0.2, 0.3, etc. ALWAYS SEPARATE WITH A COMMA.On Z; write an alternate tag to test flexibility, so, if your prompt is: "fallenshadow, standing, dress, smile", write something like: dress, nude, swimwear, underwear, etc. This will create a grid where instead of wearing a dress, she will be nude, wear a swimsuit, etc. ALWAYS SEPARATE WITH A COMMA.If you did a concept LoRA or a style lora:On your WebUI, set yourself up with all the settings you would normally use; checkpoint, scheduler, etc.Copy/paste one of your dataset prompts to the prompt area (This will test overfitting).Navigate to the LoRA subtab and add the first file; ie.: doggystyle-000001.safetensor, this will add the LoRA to the prompt as: <lora:doggystyle-000001:1>; change the :1 to :0.1Set a fixed seed; ie.: 1234567890Scroll down to the "script" area of your WebUI and select X/Y/ZSet your X and Y as "Prompt S/R"On X; write all of your LoRA's filenames; ie.: doggystyle-000001, doggystyle-000002, Shondo_Noob-000003, doggystyle-000004, etc. Depending on how many files you saved, their names, etc. ALWAYS SEPARATE WITH A COMMA.On Y; write all the strength variables from 0.1 to 1, ie.: 0.1, 0.2, 0.3, etc. ALWAYS SEPARATE WITH A COMMA.Selecting the right fileOnce the process finishes, you should have at least 2 grids, one XY with dress, and another with nude (for example). Or one if you didnt set up an Z grid. Up to you.Now look at the grid and look for the "best" result. Look at the art style bias, pose bias, look bias, etc. The more flexible the better. If on fallenshadow-000005 shondo's pose is always unique but after 000006 she's always standing the same way, ignore 000006+If at some point the art style gets ignored or changes and fixates on it; ignore it.If at some point ANYTHING starts repeating that you don't want; ignore it.The only thing that should repeat at all times is whatever corresponds to the trained concept. If you only trained a wolf with a hat but it should always be a different hat, avoid a file that gives him the same hat on the same pose with the same style.If the result image is identical to the training data; avoid it! You are not here to do the same images as your data, you are here to make new ones, remember?If colors are weird; bad.If shapes are mushy; bad.If angle is always the same; bad (unless you prompted for it).Anything that goes against the concept or the flexibility of it: BAD.Any file that has to be lower than 1 or 0.9: BAD. If your LoRA "works best" at 0.6 strenght, it's shit.THIS IS IT FOR PART 3. Now do some good cool loras.

A step-by-step guide on how to train a LoRA; part 2!Warning: This guide is based on Kohya_SSThis guide REQUIRES that you read "How I LoRA: A beginners guide to LoRA training | Part 1: Dataset Prep."This guide CAN be ported to Tensor.art's trainer; if you know what you are doing.This guide is an (almost) 1:1 of the following guide: https://civitai.com/articles/3522/valstrixs-crash-course-guide-to-lora-and-lycoris-trainingEdits were made to keep it short and only dive into the crucial details. It also removes a lot of recommendations I DO NOT follow.; for more advanced information, please support the original guide. If you want to do things MY way, keep reading.THE SETTINGS USED ARE BASED ON SDXL, DO NOT FOLLOW IF YOU ARE TRAINING ON V-PRED OR 1.5Training: BasicsNow that you have your dataset, you need to actually train it, which requires a training script. The most commonly used script, which I also use, are the Kohya Scripts. I personally use the Kohya-SS GUI, a fork of the SD-Scripts command line trainer.Once you have it installed and open, make sure you navigate to the LoRA tab at the top (it defaults to dreambooth, an older method.)There are a lot of things that can be tweaked and changed in Kohya, so we'll take it slow. Assume that anything I don't mention here can be left alone.We'll go down vertically, tab by tab.Accelerate Launch This tab is where your multi-gpu settings are, if you have them. Otherwise, skip this tab entirely, as the defaults are perfectly fine. Training precision is also here, and should match your Save precision in the following tab, but you won't touch it otherwise.ModelThis tab, as you've likely guessed, is where you set your model for training, select your dataset, etc.Pretrained model name or path:Input the full file path to the model you'll use to train.Trained Model output name:Will be the name of your output file. Name it however you like.Image folder (containing training images subfolders):Should be the full file path to your training folder, but not the one with the X_. You should set the path to the folder that folder is inside of. Ex: "C:/Training Folders/Concept Folder/".Underneath that, there are 3 checkboxes:v2: Check if you're using a SD 2.X model.v_parameterization: Check if your model supports V-Prediction (VPred).SDXL Model: Check if you're using some form of SDXL, obviously.Save trained model as:Can stay as "safetensors". "ckpt" is an older, less secure format. Unless you're purposefully using an ancient pre-safetensor version of something, ckpt should never be used.Save precision:"fp16" has higher precision data, but internally has smaller max values. "bf16" holds less precise data, but can use larger values, and seems faster to train on non-consumer cards (if you happen to have one). Choose based on your needs, but I stick with fp16 as the higher precision is generally better for more complex designs. "float" saves your LoRA in fp32 format, which gives it an overkill file size. Niche usage.MetadataA section for meta information. This is entirely optional, but could help people figure out how to use the LoRA (or who made it) if they find it off-site. I recommend putting your username in the author slot, at least.FoldersAs simple as it gets: Set your output/reg folders here, and logging directory if you want to.Output folder:Where your models will end up when they are saved during/after training. Set this to wherever you like.Regularization directory:Should be left empty unless you plan to use a Prior Preservation dataset from section 3.5, following a similar path to the image folder. Ex: "C:/Training Folders/Regularization/RegConceptA/".ParametersThe bread-and-butter of training. Mostly everything we'll set is in this section: Don't bother with the presets, most of the time.Lora Type: StandardTrain Batch Size:How many images will be trained simultaneously. The larger this number the more VRAM you will use, don't go over 1 if you have low VRAM.Max train steps:RECOMMENDED. Forces training to stop at the exact step count provided, overriding epochs. Useful if you want to stop at a flat 2000 steps or similar. 3000 steps is my recommended cap.Save every n epochs:RECOMMENDED. Saves a LoRA before it finishes every X number of epochs you set. This can be useful to go back to and see where your sweet spot might be. I usually keep this at 1, saving every epoch.Your final epoch will always be saved, so setting this to an odd number can prove useful, such as saving every 3 epochs with a 10 epoch training will give you epochs 3, 6, 9, & 10, giving you a fallback right at the end if it started to overbake.Cache latents & Cache latents to disk:These affect where your data is loaded during training. If you have a recent graphics card, "cache latents" is the better and faster choice which keeps your data loaded on the card while it trains. If you're lacking VRAM, the "to disk" version is slower but doesn't eat your VRAM to do so.Caching to disk, however, prevents the need to re-cache the data if you run it multiple times, so long as there wasn't any changes to it. Useful for tweaking trainer settings. (Recommended).The next settings are calibrated for low VRAM usage; read the original guide if you got VRAM to spare. Anything highlighted was changed for maximum VRAM optimization.LR Scheduler: Cosine With RestartsOptimizer: AdamW8bit.Optimizer extra arguments: weight_decay=0.01 betas=0.9,0.99"Learning Rate": 0.0001As a general note, the specific "text encoder" and "unet" learning rate boxes lower down will override the main box, if values are set in them.LR warmup (% of total steps): 20"LR # cycles": 3"Max resolution": 1024,1024Enable buckets: TrueMin/Max bucket resolution: 256; 2048"Text Encoder & Unet learning rate": 0.001; 0.003No half VAE: Should always be True, imo, just to save you the headache.Network Rank & Network Alpha: 8 / 8Your Alpha should be kept to the same number as your Rank, in most scenarios.Network Dropout:Recommended, but optional. A value of 0.1 is a good, universal value. Helps with overfitting in most scenarios.Advanced (Subtab)We won't touch much here, as most values have niche purposes.Gradient accumulate steps: 1Prior loss weight: 1Keep n tokens:For use with caption shuffling, to prevent the first X number of tags from being shuffled.If using shuffling, this should always be 1 at minimum, which will prevent your instance token from being thrown around.Clip skip:Should be set to the clip skip value of your model. Most anime & SDXL models use 2, most others use 1. If you're unsure, most civit models note the used value on their page.Full bf16 training: FalseGradient Checkpointing: TrueShuffle Caption: TruePersistent Data Loader: FalseMemory Efficient Attention:Use only if you're not on a Nvidia card, like AMD. This is to replace xformers CrossAttention.CrossAttention:xformers, always. (As long as you're on a Nvidia card, which you really should be.)If for whatever reason you can't use xformers, SDPA is your next best option. It eats more ram and is a bit slower, but it's better than nothing.Color augmentation:Do not.Flip Augmentation: FalseMin SNR Gamma: 1Debiased Estimation Loss: FalseBucket resolution steps: 64Random crop instead of center crop: FalseNoise offset type: MultiresMultires noise iterations: 6Multires noise discount: 0.3IP noise gamma: 0.1And that's everything! Scroll to the top, open the "configuration" dropdown, and save your settings with whatever name you'd like. Once you've done that, hit "start training" at the bottom and wait! Depending on your card, settings, and image count, this can take quite some time.Here are some visuals:Once training begins:THIS IS IT FOR PART 2: Training Basics!

How I ControlNet: A beginners guide.A step-by-step guide on how to use ControlNet, and why canny is the best model.Pun intended.Warning: This guide is based on SDXL, results on other models will vary.This guide REQUIRES a basic understanding of image generation, read my guide "How I art: A beginners guide" for basic understanding of image generation.Step 1. Setting up the workflow.For simplicity porpuses I will use the same settings, prompt and seed from the guide "How I art: A beginners guide", any changed done to these settings will be mentioned as part of the guide. Further rules will also be metioned as the guide continues.Now that we have our set up ready, let's dive into each ControlNet available.If you are using tensor.art for your generations, click on "Add ControlNet" under the "Add LoRA" and "Add Embedding" options. If you are using your own WebUI for local generation; make sure you have ControlNet installed and the ControlNet models downloaded.A1111 will have them installed by default, but if that's not the case the "extension" tab will have all you need. If you need a guide for how to install ControlNet on your WebUI, let me know in the comments.Step 2. Select a model.The most popular ones are "Openpose", "Canny", "Depth", "Scribble" and "Reference Only".For this guide I will only demonstrate "Openpose", "Canny" and "Reference". Since Depth and Scribble are practically image to image in my personal opinion, and I never use them.IF YOU WANT A GUIDE FOR ANY OF THESE MODELS LET ME KNOW IN THE COMMENTS AND I WILL DO AN INDEPENDENT GUIDE AROUND IT.Step 2.1. (For Openpose).On the Openpose window, you will either upload your reference image, or use one from the Reference list. For this guide I will use the 5th image, because it looks like a jojo pose. For this reason I also will add to my prompt:menacing \(meme\)Once your image is uploaded or the reference image is confirmed, you will be moved to the "control image" tab.This will show the pose that the openpose model detected. It goes without saying, but the more complex the pose is, the harder it is for Openpose to disect it –and as for your base model, the prompt work will decide how it gets interpreted post image control.Click on Confirm and then Generate.Much like the original pose I would say, but we aren't done yet.Step 2.1.2. SettingsOpenpose comes with 2 settings; Weigth and Control Steps.Much like a lora, the first one will dictate how much FORCE the Openpose will have on the image.Meanwhile; Control Steps will dictate for HOW LONG the Openpose will be on effect, where 1 is a 100% of the generation process. For reference: 0.5 means 50%, etc.These settings for example, Openpose will have HALF the impact, and will only take effect after 20% of the generation process has passed –to which it will stop at 70%.This means that if your image was 100 steps, Openpose starts at step 20 and ends at step 70. Every other step is controlled by the base model and prompt alone (Or LoRA if applicable).Let's see what happens:As you can see, the pose was completely ignored, so be careful what and how you change these settings. Only change them if you have to.My personal recomendation are:Never change the starting step.Only lower the strength of the model if the effect is too strong.Only lower the finishing step if you want some deviancy.Setting strenght at 0.5 and steps at 0 - 0.5 (meaning it stops at 50%) shows that the pose is still used (see the hand posture and head tilt), but the model ignores the "full body" representation.Step 2.2.1. (For Canny).Canny (Cunny) is what I like to call, a "recoloring" tool but also the best option for pose reference, angle, composition and all around what "Reference" should actually do. I will show multiple uses for this model as it is the most flexible in my opinion.Step 2.2.2. Canny to recolour.As seen above, we will first learn canny for recolouring porpuses. Remember that YCH you saw for 5k usd? I do. So, load the image and wait for the model to process it. For this step I will use this YCH I found on Twitter.The settings in Canny are the same as in Openpose and the effects are going to be the same as well. You will need to edit them as you go, for there is no such "one configuration to rule them all".I will also remove the "menacing" and "solo" tags and instead will add:solo focus, duo, flexing, looking away, musclesThe image is two characters but focuses on one, the character is not looking at the camera and is flexing his arm. When using canny, treat your prompt as an img2img prompt. Describe not only what you want, but what you see in the reference image.Without changing anything, let's generate now.Here the image took some aspects of Arturo but most are missing. This is when you start testing different settings. For example, let's lower the strenght to 80% or 0.8:As you can see, the colours are better here, but the character now has a weird shirt on. I will change my prompt to say the character is nude and try again.Almost perfect, right? Just continue playing with the settings until you have something close to what you are looking for, and if use have image editing software; recolour the missing details yourself, then finish the image through img2img.Here are the final settings:masterpiece, best quality, newest, absurdres, highres, solo focus, awff, male, wolf, anthro, bust portrait, smile, nude, nipples, looking away, flexing, muscles, duo, masterpiece, best quality, newest, absurdres, highresI apologize for any imperfections, I am not that invested on these guides to be honest. But I do hope they help.Step 2.2.3. Canny to add details.Canny can also be great to add details to an already generated image. Let's say you used NovelAI for its great poses, but want to add details that Novel simply can't do –canny is the way to go.For this process, switch all of your settings to Img2Img, that means prompt, model, loras, etc. And yes, Controlnet too; selecting Canny and loading the same image as reference.But before you get on your high horses, notice that there are some new wacky options!These are "Denoising strenght" and "Seed", now; Seed isn't exactly NEW, but it is different, because most of the times you won't have the original seed, or you are using a different base model, etc. This; again is all for test porpuses to get the idea across. Therefore, I won't re-use the seed.Denoising strenght tho, it's the important setting here. An Img2Img generation requires you to set how different the image will be from its starting point.Canny makes sure that the image itself doesn't change, but it will still change the colours, background and other things.Since we are just adding details; the denoise shouldn't go above 0.4 or 40% but each model is unique so you will have to learn how your base model works to find the sweet spot.Here's our control image:And here is the result with a 0.4 denoise and default settings on canny:As expected (for me) the image is almost the same, this is because I used the same model for the original image as for the img2img. Let's change the model to something else. I will use KiwiMix-XL - V3 for this, now we run it again, everything else untouched:As expected from Kiwi (for me) the colours changed dramatically to fit the model's art style. Kiwi has a more soft approach and overall "pastel" vibe. Therefore the image took the same route.This, paired with a basic understanding of Controlnet settings can allow you to pick and choose how the image gets interpreted by the AI.For those with local WebUI, you are in luck, since Controlnet offers ALOT more settings, such as pixel perfect, intensity of the canny control image, etc. But for now, this are the basics.Step 2.2.4. Canny for references.Here we show the "Reference" controlnet model who's daddy.For this I will use the following image:Controlnet allows you to use an image as reference, and canny does it best to what I can say. So let's return to Text2Img and learn how it works!Settings are as always, Denoise is no longer here and we load the reference image. To test it out, we will now generate as is:Already solid, but you can see the AI trying to keep the hair in place, while also struggling with the ears, it's a mess! So let's tone it down. First I will lower the strenght to 0.7 and try again:Now this one I like! But the image still feels a little... weird...If you have paid attention, canny has a little thing that makes all resulting images a little too... pixelated.. all over the edges. This is where the control steps come into play finally.Since the image is just a reference, I will test again with the following config; 0 - 0.5 and 0 - 0.7This is with controlnet stopping at 0.5:And 0.7:As you can see, each has some pros and cons. 0.5 has less artifacts, but 0.7 has a better looking face. At the end you will run multiple gens and pick your favorite; finishing with an img2img pass (WITHOUT CONTROLNET) at a low denoise such as 0.2 for example. Or depending on the model you are using.Always test!Step 2.3.1. (For Reference).My least favorite, reference. The idea is that your control image "suggests" the model what you want. So if you generate an image of a girl and your reference image is Ariana Grande, the result should be a girl that looks like Ariana Grande. And of course you sill need to describe it.Let's test it with this:So you know the drill.Prompt:masterpiece, best quality, newest, absurdres, highres, solo, awff, male, wolf, anthro, smile, clothed, safe, black clothing, raincoat, red gloves, holding mask, holding gun, full-length portrait, Persona 5, cosplay, joker \(persona 5\), masterpiece, best quality, newest, absurdres, highresAnd here's the result:Not bad uh? But I rather use cunny.And that's it!Congratulations, now you know the basics of Controlnet. Remember to always test different settings as you trial and error!