Illustrious is an incredible base model.
But it was never meant to be realistic in any way. This irks me a lot as it contains what I consider to be the best knowledge among every SDXL fine tunes that exist out there. So, I set out to create something that would steer this model towards realism, while not lobotomizing the wonderful stuff that the model inherently knows.
I have seen several realistic models based on Illustrious being released - but none of them manage to keep the inherent understanding of the "Illustrious" part in them.
After many months of tweaking and experimenting, I seem to have created one of the best realistic Illustrious models in my subjective opinion.
This model took several months and over 600 hours of trying out novel and convoluted merging methods to make. This model now has 99% of the knowledge of Illustrious/NoobAI models, while being inherently realistic. The way it converts danbooru concepts to realism is kinda astonishing to me. NSFW, Backgrounds, Portraits, Furry, Complex Poses - This model does it ALL (Do consider the limits of SDXL based models though).
With the newly released Illustrious versions merged in this, this model has become even more competent while still keeping support for 0.1/Noob LoRAs. This one has everything in it - the top SDXL realistic models, the best Pony models, and the best Illustrious and Noob models included in the merge. It also works perfectly with some Pony and SDXL LoRAs (Because of the deep merging of clips along with the UNET, not sure about all Pony LoRAs though).
The sampler and scheduler compatibility is unprecedented. I personally use DPM++ SDE with the Beta Scheduler. Recommended CFG is 4-5. Recommended steps are 60 - although you can get by with less steps too. Hi-res fix will give you incredible images as always - with the denoise set to 0.31 and the upscaling model as 4x-NMKD-Superscale.
Recommended Negative Prompt:
worst quality, low quality, normal quality, anime, 2d, illustration, sketch, comic, flat art, cartoon, toon, bad anatomy, bad hands, signature, watermarks, ugly, error, extra limb, missing limbs, missing fingers
Start with Euler A, Normal and then try out everything that catches your fancy. Lower CFGs are recommended for more realism, but higher values work just as well and maybe provide more contrast to your images.
It's also sad how some great models don't allow sharing merges of them. I made absolutely sure to only use models that allow sharing merges, openly disclosing them in the bottom of this description.
Overall, this was a huge learning experience because of the complex merging methods like DARE-TIES (previously used for my Quintessence PONY model), while focusing on specific transformer attention blocks. Note that I don't have the exact merge recipe as I work with a meager 256 GB SSD, and have to delete 100s of the intermediate models that I have made before the final result. Also a lot of luck was involved in the final result.
Keeping the concepts intact while still making the model realistic is a very complicated task, but the final result is absolutely worth it. Do try it out. Also, the elusive ugly nose problem has been fixed.
Also, for (more realism?) you can try the other great realistic Illustrious models, but I can guarantee the fact that complex gens are not possible with those because of the diluted base Illustrious knowledge.