Casual hobbyist, not an expert here.
It WAS working... About eight months ago, I trained a bunch of embeddings and hypernetworks and it all worked great.
Cut to the present, I want to do some more training. I've updated Automatic1111 several times, but nothing else about my setup has changed. However, whenever I try to train anything (embeddings, hypernetworks or loras), loss is NaN for 4 out of 5 steps right from the get go. As the training progresses, loss becomes NaN for 9 out of 10 steps, then 19 out of 20 steps around step 3,000, which is as far as I've gotten. Hypernetworks just don't work at that point and embeddings produce garbage.
I have googled like crazy, and found
A few threads, where the best hint is that (at least 8-9 months ago) xformers broke training. Well, I've messed around with xformers, uninstalled and reinstalled xformers, eaten xformers for breakfast. Behavior is the same.
Lower training rate I have set my training rate to 0.0000000000000005. Behavior is identical.
My system is on the low end for VRAM (8G). I have TWO 8G cards, so I wish I could train on both like I can for Llama. But I also think that's not it, because my OLD embeddings and hypernetworks came out great and still work.
Any thoughts here?
Depending on how the dependencies (i.e. xformers) are versioned you can make a new clone of A1111 and checkout a commit from ~8 months ago to see if it works again. Of course I would also recommend trying a fresh install.
Way ahead of you there. I've reinstalled the current version four or five times at this point.
This is a good idea. I've tried two different old versions from old commit hashes so far and both have crashed with other problems. It seems like (lol) both versions of A1111 put their venv in the same place, so the old versions are barfing on some dependencies with version numbers that are too high and they ALSO broke my current version by downgrading some other dependencies (easy fix, just wipe it out and reinstall it again). I'm trying to debug this, because I COULD see a world where I have an old version of A1111 training on one card while the NEW version generates on the other.