Fuck AI

I'm not sure that the comparison with the weather data works. Tweaking curves to more closely match the test data, and moving around a model's probability space in the hope that it sufficiently increases the probability of outputting tokens that fixes the code's problems, seem different enough to me that I don't know whether the former working well says anything about how well the latter works.

If I understand what you're describing correctly, the two models aren't improving each other, like in adversarial learning, but the adversarial model is trying to get the generative model to zone in on output that produces the user's desired behaviour based on the given test data. But that can only work as well as how much the adversarial model can be relied upon to actually perform the tasks needed to make this happen. So I think my point still stands that the objectivity of your measurements of the test results is only meaningful if the test results themselves are meaningful, which is not guaranteed given what's doing the testing.

How complex is the adversarial model? If it's anywhere near the generative model, I don't think you can have actual meaningful numbers about its reliability that allow you to reason about how meaningful the test results it produces are.