Exactly, it'd be bad PR. I've argued this before in other threads, the publishers don't want to destroy IA. They just want IA to not flagrantly interfere with their business. They only sued IA when IA poked them too hard for them to ignore.
You may note that the settlement agreement they reached with IA lets IA continue to host books that the publishers haven't released as ebooks themselves, for example. Even now they're not being as harsh as they could be.
So why aren't they? If libraries are doing exactly what IA is doing, why not sue them too? The judge issued a summary judgement in their favor so it's pretty open-and-shut, isn't it?
It's because the libraries know where the line is and they're careful not to cross it. IA jumped merrily across the line and shouted about it from the rooftops.