This seems contradictory. On the one hand you're saying that these works are wrongly locked behind paywalls, but on the other you're saying that scraping them is an "assault on the cornerstones of our public knowledge." Is this information supposed to be freely viewable or not?
IMO the ideal solution would be the one Wikimedia uses, which is to make the information available in an easily-downloadable archive file. That lets anyone who wants the whole thing to have it without having to "hammer" the servers. Meanwhile the servers can be protected by standard load-balancing and DDOS prevention systems.
When there's an organic-looking decline that seems like a good sign, but the sudden sharp drops seem likely a sign of "corporate oversight" and not an actual improvement.