@samgrover I agree intention matters, but I don't think it's clear that one is exploitative, or at least not any more than Google crawling is also exploitative. Personally, I think any crawling including the Internet Archive should probably respect robots.txt.