I reworked our robots.txt parsing to be much better. In the process, I looked at a bunch of website’s robots.txt files… Blah. No wonder the Internet Archive gave up and now ignores it.

The definition of a “crawler” is increasingly debatable. A search engine? Yes. Bookmarking and archiving? Dunno.