I can save you a lot of trouble, actually. You don’t need all of this!
Just make a custom 404 page that returns 13 MBs of junk along with status code 200 and has a few dead links (404, so it just goes to itself)
There are no bots on the domain I do this on anymore. From swarming to zero in under a week.
You don’t need tar pits or heuristics or anything else fancy. Just make your website so expensive to crawl that it’s not worth it so they filter themselves.
I don’t know a lot about this, but I would guess a normal user would like a message, that says something along the lines of “404, couldn’t find what you were looking for.” The status code and the links back to itself as well as the 13 MBs of noise should probably not irritate them. Hidden links should also not irritate normal users.
I also “don’t know a lot about this”, but I do know that your browser receiving a 200 means that everything worked properly. From what I can tell, this technique is replaces any and every 404 response with 200, thus tricking the browser (and therefore the user) into thinking the site is working as expected every time they run into a missing webpage on this site.
I can save you a lot of trouble, actually. You don’t need all of this!
Just make a custom 404 page that returns 13 MBs of junk along with status code 200 and has a few dead links (404, so it just goes to itself)
There are no bots on the domain I do this on anymore. From swarming to zero in under a week.
You don’t need tar pits or heuristics or anything else fancy. Just make your website so expensive to crawl that it’s not worth it so they filter themselves.
How would you go about doing this part? Asking for a friend who’s an idiot, totally not for me.
I use Apache2 and PHP, here’s what I did:
in .htaccess you can set
ErrorDocument 404 /error-hole.phphttps://httpd.apache.org/docs/2.4/custom-error.htmlin error-hole.php,
<?php http_response_code(200); ?> <p>*paste a string that is 13 megabytes long*</p>For the string, I used
ddto generate 13 MBs of noise from/dev/urandomand then I converted that to base64 so it would paste into error-hole.phpYou should probably hide some invisible dead links around your website as honeypots for the bots that normal users can’t see.
How does this affect a genuine user who experiences a 404 on your site?
I don’t know a lot about this, but I would guess a normal user would like a message, that says something along the lines of “404, couldn’t find what you were looking for.” The status code and the links back to itself as well as the 13 MBs of noise should probably not irritate them. Hidden links should also not irritate normal users.
I also “don’t know a lot about this”, but I do know that your browser receiving a 200 means that everything worked properly. From what I can tell, this technique is replaces any and every 404 response with 200, thus tricking the browser (and therefore the user) into thinking the site is working as expected every time they run into a missing webpage on this site.
The user doesn’t see the status code, they see what’s rendered to the screen.