Bots visiting the HFU

[ChrisSmolinski]

As some users have noticed, the HFU has recently become popular with lots of Web crawling bots, leading to inflated view counts for many threads.

According to the client info from the web server logs, these bots appear to be from the Mastodon social media site. Unfortunately they come in waves, and sometimes there are dozens, perhaps even hundreds, viewing the same topics at the same time. I assume their intent is not malicious, we're just the victims of sloppy programming.

Also unfortunate, they apparently don't obey robots.txt files. I'm working on a solution to mitigate their damage.

[RobRich]

What is likely happening are link previews. AFAIK, each instance of Mastodon generates its own preview links. If a Mastodon user posts a link to HFU, then each instance federating (sharing) that post will generate a preview link.

You might look into blocking according to HTTP headers if the underlying Mastodon process uses a standardized user agent string to fetch link content for parsing.

[Ray Lalleu]

Any new topic had hundreds of visits in the first few minutes.
That seems solved, but now there are visits by chunks of 20 at a time.

[ChrisSmolinski]

Any new topic had hundreds of visits in the first few minutes.
That seems solved, but now there are visits by chunks of 20 at a time.

We're still going to get visits by regular "good" bots. But yes, things seem better.