Got a warning for my blog going over 100GB in bandwidth this month… which sounded incredibly unusual. My blog is text and a couple images and I haven’t posted anything to it in ages… like how would that even be possible?

Turns out it’s possible when you have crawlers going apeshit on your server. Am I even reading this right? 12,181 with 181 zeros at the end for ‘Unknown robot’? This is actually bonkers.

Edit: As Thunraz points out below, there’s a footnote that reads “Numbers after + are successful hits on ‘robots.txt’ files” and not scientific notation.

Edit 2: After doing more digging, the culprit is a post where I shared a few wallpapers for download. The bots have been downloading these wallpapers over and over, using 100GB of bandwidth usage in the first 12 days of November. That’s when my account was suspended for exceeding bandwidth (it’s an artificial limit I put on there awhile back and forgot about…) that’s also why the ‘last visit’ for all the bots is November 12th.

  • slazer2au@lemmy.worldEnglish
    68·
    3 months ago

    AI scrapers are the new internet DDoS.

    Might want to throw something Infront of your blog to ward them off like Anubis or a Tarpit.

  • dual_sport_dork 🐧🗡️@lemmy.worldEnglish
    52·
    3 months ago

    I run an ecommerce site and lately they’ve latched onto one very specific product with attempts to hammer its page and any of those branching from it for no readily identifiable reason, at the rate of several hundred times every second. I found out pretty quickly, because suddenly our view stats for that page in particular rocketed into the millions.

    I had to insert a little script to IP ban these fuckers, which kicks in if I see a malformed user agent string or if you try to hit this page specifically more than 100 times. Through this I discovered that the requests are coming from hundreds of thousands of individual random IP addresses, many of which are located in Singapore, Brazil, and India, and mostly resolve down into those owned by local ISPs and cell phone carriers.

    Of course they ignore your robots.txt as well. This smells like some kind of botnet thing to me.

    • panda_abyss@lemmy.caEnglish
      18·
      3 months ago

      I don’t really get those bots.

      Like, there are bots that are trying to scrape product info, or prices, or scan for quantity fields. But why the hell do some of these bots behave the way they do?

      Do you use Shopify by chance? With Shopify the bots could be scraping the product.json endpoint unless it’s disabled in your theme. Shopify just seems to show the updated at timestamp from the db in their headers+product data, so inventory quantity changes actually result in a timestamp change that can be used to estimate your sales.

      There are companies that do that and sell sales numbers to competitors.

      No idea why they have inventory info on their products table, it’s probably a performance optimization.

      I haven’t really done much scraping work in a while, not since before these new stupid scrapers started proliferating.

      • dual_sport_dork 🐧🗡️@lemmy.worldEnglish
        20·
        3 months ago

        Negative. Our solution is completely home grown. All artisinal-like, from scratch. I can’t imagine I reveal anything anyone would care about much except product specs, and our inventory and pricing really doesn’t change very frequently.

        Even so, you think someone bothering to run a botnet to hound our site would distribute page loads across all of our products, right? Not just one. It’s nonsensical.

        • panda_abyss@lemmy.caEnglish
          9·
          3 months ago

          Yeah, that’s the kind of weird shit I don’t understand. Someone on the other hand is paying for servers and a residential proxy to send that traffic too. Why?

        • Lka1988@lemmy.dbzer0.comEnglish
          2·
          3 months ago

          Could it be a competitor for that particular product? Hired some foreign entity to hit anything related to their own product?

      • porcoesphino@mander.xyzEnglish
        2·
        2 months ago

        Have you ever tried writing a scrapper. I have for offline reference material. You’ll make a mistake like that a few times and know but there are sure to be other times you don’t notice. I usually only want a relatively small site (say a Khan Academy lesson which doesn’t save text offline, just videos) and put in a large delay between requests but I’ll still come back after thinking I have it down and it’s thrashed something

  • WolfLink@sh.itjust.worksEnglish
    71·
    3 months ago

    This is why I use CloudFlare. They block the worst and cache for me to reduce the load of the rest. It’s not 100% but it does help.

    • irmadlad@lemmy.worldEnglish
      3·
      3 months ago

      LOL Someone took exception to your use of Cloudflare. Hilarious. Anyways, yeah, what Cloudflare doesn’t get, pFsense does.

  • ohshit604@sh.itjust.worksEnglish
    4·
    3 months ago

    I just geo-restrict my server to my country, certain services I’ll run an ip-blacklist and only whitelist the known few networks.

    Works okay I suppose, kills the need for a WAF, haven’t had any issues with it.

    • benagain@lemmy.mlOPEnglish
      3·
      3 months ago

      It’s a mix, I put two screenshots together. On the left is my monthly bandwidth usage from CPanel on the right is Awstats (though I hid some sections so the Robots/Spiders section was closer to the top).

        • benagain@lemmy.mlOPEnglish
          1·
          2 months ago

          I think they’re winding down the project unfortunately, so I might have to get with the times…

          • [object Object]@lemmy.worldEnglish
            1·
            2 months ago

            I mean, I thought it was long dead. It’s twenty-five years old, and the web has changed quite a bit in that time. No one uses Perl anymore, for starters. I used Open Web Analytics, Webalizer, or somesuch by 2008 or so. I remember Webalizer being snappy as heck.

            I tinkered with log analysis myself back then, peeping into the source of AWStats and others. Learned that a humongous regexp with like two hundred alternative matches for the user-agent string was way faster than trying to match them individually — which of course makes sense seeing as regexps work as state-machines in a sort of a very specialized VM. My first attempts, in comparison, were laughably naive and slow. Ah, what a time.

            Sure enough, working on a high-traffic site taught me that it’s way more efficient to prepare data for reading at the moment of change instead of when it’s being read — which translates to analyzing visits on the fly and writing to an optimized database like ElasticSearch.

  • hdsrob@lemmy.worldEnglish
    3·
    2 months ago

    Had the same thing happen on one of my servers. Got up one day a few weeks ago and the server was suspended (luckily the hosting provider unsuspended it for me quickly).

    It’s mostly business sites, but we do have an old personal blog on there with a lot of travel pictures on it, and 4 or 5 AI bots were just pounding it. Went from 300GB per month average to 5TB on August, and 10/11 TB in September and October.

  • Omega_Jimes@lemmy.caEnglish
    2·
    2 months ago

    It’s a shame we don’t have those banner ad schemes anymore. Cybersquatting could be a viable income stream if you could convince the cleaners to click banner ads for a faction of a penny each.