[Coco] Someones bots are abusing my downloads

[didier at aida.org]

beware that a robot.txt malformed or too big is ignored

I discovered that by experience....

[I had a several robots.txt one day I discovered they were not working 
anymore, I had to rewrite them]

there is a web site somewhere to check the validity of robots.txt

I found this one: https://fr.ryte.com/free-tools/robots-txt/

I don't remember which one I used



On 01/11/2019 18:53, Bill Gunshannon wrote:
> On 11/1/19 1:49 PM, Ken Flanagan wrote:
>> On Fri., Nov. 1, 2019, 10:35 a.m. Gene Heskett, <gheskett at shentel.net>
>> wrote:
>>
>>> They ae not just indexing my place, they were sucking the whole thing,
>>> one file at a time and keeping my 250 kilobaud uplink bandwidth full for
>>> the last 4 or so hours.  That a DDOS AFAIAC. Kill them on sight.
>>> Here is typical log entry:
>>>
>>> coyote.coyote.den:80 46.229.168.141 - -
>>> [01/Nov/2019:13:33:16 -0400] "GET
>>> /gene/nitros9/3rdparty/packages/uucpbb/doc/uucp.man
>>> HTTP/1.1" 200 3142 "-" "Mozilla/5.0 (compatible; SemrushBot/6~bl;
>>> +http://www.semrush.com/bot.html)"
>>>
>> Do you have a robots.txt file on your server? If you do and they're
>> ignoring it, I would suggest contacting the respective companies.
> I doubt they will care.
>
> A better bet is to block them at the firewall so they can't
> even connect.
>
> bill
>