jDownloadsownloads.com


Author Topic: List of Crawler, Searchbots and IP adresses for blocking  (Read 15019 times)

0 Members and 1 Guest are viewing this topic.

Offline jo80

  • Newbie
  • *
  • Posts: 19
Re: List of Crawler, Searchbots and IP adresses for blocking
« Reply #15 on: 06.11.2013 05:13:09 »
Neue Liste, die letzten Wochen waren`s wieder lästig  ;D

[gelöscht durch Administrator]

Offline Arno

  • Administrator
  • ***
  • Posts: 12.325
    • jDownloads.com
Re: List of Crawler, Searchbots and IP adresses for blocking
« Reply #16 on: 06.11.2013 19:09:54 »
Gut das wir bald mit Captcha arbeiten kännen in der 2.5... ;)
Best Regards / Gruß
Arno
Please make a Donation for jDownloads and/or write a review on the Joomla! Extensions directory!

Offline jo80

  • Newbie
  • *
  • Posts: 19
Re: List of Crawler, Searchbots and IP adresses for blocking
« Reply #17 on: 14.12.2013 17:31:43 »
Gut das wir bald mit Captcha arbeiten kännen in der 2.5... ;)

Das wär` natürlich ganz super, kanns kaum erwarten!!


Nachstehend die letzten Ergänzungen zum vorigen Download:

157.55.33.250  msnbot-157-55-33-250.search.msn.com
157.55.35.75   msnbot-157-55-35-75.search.msn.com
157.55.33.114  msnbot-157-55-33-114.search.msn.com
157.56.92.176  msnbot-157-56-92-176.search.msn.com
157.56.93.39   msnbot-157-56-93-39.search.msn.com
157.55.32.61   msnbot-157-55-32-61.search.msn.com
157.55.34.94   msnbot-157-55-34-94.search.msn.com
157.55.35.33   msnbot-157-55-35-33.search.msn.com
157.55.32.222  msnbot-157-55-32-222.search.msn.com
157.55.34.24   msnbot-157-55-34-24.search.msn.com
157.56.93.94   msnbot-157-56-93-94.search.msn.com
157.55.32.39   msnbot-157-55-32-39.search.msn.com


65.55.52.88    msnbot-65-55-52-88.search.msn.com


66.249.76.25   crawl-66-249-76-25.googlebot.com
66.249.78.40   crawl-66-249-78-40.googlebot.com


5.10.83.65   5.10.83.65-static.reverse.softlayer.com
5.10.83.21   5.10.83.21-static.reverse.softlayer.com
5.10.83.41   5.10.83.41-static.reverse.softlayer.com
5.10.83.35   5.10.83.35-static.reverse.softlayer.com
5.10.83.82   5.10.83.82-static.reverse.softlayer.com
5.10.83.15   5.10.83.15-static.reverse.softlayer.com
5.10.83.31   5.10.83.31-static.reverse.softlayer.com
5.10.83.36   5.10.83.36-static.reverse.softlayer.com
5.10.83.19   5.10.83.19-static.reverse.softlayer.com
5.10.83.11   5.10.83.11-static.reverse.softlayer.com
5.10.83.46   5.10.83.46-static.reverse.softlayer.com
5.10.83.23   5.10.83.23-static.reverse.softlayer.com
5.10.83.101  5.10.83.101-static.reverse.softlayer.com

5.10.83.18   ahrefsbot-5-10-83-18.ahrefs.com

Offline treat2day

  • Newbie
  • *
  • Posts: 45
Re: List of Crawler, Searchbots and IP adresses for blocking
« Reply #18 on: 17.02.2014 05:36:49 »
I just looked at logs for the first time in 6 months and was amazed to see the results. I combined my list with the last posted text list, sorted and removed duplicates. I included both text document and excel sheet for everyone's convenience.

Thank you for this post because it confirmed my list too.

[gelöscht durch Administrator]

Offline jo80

  • Newbie
  • *
  • Posts: 19
Re: List of Crawler, Searchbots and IP adresses for blocking
« Reply #19 on: 17.02.2014 06:51:42 »
Hi t2d,

Have just again a few updates in my list, can you implant it in your list, please?  That would be a perfect answer to all bots - serious human work  ;D

have a great day!

[gelöscht durch Administrator]

Offline treat2day

  • Newbie
  • *
  • Posts: 45
Re: List of Crawler, Searchbots and IP adresses for blocking
« Reply #20 on: 17.02.2014 15:07:29 »
Yes, I combined all your posting from the beginning with my list, sorted and removed duplicates.

Is this new posted list different from the previous?

Offline jo80

  • Newbie
  • *
  • Posts: 19
Re: List of Crawler, Searchbots and IP adresses for blocking
« Reply #21 on: 19.02.2014 14:18:26 »
Yes, in the past 2 monthes I have added maybe 20 or 30 new IPs ad have added them just under the old IPs.  I dont know if its worth or not. As you may like.

Offline treat2day

  • Newbie
  • *
  • Posts: 45
Re: List of Crawler, Searchbots and IP adresses for blocking
« Reply #22 on: 24.02.2014 18:40:30 »
I continued to monitor the logs and it seems the bots are blocked but goes around it by using the next number in its NetRange (CIDR Block). This is an endless fight one IP at a time.

I don't want to blacklist the NetRange from by entire site using Akeeba admin tools.  It makes more sense to only block them for unauthorized downloading and keep them crawling the site. On the other hand, I don't want them downloading 15,000 files and never sending one customer to my site to see other content.

Is there a way to add the NetRange to JDownloads by range instead of one IP at time?

Example

You can specify an IP or IP range in the following formats:

    Single IP, i.e. 192.168.1.1
    Simple IP Range, i.e. 192.168.1.1-192.168.1.255
    Implied IP Range, i.e. 192.168.1.
    CIDR Block, i.e. 192.168.1.0/24

Offline jo80

  • Newbie
  • *
  • Posts: 19
Re: List of Crawler, Searchbots and IP adresses for blocking
« Reply #23 on: 12.08.2015 04:05:21 »
Wow, da installiere ich gerade mal wieder ein jDownloads um es einem Kunden schmackhaft zu machen, und finde meine Liste bereits eingebaut :)  Danke vielmals für die Ehre, ich sitze gerade mit einem großen Smiley da :)
Anbei die neue Liste bis gestern (heute warn`s noch nicht lästig)
:)
Johannes

[gelöscht durch Administrator]

Offline Arno

  • Administrator
  • ***
  • Posts: 12.325
    • jDownloads.com
Re: List of Crawler, Searchbots and IP adresses for blocking
« Reply #24 on: 12.08.2015 10:10:19 »
Hi,
du kannst mittlerweile ja auch ReCaptcha einsetzen. Ist vielleicht effektiver, wenn auch nicht jedermanns Sache. ;)
Ab der nächsten Version 3.2.36 (kommt heute oder morgen)  wird auch reCaptcha 2.0 unterstützt.
Best Regards / Gruß
Arno
Please make a Donation for jDownloads and/or write a review on the Joomla! Extensions directory!