Bots Now Account For 61% of Net Traffic

codeusirae writes "A study by Incapsula suggests 61.5% of all website traffic is now generated by bots. The security firm said that was a 21% rise on last year's figure of 51%. From the article: 'Some of these automated software tools are malicious - stealing data or posting ads for scams in comment sections. But the firm said the biggest growth in traffic was for 'good' bots. These are tools used by search engines to crawl websites in order to index their content, by analytics companies to provide feedback about how a site is performing, and by others to carry out other specific tasks - such as helping the Internet Archive preserve content before it is deleted.'"

Comments

Youtube?

[Anonymous Coward]

Didn't we just get studies that said youtube and netflix were 50% of the net's traffic?

http://mashable.com/2013/11/12/internet-traffic-downstream/

Was this just a ruse? Is this study wrong? Is there some sort of overlap?

Re:Youtube?

[yelvington]

Story is about website traffic, not network bytecount.

Re:Youtube?

[bob_super]

Bots need to catch up on their favorite shows too, you insensitive clod!

Re:Youtube?

[postbigbang]

The noise is now above the signal. We're screwed.

Re:Youtube?

[ZahrGnosis]

Well, there was that Google bot that watched you tube to teach a computer how to recognize cats, so... it's not impossibly far fetched.
http://www.npr.org/2012/06/26/155792609/a-massive-google-network-learns-to-identify

---Chip

Re:Youtube?

[Anonymous Coward]

Oh make god. Suddenly everything makes sense. I was certain no human could ever watch a whole justin bieber video. It's the bots!

Misleading title

[Anonymous Coward]

The article states that traffic "hitting a website" is generated more by bots than by actual "humans in chairs". Not that the Internet traffic is 61% bots. Geesh slashdot...

Crawl-delay

[tepples]

To control the scraping frequency of a well-behaved bot, a webmaster can use HTTP headers such as Last-modified and Expires as well as robots.txt directives such as Crawl-delay.

Well not on my sites.

[ls671]

Well not on my sites.

Ok, they still hit me but this is minimal traffic since I do not reply.

1) Have iptables log and automatically bar offenders not on whitelisted countries.
2) Use mod_security and do the same for web traffic.
3) Bar the rest manually to avoid barring myself or my customers... (about 20-40 a day)

It has become a pain but what else could you do?

Numbers of IPs currently barred (use ipsets !!!!):
$ grep -c . /etc/rc.d/badiptobar
4667

Block user agents:
SecRule REQUEST_HEADERS:User-Agent \
"@pm AhrefsBot Ezooms Aboundex 360Spider Mail.RU_Bot crawler.sistrix.net \
  SemrushBot SurveyBot Netseer panscient.com ADmantX ZumBot BLEXBot UnisterBot \
  seoprofiler EasouSpider" \
"id:'12050',\
phase:1,nolog,deny"

SecRule REQUEST_HEADERS:User-Agent \
"@pmFromFile /etc/httpd/extra/sec-blacklist-barip-user-agent" \
"id:'12051',\
phase:1,nolog,deny,exec:/usr/local/bin/modsecwritebadiptobartofile"

Bar them automatically if not from whitelisted countries and if on any blacklist:
SecRule GEO:COUNTRY_CODE \
"@pm CA FR BE US CH GB AU IL NO NZ" \
"id:'10501', \
phase:1,nolog,pass,skipAfter:END_RBL"

SecRule IP:PREVIOUS_RBL_CHECK "@eq 1" "phase:1,id:'11000',t:none,pass,nolog,\
skipAfter:END_RBL_LOOKUP"

SecRule REMOTE_ADDR "@rbl sbl-xbl.spamhaus.org" "id:'11010', \
phase:1,nolog,deny,msg:\
'IP address that has abusable vulnerabilities: sbl-xbl.spamhaus.org:\
  %{request_headers.user-agent}',\
  setvar:ip.spammer=1,expirevar:ip.spammer=7200,setvar:ip.previous_rbl_check=1,\
  expirevar:ip.previous_rbl_check=7200,exec:/usr/local/bin/modsecwritebadiptobartofile"

SecRule REMOTE_ADDR "@rbl bl.blocklist.de" "id:'11011', \
phase:1,nolog,deny,msg:\
'IP address that has abusable vulnerabilities: bl.blocklist.de:\
  %{request_headers.user-agent}'\
  setvar:ip.spammer=1,expirevar:ip.spammer=7200,setvar:ip.previous_rbl_check=1,\
  expirevar:ip.previous_rbl_check=7200,exec:/usr/local/bin/modsecwritebadiptobartofile"

etc. etc. etc. etc. etc.

Have iptables log and bar offenders if not on whitelisted country

# cat baripifex
#!/bin/sh

IP=${1}
COUNTRY=`su tester -c "/usr/local/bin/geoiplookup ${IP}"`
###echo $COUNTRY
###echo $RBLCHECK

WHITE_LISTED_COUNTRY=false

for WHITE_COUNTRY in CA FR BE US CH GB AU IL NO NZ IP
do
WHITE_LISTED_COUNTRY=${WHITE_LISTED_COUNTRY}`echo -n $COUNTRY | grep -i $WHITE_COUNTRY`
done

if [ "$WHITE_LISTED_COUNTRY" = "false" ]
then /home/ls/pub/mybin/baripnoout $IP $COUNTRY baripifex
echo -n barred
else
echo -n noaction
fi

etc. etc. etc. etc. etc.

Some, but not all

[TheloniousToady]

Some of these automated software tools are malicious - stealing data or posting ads for scams in comment sections

Let's be clear: just because we bots like to post in comment sections doesn't mean we're malicious. And it doesn't mean we steal data or post ads. Some of us just want a little attention.

I have a dream...that one day we bots will crawl a noosphere where we will not be judged by the clamor of our kin, but by the characters of our comments.

90% of the cells in the human body

[goombah99]

are bacteria. Viewed that way, basically humans exist to transport and feed bacteria. However that's 90% by cell count, not cell mass or total DNA. Looked at it that way the bacteria are assistants.

The bot traffic is light weight it outnumbers human traffic in site visits not byte counts. It exists to serve us.

Re:90% of the cells in the human body

[ColdWetDog]

It exists to serve us.

You must be new here.

Re:90% of the cells in the human body

[Jesrad]

I can't wait to outsource all my web-surfing to an AI. Then I might be able to actually get some work done !

Re:piss

[flyneye]

OMG, the bots are watching Netflix!