Wedge

Public area => The Pub => Off-topic => Topic started by: Norodo on September 8th, 2011, 04:22 AM

Title: Baidu bloody loves me.
Post by: Norodo on September 8th, 2011, 04:22 AM
What is this... It does this all the time. Are you toying with my head, or is Baidu very Norodo-happy?

(http://i.imgur.com/kRw5Ll.png)(http://imgur.com/kRw5L)
Title: Re: Baidu bloody loves me.
Post by: billy2 on September 8th, 2011, 07:56 AM
its a very active bot.

Baidu (180.76.5.183)    06:48:39    Viewing the album "St Abbs Oct 16 - 17th 2010" in the gallery
Baidu (180.76.5.94)    06:42:21    Viewing the album "St Abbs Oct 16 - 17th 2010" in the gallery

Title: Re: Baidu bloody loves me.
Post by: Arantor on September 8th, 2011, 08:52 AM
Baidu is a horrifically vicious bot. Where Google might send two or three requests at a time, I've seen Baidu send 20.

Interestingly, because of default Bad Behaviour checks, Baidu is actually currently blocked, and I'm honestly not sure whether I should change that or not - and I'm more inclined not to change it for the most part.
Title: Re: Baidu bloody loves me.
Post by: Nao on September 8th, 2011, 08:54 AM
It's because Wedge will not be released in English. Its language strings are from now on hardcoded to Mandarin. Did we forget to tell you? China is the biggest market now.
Oh and because ie6 is the leading browser in China, I've dropped compatibility with other browsers. Standards are too annoying to follow.
Title: Re: Baidu bloody loves me.
Post by: Arantor on September 8th, 2011, 08:56 AM
/mequietly chokes on first cup of tea of the day :lol:
Title: Re: Baidu bloody loves me.
Post by: CJ Jackson on September 8th, 2011, 01:47 PM
We are all being stalked by Baidu, RUN!!!!!
Title: Re: Baidu bloody loves me.
Post by: Arantor on September 8th, 2011, 02:00 PM
I'm not being stalked by Baidu on at least one site of mine ;)
Title: Re: Baidu bloody loves me.
Post by: Dragooon on September 8th, 2011, 03:44 PM
Hmm yeah...Baidu keeps stalking me as well. I might as well blow up their asses.
Title: Re: Baidu bloody loves me.
Post by: Arantor on September 8th, 2011, 03:48 PM
Protip: examine the HTTP headers.

If there's no Accept header, chances are it's a dodgy request - or it's Baidu.
Title: Re: Baidu bloody loves me.
Post by: Norodo on September 9th, 2011, 03:14 AM
Baidu    3:00   Viewing Norodo's profile.
Baidu    3:05   Viewing Norodo's profile.
Baidu    3:10   Viewing Norodo's profile.

Ok Baidu. I think you've seen all there is to see by now.

(click to show/hide)
Why must you stalk me? It keeps happening.
Title: Re: Baidu bloody loves me.
Post by: Arantor on September 9th, 2011, 03:19 AM
Because Baidu's a shitty spider.
Title: Re: Baidu bloody loves me.
Post by: Norodo on September 9th, 2011, 03:47 AM
But...

(http://i.imgur.com/S6UPy.png)

Why me?
Title: Re: Baidu bloody loves me.
Post by: Nao on September 9th, 2011, 07:44 AM
Do you want me to add a robots.txt for that page to push baidu away? I can even change the page for it, to post a message to them :P
Title: Re: Baidu bloody loves me.
Post by: billy2 on September 9th, 2011, 08:46 AM
Quote from Nao on September 9th, 2011, 07:44 AM
Do you want me to add a robots.txt for that page to push baidu away? I can even change the page for it, to post a message to them :P
Baidu is so persistent you will probably draw 30 Baidu bots to the page  :lol:
Title: Re: Baidu bloody loves me.
Post by: Arantor on September 9th, 2011, 08:52 AM
Yeah, they don't really honour robots.txt.
Title: Re: Baidu bloody loves me.
Post by: Pandos on September 9th, 2011, 11:00 AM
Why not stopping Baidu by iptables?

 iptables -I INPUT -s  119.63.192.0/24  -j DROP
Title: Re: Baidu bloody loves me.
Post by: Arantor on September 9th, 2011, 11:01 AM
Because not everyone has control of iptables (though I think Nao has access to it here)

But you can definitely exclude them by examining the HTTP request.
Title: Re: Baidu bloody loves me.
Post by: Nao on September 9th, 2011, 01:54 PM
I may have access to iptables, but... Why exactly would I stop Baidu? As long as it's not killing the bandwidth, I could care less about it...

If you don't like seeing it in the who's online area, I can remove it from the list of recognized bots... :P
Title: Re: Baidu bloody loves me.
Post by: Arantor on September 9th, 2011, 02:07 PM
That's the thing: it consumes 8-10x more bandwidth than any other search engine, and it's going to keep going.
Title: Re: Baidu bloody loves me.
Post by: Norodo on September 9th, 2011, 02:18 PM
That and for people outside China it's just not a bot that is needed. We have -much- better search engines.
Title: Re: Baidu bloody loves me.
Post by: Arantor on September 14th, 2011, 02:19 PM
In other news, now it's viewing my profile 3 times over.
Title: Re: Baidu bloody loves me.
Post by: Dismal Shadow on September 14th, 2011, 02:59 PM
Quote from Arantor on September 14th, 2011, 02:19 PM
In other news, now it's viewing my profile 3 times over.
I say ban they. :p
Title: Re: Baidu bloody loves me.
Post by: PantsManUK on September 15th, 2011, 02:23 PM
Maybe they're getting better; I told Baidu to bugger off in my robots.txt and (so far) they've not returned (finger crossed, touch wood...)
Title: Re: Baidu bloody loves me.
Post by: Arantor on September 15th, 2011, 02:25 PM
Quote from PantsManUK on September 15th, 2011, 02:23 PM
Maybe they're getting better; I told Baidu to bugger off in my robots.txt and (so far) they've not returned (finger crossed, touch wood...)
Wait, Baidu LISTENED TO YOU? Colour me surprised.
Title: Re: Baidu bloody loves me.
Post by: busterone on September 16th, 2011, 04:26 AM
They have never honored my robots.txt.  I was sometimes getting 40 to 50 of them at a time. The damn little roaches were everywhere at once.  I don't have access to iptables, so I just banned them in .htaccess about 6 months or so ago.
Title: Re: Baidu bloody loves me.
Post by: Nao on September 16th, 2011, 07:45 AM
Maybe it requires a specific format in the robots.txt...? Like, Baidu has to be written in hanzi? :P