Abusing the System

2005-08-18 18:39:36

#1

chuyskywalker: Admin; +2,439|7095|"Frisco"

This evening I was targeted by a data scraping program. A data scraping program is basically running a bot that relentlessly pulls data from my server at breakneck speeds. This pushed my server to far surpass it's normal 10-20 MySQL queries per second hitting a top of 150 Mysql queries per second.

While a part of me is very proud that the server and site handled 10 times the traffic without batting an eye, I am extremely upset by the behaviour of this person. From here on out I will be actively monitoring the server for people running data scraping programs and I will ban IP addresses if needed.

I put a lot of time and effort into running this site and pay for it out of pocket (with the help of ads) and I do not appreciate, and will not allow, people to ruthlessly steal my content.

2005-08-18 18:51:55

#2

TriggerHappy998: just nothing; +387|7094|-

Busted!

2005-08-18 19:14:40

#3

Reign_Of_Chaos: Member; +0|7074

did a trace on that IP and found this

OrgName: Google Inc.
OrgID: GOGL
Address: 1600 Amphitheatre Parkway
City: Mountain View
StateProv: CA
PostalCode: 94043
Country: US

NetRange: 66.249.64.0 - 66.249.95.255
CIDR: 66.249.64.0/19
NetName: GOOGLE
NetHandle: NET-66-249-64-0-1
Parent: NET-66-0-0-0-0
NetType: Direct Allocation
NameServer: NS1.GOOGLE.COM
NameServer: NS2.GOOGLE.COM
Comment:
RegDate: 2004-03-05
Updated: 2004-11-10

OrgTechHandle: ZG39-ARIN
OrgTechName: Google Inc.
OrgTechPhone: +1-650-318-0200
OrgTechEmail: [email protected]

ARIN WHOIS database, last updated 2005-08-18 19:10
Enter ? for additional hints on searching ARIN's WHOIS database.

rofl

2005-08-18 19:14:54

#4

NM156: The H4xor Mod; +161|7094|North Texas

LOL.... It's googlebot.com! Disallow robots...

Code:

C:\ping -a 66.249.65.67

Pinging crawl-66-249-65-67.googlebot.com [66.249.65.67] with 32 bytes of data:

Reply from 66.249.65.67: bytes=32 time=45ms TTL=237
Reply from 66.249.65.67: bytes=32 time=43ms TTL=237
Reply from 66.249.65.67: bytes=32 time=49ms TTL=237
Reply from 66.249.65.67: bytes=32 time=52ms TTL=237

Ping statistics for 66.249.65.67:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 43ms, Maximum = 52ms, Average = 47ms

2005-08-18 19:16:18

#5

chuyskywalker: Admin; +2,439|7095|"Frisco"

Holy crap, google beat the shit outta my site. Turns out it wasn't the only one. After banning that bot for a minute, another IP showed up. Banned that one too and unbanned google -- seems to tamed the gbot down. Crazy.

2005-08-18 19:36:13

#6

Kushiel: Human Shield; +1|7076|Boynton Beach, FL

chuyskywalker wrote:
Holy crap, google beat the shit outta my site. Turns out it wasn't the only one. After banning that bot for a minute, another IP showed up. Banned that one too and unbanned google -- seems to tamed the gbot down. Crazy.

What can you say Chuy... you're popular.. lol Damn.. now if only in high school j/k

2005-08-18 20:11:12

#7

Krauser98: Extra Green Please!; +53|7077|USA! USA! USA!

I think Google is going to try to take over the world, but in a useful and helpful way.

2005-08-18 22:06:48

#8

priznat: Member; +0|7094

I seem to recall you can contact google and tell them to NOT do that to your site, they are pretty good about that kind of thing, apparently..

At least you are well indexed now thought

2005-08-18 23:15:30

#9

lowyaukee: Member; +2|7081

Frankly, I don't understand the stuffs above. I just want to say that I love your works very much, this site, the stats, the forum.

Wish you all the best!

2005-08-18 23:51:38

#10

midgetspy: Member; +3|7075

To prevent robots from spidering certain parts (or all) of your site, include a robots.txt in the root:

http://www.robotstxt.org/wc/norobots.html

Nic

2005-08-19 00:06:50

#11

B.Schuss: I'm back, baby... ( sort of ); +664|7088|Cologne, Germany

I am not experienced in this field of tech stuff. so maybe someone could explain to me why google would want to do that to other sites ?

Are they interested in your user database ?

Last edited by B.Schuss (2005-08-19 00:07:47)

2005-08-19 00:13:59

#12

priznat: Member; +0|7094

Google bots "crawl" over the internet, indexing sites and taking a cached version, like a "snapshot"..

This is part of the way search engines like google can access search results so quick, it has local versions of a lot of sites filed away on its own server.. Just updates every once in a while with a new freshly scraped version..

As is my understanding, anyway..

2005-08-19 00:23:08

#13

B.Schuss: I'm back, baby... ( sort of ); +664|7088|Cologne, Germany

I see. thx for that.

btw, is sich behaviour legal ?

2005-08-19 00:29:36

#14

chuyskywalker: Admin; +2,439|7095|"Frisco"

No no, sorry. Google contributed, but there was another culprite at fault really (who I have indentified and contacted regarding this incident).

Also, google has to spider my pages in order to target the ads to the content, so it makes sense.

2005-08-19 06:23:50

#15

xbw_shane: Member; +-1|7094|Australia

yer i have Google's bots crawl through my own forum from time to time. i would love to see how much bandwidth they pull from my own server or how it affects my statistics. i dont have google ads but submitted my URL to it a while back.

of course i could stop them though if i really wanted as someone else mentioned.

2005-08-19 07:28:23

#16

CaffeineJunkie: Member; +0|7084|CT, USA

@Skywalker, just add a robots.txt that tells google not to index your sites past /

Or do they have to? They might even google index all 100,000 people.. Wow.. that would hurt.

2005-08-19 07:34:25

#17

NiMhurchu: Suicide Operations; +13|7094|Germany

I think Jeff's way of coding the URL might add to the problem, too.
A searchbot like Google tries to catch every page in every sub-sub-subfolder -- and Jeff's URL is made up like that: http://www.bf2s.com/player/Name1/vs/Name2 for example.
No offense, though, maybe there is a way with the robots.txt. Maybe it can be coded dynamically, so that if a subfolder is loaded, a robots.txt is handed out to the bot dynamically.

2005-08-19 07:57:07

#18

chuyskywalker: Admin; +2,439|7095|"Frisco"

NiMhurchu wrote:
I think Jeff's way of coding the URL might add to the problem, too.
A searchbot like Google tries to catch every page in every sub-sub-subfolder -- and Jeff's URL is made up like that: http://www.bf2s.com/player/Name1/vs/Name2 for example.

Unless linked directly to that page, google wouldn't find it seeing as the only way to get to the comparision pages is via a POST form submission. However, player pages to get spidered. Not too big a deal, just has to be that way so it can make ads work.

2005-08-19 09:09:15

#19

CaffeineJunkie: Member; +0|7084|CT, USA

http://www.google.com/search?hl=en& … s.com+bf2s

Think about it, google takes the first page, finds all these top players.. moves on to their page.. each one of those pages links to more players (most killed, etc) and so on and so forth... Google indexed many a pages.

2005-08-19 10:01:15

#20

haligan: Member; +-1|7073

CaffeineJunkie wrote:
http://www.google.com/search?hl=en&hs=87n&lr=&safe=off&client=firefox-a&rls=org.mozilla:en-US:official&q=+site:bf2s.com+bf2s

Think about it, google takes the first page, finds all these top players.. moves on to their page.. each one of those pages links to more players (most killed, etc) and so on and so forth... Google indexed many a pages.

You are correct, depending on how or if caching is done on this site, that would add to the performance hit on the MySQL db.

Since this is my first post here I'd like to say thanks for the stats, but I wouldn't call what you have queried your content. At most it is EAs, but don't take that as being ungrateful for what you have setup.

2005-08-19 12:44:20

#21

Sarum: The Angry Geek; +11|7094

What you've all missed is that Chuy has said, while Googlebot contributed, it wasn't the prime cause. That was someone else, who he's contacted, but sensibly identified to us since we're like a pack of rabid dogs. So less talk about Google, it wasn't the cause.

2005-08-19 13:52:50

#22

blue60007: Member; +0|7091

Heh, well this happened to a server I admin (well sorta co-admin), we noticed a spike in server usage, and found the culprit to be google bots indexing our forums, went away after a day or two.

2005-08-23 14:10:08

#23

Preacher: In Hoc Signo Vinces; +1|7082|Netherlands, the

I put a lot of time and effort into running this site and pay for it out of pocket (with the help of ads) ....

Time for a Paypal button??? If it gets people a better update..I bet there will be some $ comming your way

2005-08-23 22:53:55

#24

chuyskywalker: Admin; +2,439|7095|"Frisco"

Paypal or not, I can't update any faster than IGN will withstand.

2005-08-23 23:25:23

#25

B.Schuss: I'm back, baby... ( sort of ); +664|7088|Cologne, Germany

well, then it's probably time for IGN to get a paypal button..

BF2S Forums Abusing the System

Code:

chuyskywalker wrote:

NiMhurchu wrote:

CaffeineJunkie wrote:

Board footer