Pages: 1 2 [3] 4
[BOINCstats] Willy
 
Forum moderator - Administrator - Developer - Tester - Translator
BAM!ID: 1
Joined: 2006-01-09
Posts: 9419
Credits: 350,105,499
World-rank: 4,518

2007-05-01 16:20:08

Blocked 85.255.118.116 for requesting 9030 pages in under 5 minutes. That's 30 pages a second.

Come on, everybody will agree THAT is not normal.
Please do not PM, IM or email me for support (they will go unread/ignored). Use the forum for support.
Lee Carre
 
BAM!ID: 41
Joined: 2006-04-19
Posts: 262
Credits: 299,581
World-rank: 394,907

2007-05-09 09:14:04

Blocked 85.255.118.116 for requesting 9030 pages in under 5 minutes. That's 30 pages a second.

Come on, everybody will agree THAT is not normal.

*splutters and spills coffee*
30 a second?! where do these people get the bandwidth?
Want to search the BOINC Wiki, BOINCstats, or various BOINC forums from within firefox? Try the BOINC related Firefox Search Plugins
[BOINCstats] Willy
 
Forum moderator - Administrator - Developer - Tester - Translator
BAM!ID: 1
Joined: 2006-01-09
Posts: 9419
Credits: 350,105,499
World-rank: 4,518

2007-05-10 16:15:06

Blocked 85.182.33.32, hundreds of request in a row. This one even managed to get to load up to 8+.
Please do not PM, IM or email me for support (they will go unread/ignored). Use the forum for support.
Lee Carre
 
BAM!ID: 41
Joined: 2006-04-19
Posts: 262
Credits: 299,581
World-rank: 394,907

2007-05-10 21:42:50
last modified: 2007-05-10 21:46:21

Blocked 85.182.33.32, hundreds of request in a row. This one even managed to get to load up to 8+.

you think they'd apply some thought and be a little more discrete, rather than making a lot of noise that you're bound to notice lol
fools
Want to search the BOINC Wiki, BOINCstats, or various BOINC forums from within firefox? Try the BOINC related Firefox Search Plugins
bryan
BAM!ID: 3339
Joined: 2006-07-21
Posts: 15
Credits: 0
World-rank: 0

2007-06-02 04:31:41

Willy,

Have you had any scraping from 194.97.107.175? Over the last 10 days, the IP has pulled about 500,000 pages from our server over at RieselSieve (yeah, I just noticed today). It seems to be the same 40 pages or so in 1-minute intervals. Thought I'd check, because I can't figure out the interest in userw.php.
[BOINCstats] Willy
 
Forum moderator - Administrator - Developer - Tester - Translator
BAM!ID: 1
Joined: 2006-01-09
Posts: 9419
Credits: 350,105,499
World-rank: 4,518

2007-06-02 07:23:17

Willy,

Have you had any scraping from 194.97.107.175? Over the last 10 days, the IP has pulled about 500,000 pages from our server over at RieselSieve (yeah, I just noticed today). It seems to be the same 40 pages or so in 1-minute intervals. Thought I'd check, because I can't figure out the interest in userw.php.


Not a single hit on BS.
Please do not PM, IM or email me for support (they will go unread/ignored). Use the forum for support.
PovAddict
BAM!ID: 115
Joined: 2006-05-10
Posts: 1013
Credits: 5,785,239
World-rank: 76,273

2007-06-08 17:33:59

I used to get dozens of hits on userw.php from different IP addresses. The user-agent string was empty or at least surely not corresponding to a mobile phone. They're probably using it because the output is smaller than show_user.php, so it's good for scraping stats. I modified userw.php to give a message instead of returning the stats.
[BOINCstats] Willy
 
Forum moderator - Administrator - Developer - Tester - Translator
BAM!ID: 1
Joined: 2006-01-09
Posts: 9419
Credits: 350,105,499
World-rank: 4,518

2007-07-13 19:22:38

Blocked IP 216.32.90.58. This IP can't even get the index anymore. I never seen a scraper capable of getting even the old server as high as 120 load.

<rant mode>
You sir, are an idiot (of course you need another IP to be able to read this).

And no, switching browsers doesn't get you past the captcha. I wait, you didn't see that, because you kept requesting pages even though all you got was the captcha.
</rant mode>
Please do not PM, IM or email me for support (they will go unread/ignored). Use the forum for support.
PovAddict
BAM!ID: 115
Joined: 2006-05-10
Posts: 1013
Credits: 5,785,239
World-rank: 76,273

2009-01-06 19:40:47

[17:13:51] <Soul_keeper> http://sterlingdesktops.com/boincrace/test.png chart now auto updates daily at 10:10am PST
[17:32:16] <PovAddict> Soul_keeper: you aren't HTML-scraping boincstats.com are you?
[17:32:23] <Soul_keeper> PovAddict, yeah
[17:35:07] <PovAddict> what makes you think you're allowed to do that
[17:35:13] <PovAddict> http://boincstats.com/forum/forum_thread.php?id=1161
[17:35:50] <Soul_keeper> if they didn't want me accessing their website then it shouldn't be on the internet
[17:36:23] <PovAddict> that's like saying "if you didn't want me to steal that image you shouldn't have put it on the internet", just because it's on the internet doesn't mean anyone can copy it
Not running BOINC anymore for several reasons...
royanee
BOINCstats SOFA member
BAM!ID: 61534
Joined: 2008-11-14
Posts: 29
Credits: 458,076,447
World-rank: 3,721

2009-01-09 15:21:17

[17:13:51] <Soul_keeper> http://sterlingdesktops.com/boincrace/test.png chart now auto updates daily at 10:10am PST
[17:32:16] <PovAddict> Soul_keeper: you aren't HTML-scraping boincstats.com are you?
[17:32:23] <Soul_keeper> PovAddict, yeah
[17:35:07] <PovAddict> what makes you think you're allowed to do that
[17:35:13] <PovAddict> http://boincstats.com/forum/forum_thread.php?id=1161
[17:35:50] <Soul_keeper> if they didn't want me accessing their website then it shouldn't be on the internet
[17:36:23] <PovAddict> that's like saying "if you didn't want me to steal that image you shouldn't have put it on the internet", just because it's on the internet doesn't mean anyone can copy it


I like how the parent directory (http://sterlingdesktops.com/boincrace/) does a directory listing, with plain text scripts that independently display for the world that scraping is exactly what he's doing.
PovAddict
BAM!ID: 115
Joined: 2006-05-10
Posts: 1013
Credits: 5,785,239
World-rank: 76,273

2009-01-27 02:44:02

His image files are showing quite recent "last updated" timestamps... I think Willy blocked the wrong IP or something.
Not running BOINC anymore for several reasons...
asb2106
 
BAM!ID: 33076
Joined: 2007-08-23
Posts: 2494
Credits: 172,859,744
World-rank: 7,348

2009-01-27 19:08:14
last modified: 2009-01-27 19:09:29

Scraping.....??

Could someone describe in general what it is, and what benefits come from it?

**Nevermind** - just read the original post.

I assume the negatives for BOINCstats would be the extra bandwidth that is being taken....
[BOINCstats] Willy
 
Forum moderator - Administrator - Developer - Tester - Translator
BAM!ID: 1
Joined: 2006-01-09
Posts: 9419
Credits: 350,105,499
World-rank: 4,518

2009-01-27 19:43:54

I assume the negatives for BOINCstats would be the extra bandwidth that is being taken....


And the extra demand on the CPU and database.

Scrapers profit from (read: steal) the work of others and present the data as their own.
Please do not PM, IM or email me for support (they will go unread/ignored). Use the forum for support.
asb2106
 
BAM!ID: 33076
Joined: 2007-08-23
Posts: 2494
Credits: 172,859,744
World-rank: 7,348

2009-01-27 20:09:41

Scrapers profit from (read: steal) the work of others and present the data as their own.

Darn scrapers! Do your own work!!!!
[BOINCstats] Willy
 
Forum moderator - Administrator - Developer - Tester - Translator
BAM!ID: 1
Joined: 2006-01-09
Posts: 9419
Credits: 350,105,499
World-rank: 4,518

2009-02-07 07:59:24

His image files are showing quite recent "last updated" timestamps... I think Willy blocked the wrong IP or something.


Been monitoring it and the image still shows January 25th, so I think I've got him.
Please do not PM, IM or email me for support (they will go unread/ignored). Use the forum for support.
Guest

2009-04-26 19:40:15
last modified: 2009-04-26 19:50:56

It's a shame that I was singled out for an example to be made of, I only wget 3 stats pages a day at a several minute delayed stagger.
If this is considered a load on your server as opposed to viewing the same 3 sites serveral times a day to compare results, then I appologize.

Originally I had it grabbing about 8 sites, then I was told by someone this is frowned on "scraping" so I reduced it to 3 and added a sleep delay.
This DOES use equal/less bandwidth than just viewing the 3 sites in my web browser, which this supplements.

Sounds to me like someone has nothing better to do than look for someone to mess with.

[BOINCstats] Willy
 
Forum moderator - Administrator - Developer - Tester - Translator
BAM!ID: 1
Joined: 2006-01-09
Posts: 9419
Credits: 350,105,499
World-rank: 4,518

2009-04-26 20:42:57

It's a shame that I was singled out for an example to be made of, I only wget 3 stats pages a day at a several minute delayed stagger.
If this is considered a load on your server as opposed to viewing the same 3 sites serveral times a day to compare results, then I appologize.

Originally I had it grabbing about 8 sites, then I was told by someone this is frowned on "scraping" so I reduced it to 3 and added a sleep delay.
This DOES use equal/less bandwidth than just viewing the 3 sites in my web browser, which this supplements.

Sounds to me like someone has nothing better to do than look for someone to mess with.



I've explained why I don't allow scraping and I also said I would take measures to prevent it. Be it one page or a thousand, it's my site and I decide whether I allow it or not.

You were warned not to scrape yet you still went on with it. Now live with the consequence.
Please do not PM, IM or email me for support (they will go unread/ignored). Use the forum for support.
PovAddict
BAM!ID: 115
Joined: 2006-05-10
Posts: 1013
Credits: 5,785,239
World-rank: 76,273

2009-04-26 21:45:51

"Warned" includes random irc spam taken out of context

Willy can come and see the full log with its context then. But I think there's no context that can 'save' things like "then their site is a load of crap", or the line at 16:47:32 which I won't repeat here.
Not running BOINC anymore for several reasons...
[BOINCstats] Willy
 
Forum moderator - Administrator - Developer - Tester - Translator
BAM!ID: 1
Joined: 2006-01-09
Posts: 9419
Credits: 350,105,499
World-rank: 4,518

2009-04-26 22:05:00

sterling wrote:
I don't need your site, I'll take my donations elsewhere.


Heh, that's funny. Making a donation costs about five minutes. You've been scraping for over four months, you have a BAM! account for more than two years. Couldn't find the time??

Anyway, your last post convinced me not to lift the ban.
Please do not PM, IM or email me for support (they will go unread/ignored). Use the forum for support.
Kai Strang
 
Tester - BOINCstats SOFA member
BAM!ID: 14104
Joined: 2006-12-06
Posts: 1156
Credits: 266,085,161
World-rank: 5,464

2009-05-17 18:17:40

sterling wrote:

A few recommendations:
[...]
You (povaddict + willy) are nobody's master, and your site is nothing special.
sterling

If that´s the way you see it, I´d recommend you take yourself elsewhere and stop annoying people.

Just a thought ...
Guest

2009-05-17 20:16:21
last modified: 2009-05-17 20:38:55

Kai Strang wrote:
sterling wrote:

A few recommendations:
[...]
You (povaddict + willy) are nobody's master, and your site is nothing special.
sterling

If that´s the way you see it, I´d recommend you take yourself elsewhere and stop annoying people.

Just a thought ...



already done.


some random parting thoughts:

I did the calculations, I was taking a whipping 36KB per day from this website (lynx -dump 3 12KB pages staggered which means 12KB every min+, much less than viewing the website ONCE since I didn't grab any images)
for example if this site has 100KB/s upload that would take a whopping .4 seconds or so to download if done all at once.

to put that in perspective those smiley face icons that pop up every time someone posts a forum message take atleast 45KB alone. And viewing a single forum post by willy takes over 100KB .....

I think I hurt someone's feelings, I wasn't banned for "scrapping". povaddict decided he wanted to exercise his "power" on a website where he had pull because he felt threatened/insecure.


anyways this is my last post.
hopefully you all learned something.

you're welcome
[BOINCstats] Rick
 
BAM!ID: 67731
Joined: 2009-04-08
Posts: 730
Credits: 184
World-rank: 3,478,812

2009-05-17 21:20:19

sterling wrote:
Kai Strang wrote:
If that´s the way you see it, I´d recommend you take yourself elsewhere and stop annoying people.


... already done. anyways this is my last post.


Yes, well, you already said that once before.

sterling wrote:
I don't need your site, I'll take my donations elsewhere. Enjoy your little position of authority without me, bye.


This tme we hope you mean it and make it permanent!


Pages: 1 2 [3] 4

Index :: Announcements :: Scrapers ye be warned!
Reason: