LiveLeak = 500,000+ Visitors/Day -> Not Indexed by Google?
LiveLeak.com is a sometimes-controversial video-based site which receives over a half-million visitors a day and was, until just now, completely unindexed in Google - because they accidentally told Google not to index their site! The funniest part is that, as of 4 AM PST, their own front page still has a ‘try it for yourself’ link that encourages users to see that they aren’t indexed by Google.
In a rather tragic personal video a depressed-sounding LiveLeak administrator describes how he has been trying to contact Google about this potential act of censorship. Meanwhile, this video has received comments ranging from ranging from “google has stepped out of bounds” to “google is with the government in censoring the internet and they must be stopped!!!” Meanwhile, of course, they were informed (by a friend of mine) of the problem and, as you can see below, have fixed it:
Even the Wikipedia article on Google censorship contains an (albeit now contested) sub-section pointing out this issue. How can a site with a half a million visitors a day and numerous web professionals looking into the problem NOT figure this out sooner? Go(d)ogle only knows, but now that they have removed the bot they are sure to get a flood of links from Google. Maybe it was just reverse psychology to begin with :P
RobotGod said,
April 11, 2007 @ 4:11 am
Whoops, I’m a dumbass - I misread this article the first time - my bad! Didn’t realize you were posting a later screenshot LOL!
If you wanted to block all robots then you’d do this:
User-agent: *
Disallow: /
Trojan said,
April 11, 2007 @ 4:36 am
Erm…You do know that isn’t how it works right? Google were indexing another URL for the site just not the .com. Funny post though, nice one :)
Trojan said,
April 11, 2007 @ 4:38 am
Just to help you out
http://en.wikipedia.org/wiki/Robots_exclusion_file
Trojan said,
April 11, 2007 @ 9:12 am
Seriously bro, you’re going to look pretty dumb soon. That robot.txt is right and allows all robots. Look it up or look foolish :)
No, Google non censura LiveLeak « il blog di pda said,
April 11, 2007 @ 9:23 am
[…] 11th, 2007 Il pericolo censura non c’è. E’ il file robots.txt del sito che lo ha spinto fuori dai.. robot di Google ;) Posted by pipda Filed in altri […]
kroq said,
April 11, 2007 @ 8:26 pm
Hrm … yes, dumb. Actually, it seems pretty dumb to me that you couldn’t figure out I took that screenshot AFTER they fixed the problem :S But I digress - thanks for pointing out that I needed to make my article crystal clear on that point!
Trojan said,
April 11, 2007 @ 8:27 pm
Kroq
The screenshot is the same as it was when this whole thing started cause it was the first thing I went to look at. Someone on Digg came to the same conclusion as you and then made a retraction. Sorry.
Qualsiasi.NET said,
April 11, 2007 @ 10:53 pm
500.000 unici al giorno, e non indicizzato….
[…] Parlo di LiveLeak, un portale di video che non stanno sempre comodi a tutti, secondo quello che leggo in giro sui vari blog. Personalmente non l’ho visitato a fondo.
La questione è un altra, sembrerebbe che questo sito non sia indicizzato da G…
kroq said,
April 12, 2007 @ 4:53 am
No apologies necessary … after all, you’re wrong. It was on there originally - I just didn’t get a screenshot of it because I didn’t realize they were about to remove it. Thanks for playing tho :D
Trojan said,
April 12, 2007 @ 5:22 am
The robots.txt file didn’t change though. This is the point I’m getting at. On the day the video came out a Digg member posted the same .txt saying that was blocking it then retracted his statement after he realised. However, your story does make for more interesting copy :)
Anonymous said,
April 12, 2007 @ 9:53 am
The robots.txt indicated “Disallow: /”. Yahoo and MSN/Live.com dont check the robots.txt file as much as google and rather reference a cached version which could have been out of date. That would explain how those two engines had liveleak content in their index and google did not.
Unless you really want to think Google cock-blocked LiveLeak’s poorly-run and slow video service from taking their own traffic away. Seeing as though YouTube removes 99% of the material that makes up LiveLeak’s index on sight, that a very stupid conclusion to come to.
kroq said,
April 12, 2007 @ 11:30 pm
you know … all i REALlY ask is that people READ the post before they respond. that screnshot was taken later
Anonymous said,
April 13, 2007 @ 3:15 am
My last response was directed to Trojan and the hordes of idiots who came up with weird-ass conspiracy theories instead of any sort of logical, real-world conclusion.
kroq said,
April 16, 2007 @ 12:48 am
lol whoops
Trojan said,
April 16, 2007 @ 5:14 am
Did I say I thought Google blocked it? I doubt Google would care about a website so far under their product. I was just talking about the .txt file which was the same when I checked it when this first came out as it was when this entry was written. Also appears they had other url’s indexed, I was just interested if you all want to get defensive and put words in my mouth about conspiracies it says more about you than me.
kroq said,
May 20, 2007 @ 2:04 pm
Yes, ummm, it’s a little hard NOT to put words in your mouth, since the ones you use in the first place are confusing at best. Sorry!
Fred Purseleather said,
June 10, 2007 @ 2:25 pm
@Trojan:
Are you fucking kidding me? You’re actually hiding behind that ingorant, half-assed response?
Please explain yourself or get laughed at.
Trojan said,
June 18, 2007 @ 3:28 am
@Fred
Whut? Laugh away big man, it’s all good :)
AllSux.com » BREAKING: Allsux Author Picked as Hottest Member from 200,000 MyBlogLog Users?! I Sux! said,
August 7, 2007 @ 5:03 pm
[…] This site, for new readers tuning in, has taken a lot of big-name internet companies to task, from LiveLeak to Digg and many others along the way. Possibly the only site I have posted only positive praise […]
Majestic said,
August 8, 2007 @ 3:05 am
@kroq aka Anonymous/Fred
Don’t be an idiot! We all know Anonymous, Fred and kroq are the same person.
kroq said,
August 8, 2007 @ 3:22 am
LOLZ ummm … back that up with some semantic analysis or IP work or, well, anything. Otherwise … congrads! I have approved your comment for everyone else to see how much of an idiot you are :D