php - Google proxy is a fake crawler? For example: google-proxy-66-249-81-131.google.com -


*edit: solution question below question because not possible post answer. people decide close question.*

recently discover variants of google proxy visits sites. doubt these legal google crawlers because these crawlers not behind proxy (like hostname describes) , identify browser. hostname formatted similar/like google bot string 'proxy' added it.

my php blocking class blocks these crawlers, correct block these ones? , these google or fake?

here info 1 of these crawlers:

blockedip notifier report - ip:66.249.81.131:: has been blocked  ticket id : {evnt_136877_2013040520130402_33147_10348}   event type : access blocked   event date : 04/05/2013 - 19:17:47 (server date-time)   event counter : first occurring   processed url : http://streambutler.net/   url : http://www.google.com/search   domain : streambutler.net  domain ip : 95.170.70.213   visitor ip : 66.249.81.131   proxy ip : 66.249.81.131    critical : yes   action required : no     additional information problem : bad proxy - via 66.249.81.131  hostname : google-proxy-66-249-81-131.google.com  block : yes  refferer : http://www.google.com/search  agentstring : mozilla/5.0 (x11; linux x86_64) applewebkit/537.4 (khtml, g...  browser : chrome 22.0.1229  platform : linux  robot : no  mobile : no  tablet : no  console : no  crawler : no  agent_type : browser  agent_name : chrome  agent_version : 22.0.1229  os_type : linux  os_name : linux  agent_languagetag : en  status : ok  request : 66.249.81.131  languagecode :  country : united states  region : california  city : mountain view  zipcode : 94043  latitude : 37.406  longitude : -122.079  timezone : -07:00   available  : \'http  areacode : 0  dmacode : 0  continentcode : na  currencycode : usd  currencysymbol : $  currencysymbol_utf8 : $  currencyconverter : 1  extended : 1  organization : null

other variants found

  • google-proxy-66-249-81-131.google.com (identifies firefox 6.0 ???)
  • google-proxy-66-249-81-148.google.com (tries access javascript file)
  • google-proxy-66-249-81-131.google.com
  • google-proxy-66-249-81-111.google.com (tries access javascript file)
  • google-proxy-66-249-81-164.google.com

edit: next 1 weird one, firefox 6.0 on windows 7 , same ip example above not proxy in next log? if mobile proxy, weird or not?

ticket id : {evnt_164838_2013040520130402_33147_10348}   event type : access blocked   event date : 04/05/2013 - 19:19:07 (server date-time)   event counter : first occurring   processed url : http://streambutler.net/   url : unknown or direct link   domain : streambutler.net  domain ip : 95.170.70.213   visitor ip : 66.249.81.131   proxy ip : (not present)    critical : yes   action required : no     additional information problem : blocked server ip address (analysis) - 66.249.81.131  hostname : google-proxy-66-249-81-131.google.com  block : yes  refferer : (direct access)  agentstring : mozilla/5.0 (windows nt 6.1; rv:6.0) gecko/20110814 firefox/6.0 ...  browser : firefox 6.0  platform : windows 7  robot : no  mobile : no  tablet : no  console : no  crawler : no  agent_type : browser  agent_name : firefox  agent_version : 6.0  os_type : windows  os_name : windows 7  agent_languagetag : en  status : ok  request : 66.249.81.131  languagecode :  country : united states  region : california  city : mountain view  zipcode : 94043  latitude : 37.406  longitude : -122.079  timezone : -07:00  available  : \'http  areacode : 0  dmacode : 0  continentcode : na  currencycode : usd  currencysymbol : $  currencysymbol_utf8 : $  currencyconverter : 1  extended : 1  organization : null

edit: solution:

got it! these 'crawlers' not crawlers part of live website preview used in google search engine.

i have tried this, show 1 of websites in preview , yes, there is, received blockedip message.

if want users able view preview of website, have accept these 'crawlers'.

like others said: "the root domain of url google.com , can't spoofed".

conclusion: can trust these bot's or crawlers , used show preview in google search.

i haven't confirmed, suspect these ips may associated google's data compression proxy google chrome mobile:

https://developers.google.com/chrome/mobile/docs/data-compression

if case, blocking them cause site display incorrectly innocent mobile users.

it may associated google+ crawler used grab snippets pages using google +1 button:

https://code.google.com/p/google-plus-platform/issues/detail?id=178

bottom line is, these ips used web requests kicked off internal google stuff. not public web proxies.


Comments

Post a Comment

Popular posts from this blog

monitor web browser programmatically in Android? -

i've got tricky question here. need users make payment bank (namely barclaycard) in uk. so, have https url , add parameters (such amount pay, order reference, etc) url, start http connection intent.actionview, redirect user browser can enter credit card details on bank's webpage , make payment our account successfully. far ? the code use below (i changed values privacy reasons) problem is, need app when user has completed/failed/cancelled payment. barclaycardautomatically redirects particular url when payment has succeeded, 1 if failed. there no way of knowing when barclaycard payment has succeeded go android app somehow ? button cardbutton = (button) findviewbyid(r.id.card_button); cardbutton.setonclicklistener(new view.onclicklistener() { @override public void onclick(view arg0) { string prehashstring = new string(); string prohashstring = new string(); string shapassphrase = new string(); shapassphrase = "gsvth£h70zkh...
Read more

Shrink a YouTube video to responsive width -

i have youtube video embedded on our website , when shrink screen tablet or phone sizes stops shrinking @ around 560px in width. standard youtube videos or there can add code make go smaller? you can make youtube videos responsive css. wrap iframe in div class of "videowrapper" , apply following styles: .videowrapper { float: none; clear: both; width: 100%; position: relative; padding-bottom: 56.25%; padding-top: 25px; height: 0; } .videowrapper iframe { position: absolute; top: 0; left: 0; width: 100%; height: 100%; } the .videowrapper div should inside responsive element. padding on .videowrapper necessary keep video collapsing. may have tweak numbers depending upon layout.
Read more

wpf - PdfWriter.GetInstance throws System.NullReferenceException -

i'm trying create pdf document using itextsharp 5.3.4 using following document document = new document(); filestream stm = new filestream(filename, filemode.create); pdfwriter writer = pdfwriter.getinstance(document, stm); i'm getting system.nullreferenceexception following stack trace: system.nullreferenceexception occurred hresult=-2147467261 message=la référence d'objet n'est pas définie à une instance d'un objet. source=itextsharp stacktrace: à itextsharp.text.version.getinstance() innerexception: i've verified neither document nor stm null, , if select "continue" in vs12 document created - exception thrown. updated itextsharp 5.4.0 , it's still occurring. can't find information on anywhere - got ideas? make sure not catching exceptions. nullreferenceexception can 1 caught , handled inside itextsharp, don't care it. fact can continue supports theory. change following setting verify: debug -> exceptions ...
Read more