python - How to crawl and download files from a dynamic URL? -


i have own python crawler(based on cs101 udacity.com), trying download files(installers) download.cnet.com, when crawler crawling, want work this:

  1. tell if link download link:

    response = urllib2.urlopen('http://example.com/')

    content_type = response.info().get('content-type')

    print content_type

  2. if crawler gets:

    application/octet-stream
  3. the crawler download installer link

the problem download.com doesn't seem provide real download link, , crawler can't find download link dynamic links. example, when tried download opera in download.com, have message this: "your download begin in moment. if doesn't, restart download." when checked "restart download" link, expecting real download link(e.g. download.com/blah/opera.exe), instead got wierd address crawler couldn't understand.

so have confirmed http://googlewebmastercentral.blogspot.no/2008/09/dynamic-urls-vs-static-urls.html download.com using dynamic links, how should in order let crawler find link can download installer download.com?

as you've said, you're getting javascript or ajax in page activates download in "real" browser while stymying efforts automate it.

here's discussion of same issue: stackoverflow: mechanize , javascript. noted there, 1 option use alternative python such phantomjs or browser automation framework (with optional "remote control") such selenium.


Comments

Post a Comment

Popular posts from this blog

ios - iPhone/iPad different view orientations in different views , and apple approval process -

i force app's home screen portrait , other views landscape. apple rejects app if this? if not , how force first screen launch in portrait mode? , force other screens launch in landscape. this not duplicate question search every possible answer in website. i force app's home screen portrait , other views landscape. apple rejects app if this? no apple not reject app this. have lot of apps doesn't support orientation. forcing on 1 screen won't cause app reject. if not , how force first screen launch in portrait mode? , force other screens launch in landscape. can on write : -(nsuinteger)supportedinterfaceorientations; function.
Read more

java Extracting Zip file -

i'm looking way extract zip file. far have tried java.util.zip , org.apache.commons.compress, both gave corrupted output. basically, input zip file contain 1 single .doc file. java.util.zip: output corrupted. org.apache.commons.compress: output blank file, 2 mb size. so far commercial software winrar work perfectly. there java library make use of this? this method using java.util library: public void extractzipnative(file filezip) { zipinputstream zis; stringbuilder sb; try { zis = new zipinputstream(new fileinputstream(filezip)); zipentry ze = zis.getnextentry(); byte[] buffer = new byte[(int) ze.getsize()]; fileoutputstream fos = new fileoutputstream(this.tempfolderpath+ze.getname()); int len; while ((len=zis.read(buffer))>0) { fos.write(buffer); } fos.flush(); fos.close(); } catch (filenotfoundexception e) { // todo auto-generated catch block ...
Read more

C# WinForm - loading screen -

i ask how make loading screen (just picture or something) appears while program being loaded, , disappears when program has finished loading. in fancier versions, have seen process bar (%) displayed. how can have that, , how calculate % show on it? i know there form_load() event, not see form_loaded() event, or % property / attribute anywhere. all need create 1 form splash screen , show before main start showing landing page , close splash once landing page loaded. class splashform { //delegate cross thread call close private delegate void closedelegate(); //the type of form displayed splash screen. private static splashform splashform; static public void showsplashscreen() { // make sure launched once. if (splashform != null) return; thread thread = new thread(new threadstart(splashform.showform)); thread.isbackground = true; thread.setapartmentstate(apartmentstate.sta); thread....
Read more