java - Selenium takes lots of time to get dynamic page of given URL -


i doing project in java. in project have work dom. first load dynamic page of given url, using selenium. parse them using jsoup.

i want dynamic page source code of given url

code snapshot:

public static void main(string[] args) throws ioexception {       // selenium      webdriver driver = new firefoxdriver();      driver.get("any url here");        string html_content = driver.getpagesource();      driver.close();       // jsoup makes dom here parsing html content      document doc = jsoup.parse(html_content);       // operations using dom tree } 

but problem is, selenium takes around 95% of whole processing time, undesirable.

selenium first opens firefox, loads given page, gets dynamic page source code.

can tell me how can reduce time taken selenium, replacing tool efficient tool. other advice welcome.

edit no. 1

there code given on link.

firefoxprofile profile = new firefoxprofile(); profile.setpreference("general.useragent.override", "some ua string"); webdriver driver = new firefoxdriver(profile); 

but second line here, didn't understand. documentation poor of selenium.

edit no. 2

system.out.println("fetching %s..." + url1); system.out.println("fetching %s..." + url2);

    webdriver driver = new firefoxdriver(createfirefoxprofile());      driver.get("url1");       string hml1 = driver.getpagesource();      driver.get("url2");     string hml2 = driver.getpagesource();     driver.close();      document doc1 = jsoup.parse(hml1);     document doc2 = jsoup.parse(hml2); 

try this:

public static void main(string[] args) throws ioexception {      // selenium     webdriver driver = new firefoxdriver(createfirefoxprofile());     driver.get("any url here");     string html_content = driver.getpagesource();     driver.close();      // jsoup makes dom here parsing html content     // operations using dom tree }  private static firefoxprofile createfirefoxprofile() {     file profiledir = new file("/tmp/firefox-profile-dir");     if (profiledir.exists())         return new firefoxprofile(profiledir);     firefoxprofile firefoxprofile = new firefoxprofile();     file dir = firefoxprofile.layoutondisk();     try {         profiledir.mkdirs();         fileutils.copydirectory(dir, profiledir);     } catch (ioexception e) {         e.printstacktrace();     }     return firefoxprofile; } 

the createfirefoxprofile() method creates profile if 1 doesn't exist. uses if profile exists. selenium doesn't need create profile-dir structure each , every time.


Comments

Popular posts from this blog

monitor web browser programmatically in Android? -

Shrink a YouTube video to responsive width -

wpf - PdfWriter.GetInstance throws System.NullReferenceException -