java - OutOfMemoryError while using HtmlUnit for scrapping -


i using htmlunit login on site , download data table

when run code is causing java.lang.outofmemoryerror , not run further.

following code :

webclient webclient = new webclient(browserversion.internet_explorer_6); webclient.getoptions().setjavascriptenabled(true); webclient.getoptions().setcssenabled(false); webclient.getoptions().setredirectenabled(true); webclient.getcookiemanager().setcookiesenabled(true);                             webclient.getoptions().setprintcontentonfailingstatuscode(false); webclient.setajaxcontroller(new nicelyresynchronizingajaxcontroller()); webclient.getoptions().settimeout(50000); webclient.getoptions().setuseinsecuressl(true); webclient.getoptions().setpopupblockerenabled(true);  htmlpage htmlpage=webclient.getpage(url); thread.sleep(200);                             //~~~~~~~log-in htmltextinput uname=(htmltextinput)htmlpage.getfirstbyxpath("//*[@id=\"username\"]"); uname.setvalueattribute("xxx"); htmlpasswordinput upass=(htmlpasswordinput)htmlpage.getfirstbyxpath("//*[@id=\"password\"]"); upass.setvalueattribute("xxx"); htmlsubmitinput submit=(htmlsubmitinput)htmlpage.getfirstbyxpath("//*[@id=\"login-button\"]/input"); htmlpage=(htmlpage) submit.click(); thread.sleep(200); webclient.waitforbackgroundjavascript(10000); (int = 0; < 250; i++) {  if (!htmlpage.astext().contains("loading...")) {      break;   }     synchronized (htmlpage) {      htmlpage.wait(500);  } }  system.out.println(htmlpage.astext());

and following stacktrace

java.lang.outofmemoryerror: java heap space @ net.sourceforge.htmlunit.corejs.javascript.node.newstring(node.java:155) @ net.sourceforge.htmlunit.corejs.javascript.node.newstring(node.java:151) @ net.sourceforge.htmlunit.corejs.javascript.irfactory.createpropertyget(irfactory.java:1990) @ net.sourceforge.htmlunit.corejs.javascript.irfactory.transformpropertyget(irfactory.java:968) @ net.sourceforge.htmlunit.corejs.javascript.irfactory.transform(irfactory.java:106) @ net.sourceforge.htmlunit.corejs.javascript.irfactory.transformpropertyget(irfactory.java:964) @ net.sourceforge.htmlunit.corejs.javascript.irfactory.transform(irfactory.java:106) @ net.sourceforge.htmlunit.corejs.javascript.irfactory.transformpropertyget(irfactory.java:964) @ net.sourceforge.htmlunit.corejs.javascript.irfactory.transform(irfactory.java:106) @ net.sourceforge.htmlunit.corejs.javascript.irfactory.transformfunctioncall(irfactory.java:595) @ net.sourceforge.htmlunit.corejs.javascript.irfactory.transform(irfactory.java:86) @ net.sourceforge.htmlunit.corejs.javascript.irfactory.transforminfix(irfactory.java:775) @ net.sourceforge.htmlunit.corejs.javascript.irfactory.transform(irfactory.java:161) @ net.sourceforge.htmlunit.corejs.javascript.irfactory.transformassignment(irfactory.java:368) @ net.sourceforge.htmlunit.corejs.javascript.irfactory.transform(irfactory.java:152) @ net.sourceforge.htmlunit.corejs.javascript.irfactory.transformexprstmt(irfactory.java:488) @ net.sourceforge.htmlunit.corejs.javascript.irfactory.transform(irfactory.java:149) @ net.sourceforge.htmlunit.corejs.javascript.irfactory.transformblock(irfactory.java:406) @ net.sourceforge.htmlunit.corejs.javascript.irfactory.transform(irfactory.java:82) @ net.sourceforge.htmlunit.corejs.javascript.irfactory.transformif(irfactory.java:762) @ net.sourceforge.htmlunit.corejs.javascript.irfactory.transform(irfactory.java:110) @ net.sourceforge.htmlunit.corejs.javascript.irfactory.transformblock(irfactory.java:406) @ net.sourceforge.htmlunit.corejs.javascript.irfactory.transform(irfactory.java:82) @ net.sourceforge.htmlunit.corejs.javascript.irfactory.transformif(irfactory.java:762) @ net.sourceforge.htmlunit.corejs.javascript.irfactory.transform(irfactory.java:110) @ net.sourceforge.htmlunit.corejs.javascript.irfactory.transformblock(irfactory.java:406) @ net.sourceforge.htmlunit.corejs.javascript.irfactory.transform(irfactory.java:82) @ net.sourceforge.htmlunit.corejs.javascript.irfactory.transformif(irfactory.java:768) @ net.sourceforge.htmlunit.corejs.javascript.irfactory.transform(irfactory.java:110) @ net.sourceforge.htmlunit.corejs.javascript.irfactory.transformblock(irfactory.java:406) @ net.sourceforge.htmlunit.corejs.javascript.irfactory.transform(irfactory.java:82) @ net.sourceforge.htmlunit.corejs.javascript.irfactory.transformfunction(irfactory.java:560)

i have put follwoing lines in catlina.sh file allot heap memory still getting same error (my ram size 2gb).

if [ -z "$logging_manager" ];      java_opts="$java_opts -djava.util.logging.manager=org.apache.juli.classloaderlogmanager" else      java_opts="$java_opts $logging_manager" fi  # uncomment following line make umask available when using # org.apache.catalina.security.securitylistener    java_opts="$java_opts -dorg.apache.catalina.security.securitylistener.umask=`umask`"    java_opts="$java_opts  -xms512m -xmx2048m -xx:maxpermsize=512m"    java_opts="-server -xx:+useconcmarksweepgc"

you include $java_opts @ last line of code,may code works

java_opts="$java_opts -server -xx:+useconcmarksweepgc"

Comments

Post a Comment

Popular posts from this blog

monitor web browser programmatically in Android? -

i've got tricky question here. need users make payment bank (namely barclaycard) in uk. so, have https url , add parameters (such amount pay, order reference, etc) url, start http connection intent.actionview, redirect user browser can enter credit card details on bank's webpage , make payment our account successfully. far ? the code use below (i changed values privacy reasons) problem is, need app when user has completed/failed/cancelled payment. barclaycardautomatically redirects particular url when payment has succeeded, 1 if failed. there no way of knowing when barclaycard payment has succeeded go android app somehow ? button cardbutton = (button) findviewbyid(r.id.card_button); cardbutton.setonclicklistener(new view.onclicklistener() { @override public void onclick(view arg0) { string prehashstring = new string(); string prohashstring = new string(); string shapassphrase = new string(); shapassphrase = "gsvth£h70zkh...
Read more

Shrink a YouTube video to responsive width -

i have youtube video embedded on our website , when shrink screen tablet or phone sizes stops shrinking @ around 560px in width. standard youtube videos or there can add code make go smaller? you can make youtube videos responsive css. wrap iframe in div class of "videowrapper" , apply following styles: .videowrapper { float: none; clear: both; width: 100%; position: relative; padding-bottom: 56.25%; padding-top: 25px; height: 0; } .videowrapper iframe { position: absolute; top: 0; left: 0; width: 100%; height: 100%; } the .videowrapper div should inside responsive element. padding on .videowrapper necessary keep video collapsing. may have tweak numbers depending upon layout.
Read more

wpf - PdfWriter.GetInstance throws System.NullReferenceException -

i'm trying create pdf document using itextsharp 5.3.4 using following document document = new document(); filestream stm = new filestream(filename, filemode.create); pdfwriter writer = pdfwriter.getinstance(document, stm); i'm getting system.nullreferenceexception following stack trace: system.nullreferenceexception occurred hresult=-2147467261 message=la référence d'objet n'est pas définie à une instance d'un objet. source=itextsharp stacktrace: à itextsharp.text.version.getinstance() innerexception: i've verified neither document nor stm null, , if select "continue" in vs12 document created - exception thrown. updated itextsharp 5.4.0 , it's still occurring. can't find information on anywhere - got ideas? make sure not catching exceptions. nullreferenceexception can 1 caught , handled inside itextsharp, don't care it. fact can continue supports theory. change following setting verify: debug -> exceptions ...
Read more