Memory efficient way to import large files and data into MongoDB? -


after experimenting mongodb, tried few different methods of importing/inserting large amounts of data collections. far efficient method i've found mongoimport. works perfectly, there still overhead. after import complete, memory isn't made available unless reboot machine.

example:

mongoimport -d flightdata -c trajectory_data --type csv --file trjdata.csv --headerline

where headerline , data like:

'fid','acid','flight_index','orig_index','orig_time','cur_lat', ... '20..','j5','79977,'79977','20110116:15:53:11','1967', ...

with 5.3 million rows 20 columns, 900mb, end this:

overhead

this won't work me in long run; may not able reboot, or run out of memory. more effective way of importing mongodb? i've read periodic ram flushing, how implement example above?

update: don't think case benefit adjusting fsync, syncdelay, or journaling. i'm curious when idea, , best practice, if running on high ram servers.

i'm guessing memory being used mongodb itself, not mongoimport. mongodb design tries keep of data memory , relies on os swap memory-mapped files out when there's not enough room. i'd give 2 pieces of advice:

  1. don't worry os telling how memory "free" -- modern well-running os use every bit of ram available something.

  2. if can't abide #1, don't run mongodb on laptop.


Comments

Post a Comment

Popular posts from this blog

ios - iPhone/iPad different view orientations in different views , and apple approval process -

i force app's home screen portrait , other views landscape. apple rejects app if this? if not , how force first screen launch in portrait mode? , force other screens launch in landscape. this not duplicate question search every possible answer in website. i force app's home screen portrait , other views landscape. apple rejects app if this? no apple not reject app this. have lot of apps doesn't support orientation. forcing on 1 screen won't cause app reject. if not , how force first screen launch in portrait mode? , force other screens launch in landscape. can on write : -(nsuinteger)supportedinterfaceorientations; function.
Read more

java Extracting Zip file -

i'm looking way extract zip file. far have tried java.util.zip , org.apache.commons.compress, both gave corrupted output. basically, input zip file contain 1 single .doc file. java.util.zip: output corrupted. org.apache.commons.compress: output blank file, 2 mb size. so far commercial software winrar work perfectly. there java library make use of this? this method using java.util library: public void extractzipnative(file filezip) { zipinputstream zis; stringbuilder sb; try { zis = new zipinputstream(new fileinputstream(filezip)); zipentry ze = zis.getnextentry(); byte[] buffer = new byte[(int) ze.getsize()]; fileoutputstream fos = new fileoutputstream(this.tempfolderpath+ze.getname()); int len; while ((len=zis.read(buffer))>0) { fos.write(buffer); } fos.flush(); fos.close(); } catch (filenotfoundexception e) { // todo auto-generated catch block ...
Read more

C# WinForm - loading screen -

i ask how make loading screen (just picture or something) appears while program being loaded, , disappears when program has finished loading. in fancier versions, have seen process bar (%) displayed. how can have that, , how calculate % show on it? i know there form_load() event, not see form_loaded() event, or % property / attribute anywhere. all need create 1 form splash screen , show before main start showing landing page , close splash once landing page loaded. class splashform { //delegate cross thread call close private delegate void closedelegate(); //the type of form displayed splash screen. private static splashform splashform; static public void showsplashscreen() { // make sure launched once. if (splashform != null) return; thread thread = new thread(new threadstart(splashform.showform)); thread.isbackground = true; thread.setapartmentstate(apartmentstate.sta); thread....
Read more