java - Caliper: micro- and macro benchmarks -

for elki need (and have) more flexible sorting implementations provided standard java jdk , collections api. (sorting not ultimate goal. use partial sorting bulk loading index structures such k-d-tree , r*-tree, , want make rather generic implementation of these available, more generic in elki - either way, optimizing sort means optimizing index construction time).

however, sorting algorithms scale differently depending on data size. tiny arrays, known fact insertion sort can perform (and in fact, quicksort implementations fall insertion sort below threshold); not theory cpu pipelining , code size effects not considered sorting theory.

so i'm benchmarking number of sorting implementations find best combination particular needs; want more flexible implementations on par jdk default implementations (which fine tuned, maybe different jdk version).

in long run, need these things easy reproduce , re-run. @ point, we'll see jdk8. , on dalvik vm, results may different on java 7. heck, might different on amd, core i7 , atom cpus, too. maybe cervidae include different sorting strategies, , choose appropriate 1 on class loading time.

my current efforts on github: https://github.com/kno10/cervidae

so actual question. latest caliper commit added experimental code macrobenchmarks. however, i'm facing problem need both. caliper macrobenchmarks fail when runtime less 0.1% of timer resolution; 10000 objects algorithms hit threshold. @ same time, microbenchmarks complain should doing macrobenchmark when runs take long...

so benchmarking different sort sizes, i'd need approach dynamically switches microbenchmarking macrobenchmarking depending on runtime. in fact, i'd prefer if caliper automagically realize runtime large enough macro benchmark, , single iteration.

right now, i'm trying emulate using:

@macrobenchmark public int macrobenchmark() { ... }  public int timemicrobenchmark(int reps) {     int ret = 0;     (int = 0; < reps; i++) {         ret += macrobenchmark();     } } 

to share benchmarking code across both scenarios. alternate code use

@macrobenchmark public int macrobenchmark() {     return timemicrobenchmark(1); }  public int timemicrobenchmark(int reps) { ... } 

which of 2 "adapters" preferrable? other hints getting consistent benchmarking micro way macro?

given caliper webui currenty not functional, use analyzing results? i'm using tiny python script process json result , report weighted means. , in fact, liked old text reporting better web ui.

oh, , there way have caliper re-run benchmark when hotspot compilation occurred in benchmarking loop? right logs error, maybe re-start part of benchmark?

i think issue output microbenchmark instrument being misinterpreted "complaint". says:

"info: experiment not require microbenchmark. granularity of timer (%s) less 0.1%% of measured runtime. if experiments benchmark have runtimes greater %s, consider macrobenchmark instrument."

the message worded convey individual experiment lengthy, since other experiments benchmark method may not be, it's not error. there bit more overhead microbenchmark instrument, while experiment may not require microbenchmark, results still valid.