Re: [IMP-dev] profiling IMP files on Unix

A general note - my experience is with profiling C++ heavy runs (even if bound by python) that are serial (working on one CPU each). There are good solutions for profiling the python script itself (the time spent performing actual python operations, not just calling IMP binary libraries). I can elaborate more if relevant (but test what I sent first!). Also, profiling parallel programs might work but I’m not sure - start with profiling serial runs before trying parallel, if possible.

On Tue, Jun 19, 2018 at 12:52 PM Barak Raveh <">> wrote:

This is for binary files or python files on unix machines, examples are given in bash.
I haven't profiled Mac files though I think Charles has. Only profile files in fast mode - no point in profiling debug builds!

Profiling python IMP files:

1) Use the CPU_PROFILE flag to specify an output .prof file (on some systems you might also need to add LD_PRELOAD=/usr/lib64/libprofiler.so - probably not):

$ CPUPROFILE=./barak.prof /home/barak/imp_git/fast/setup_environment.sh python expensive_test_statistics_from_simulation_data.py &> LOG

2) Use "pprof --pdf <path-to-python-binary> <path-to-.prof-file>" to print to PDF format - perhaps the process id will be appended to the output profile file. Use "--gv", "--txt" or some of the other pprof options if you like.

pprof --pdf /usr/bin/python barak.prof_31784 > 31784.pdf

Profiling binary IMP files:

1) Run your binary IMP program as you usually would, but with the --cpu_profile flag.
(this is a generic IMP flag, which will create a .prof file with profiling information. The default .prof file will be created in the current folder, or edit CPU_PROFILE to change the path of the output .prof file).

2) Use "pprof --pdf <path-to-binary-IMP-file> <path-to-.prof-file>" to print to PDF format. Use "--gv", "--txt" or some of the other pprof options if you like.

Comments:

1) Documentation of gperftools https://github.com/gperftools/gperftools

2) See also possibly partially updated IMP page on profiling https://github.com/salilab/imp/blob/4941f4e11403bac687a64b273f80f1ccc70b3522/doc/manual/profiling.md

3) Try "pprof --help" for some interesting options