Profile Oslo Service processes
This patch enables profiling (capturing function call trace like cProfile [1]) worker processes on the fly while service is running. User requests the oslo service process to start profiling by writing "prof()" command to backdoor socket, once the service (like neutron-server) finishes expected processing (example finishing API call), user again writes "prof()" command with file name as argument to dump the function calltrace stats. Stats file (in pstat format with user provided filename by adding .prof) will be generated in temp directory. For example, to profile neutron server process, 1) echo "prof()" | nc localhost 8002 2) Issue neutron command (or run rally scenarios tests) neutron net-create n1 neutron port-create --name p1 n1 neutron port-delete p1 neutron net-delete n1 3) echo "prof('neutron')" | nc localhost 8002 where 8002 is the port which we set like below in neutron.conf backdoor_port=8002 We can later print the stats from the trace file like below stats = pstats.Stats('/tmp/neutron.prof') stats.print_stats() The trace file will look like in (for above neutron API calls) [2]. We use Yappi with context set to greenlet [3] to profile greenlets. We can't use GreenletProfiler [4], which does the same [5] 1) as it is no more maintained 2) Also compiling yappi source inside GreenletProfiler is failing for python3. [1] https://docs.python.org/2/library/profile.html [2] https://gist.github.com/venkataanil/64d5e672bf0206dc151e73fc1058a983 [3] https://bitbucket.org/sumerc/yappi/pull-requests/3 [4] https://pypi.org/project/GreenletProfiler/ [5] https://emptysqua.re/blog/greenletprofiler/ Depends-On: Ibea0cdb732923f1b53d5cb6aeeb4041fb5973494 Change-Id: Id2418093494f1e233a653f6c73bd6894e4a40184
This commit is contained in:
parent
d987a4a84c
commit
a04daefbb1
@ -179,3 +179,40 @@ logging options by sending a SIGHUP.
|
||||
logging.setup(cfg.CONF, 'foo')
|
||||
|
||||
|
||||
Profiling
|
||||
~~~~~~~~~
|
||||
|
||||
Processes spawned through oslo_service.service can be profiled (function
|
||||
calltrace) through eventlet_backdoor module. Service has to configure
|
||||
backdoor_port option to enable it's workers to listen on TCP ports.
|
||||
Then user can send "prof()" command to capture worker processes function
|
||||
calltrace.
|
||||
|
||||
1) To start profiling send "prof()" command on processes listening port
|
||||
|
||||
2) To stop profiling and capture "pstat" calltrace to a file, send prof
|
||||
command with filename as argument i.e "prof(filename)"
|
||||
on worker processes listening port. Stats file (in pstat format) with
|
||||
user provided filename by adding .prof as suffix will be generated
|
||||
in temp directory.
|
||||
|
||||
For example, to profile neutron server process (which is listening on
|
||||
port 8002 configured through backdoor_port option),
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ echo "prof()" | nc localhost 8002
|
||||
$ neutron net-create n1; neutron port-create --name p1 n1;
|
||||
$ neutron port-delete p1; neutron port-delete p1
|
||||
$ echo "prof('neutron')" | nc localhost 8002
|
||||
|
||||
|
||||
This will generate "/tmp/neutron.prof" as stats file. Later user can print
|
||||
the stats from the trace file like below
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import pstats
|
||||
|
||||
stats = pstats.Stats('/tmp/neutron.prof')
|
||||
stats.print_stats()
|
||||
|
@ -72,3 +72,4 @@ traceback2==1.4.0
|
||||
unittest2==1.1.0
|
||||
WebOb==1.7.1
|
||||
wrapt==1.7.0
|
||||
Yappi==0.98
|
||||
|
@ -23,10 +23,12 @@ import os
|
||||
import pprint
|
||||
import socket
|
||||
import sys
|
||||
import tempfile
|
||||
import traceback
|
||||
|
||||
import eventlet.backdoor
|
||||
import greenlet
|
||||
import yappi
|
||||
|
||||
from oslo_service._i18n import _
|
||||
from oslo_service import _options
|
||||
@ -89,6 +91,30 @@ def _find_objects(t):
|
||||
return [o for o in gc.get_objects() if isinstance(o, t)]
|
||||
|
||||
|
||||
def _capture_profile(fname=''):
|
||||
if not fname:
|
||||
yappi.set_clock_type('cpu')
|
||||
# We need to set context to greenlet to profile greenlets
|
||||
# https://bitbucket.org/sumerc/yappi/pull-requests/3
|
||||
yappi.set_context_id_callback(
|
||||
lambda: id(greenlet.getcurrent()))
|
||||
yappi.set_context_name_callback(
|
||||
lambda: greenlet.getcurrent().__class__.__name__)
|
||||
yappi.start()
|
||||
else:
|
||||
yappi.stop()
|
||||
stats = yappi.get_func_stats()
|
||||
# User should provide filename. This file with a suffix .prof
|
||||
# will be created in temp directory.
|
||||
try:
|
||||
stats_file = os.path.join(tempfile.gettempdir(), fname + '.prof')
|
||||
stats.save(stats_file, "pstat")
|
||||
except Exception as e:
|
||||
print("Error while saving the trace stats ", str(e))
|
||||
finally:
|
||||
yappi.clear_stats()
|
||||
|
||||
|
||||
def _print_greenthreads(simple=True):
|
||||
for i, gt in enumerate(_find_objects(greenlet.greenlet)):
|
||||
print(i, gt)
|
||||
@ -162,6 +188,7 @@ def _initialize_if_enabled(conf):
|
||||
'fo': _find_objects,
|
||||
'pgt': _print_greenthreads,
|
||||
'pnt': _print_nativethreads,
|
||||
'prof': _capture_profile,
|
||||
}
|
||||
|
||||
if conf.backdoor_port is None and conf.backdoor_socket is None:
|
||||
|
5
releasenotes/notes/profile-worker-5d3fd0f0251d62b8.yaml
Normal file
5
releasenotes/notes/profile-worker-5d3fd0f0251d62b8.yaml
Normal file
@ -0,0 +1,5 @@
|
||||
---
|
||||
features:
|
||||
- |
|
||||
Add support for profiling (capture function calltrace) service's worker
|
||||
processes.
|
@ -17,3 +17,4 @@ oslo.i18n>=3.15.3 # Apache-2.0
|
||||
PasteDeploy>=1.5.0 # MIT
|
||||
Routes>=2.3.1 # MIT
|
||||
Paste>=2.0.2 # MIT
|
||||
Yappi>=0.98 # MIT
|
||||
|
Loading…
x
Reference in New Issue
Block a user