Improved performance of parse_file_contents() method
Before change Ic6236665f2d55b24a56a99120ac57fc2b18e32eb policy file had been parsed by jsonutils.loads() For some reasons jsonutils.loads() is much faster than yaml.safe_load(). For example this script takes 30 seconds: s = '{"a": 3, "b": "c"}' for i in xrange(100000): yaml.safe_load(s) And this script takes only 1 second! s = '{"a": 3, "b": "c"}' for i in xrange(100000): jsonutils.loads(s) Moreover policy rules are presented in JSON format by default (at least in Nova). This patch proposes to parse file by jsonutils.loads(). If it was parsed - use parsed data. If it's failed - try to parse with yaml.safe_load(). It will have no/minimum performance effect on real env because, as I understand, we parse policy file only one time during service start. But it will have great performance improvement in our jenkins jobs. Statistics for nova py27, py34 and functional jobs: * with this change I0c18e9746b742a2fff60795da8a5daddda7cf469 * without this change Ie4af661f4ca16b63444692f3736835f9ce42adc1 Here was taken pure time of running "tox -e py27/py34/functional" commands without (sec) with (sec) py27 421 240 py34 330 170 functional 662 495 As you see performance of unit tests was increased almost in two times. These changes covered by tests in test_policy.py file. There we test method Rules.load() with invalid/valid JSON/YAML. As you can see method Rules.load() directly uses parse_file_contents(). Change-Id: I43782d245d7652ba69613b26fe598ac79ec19929
This commit is contained in:
parent
a66ee73e8c
commit
f56b767ddd
@ -325,11 +325,19 @@ def parse_file_contents(data):
|
||||
'policy_name2': 'policy2,...}
|
||||
"""
|
||||
try:
|
||||
parsed = yaml.safe_load(data)
|
||||
except yaml.YAMLError as e:
|
||||
# For backwards-compatibility, convert yaml error to ValueError,
|
||||
# which is what JSON loader raised.
|
||||
raise ValueError(six.text_type(e))
|
||||
# NOTE(snikitin): jsonutils.loads() is much faster than
|
||||
# yaml.safe_load(). However jsonutils.loads() parses only JSON while
|
||||
# yaml.safe_load() parses JSON and YAML. So here we try to parse data
|
||||
# by jsonutils.loads() first. In case of failure yaml.safe_load()
|
||||
# will be used instead.
|
||||
parsed = jsonutils.loads(data)
|
||||
except ValueError:
|
||||
try:
|
||||
parsed = yaml.safe_load(data)
|
||||
except yaml.YAMLError as e:
|
||||
# For backwards-compatibility, convert yaml error to ValueError,
|
||||
# which is what JSON loader raised.
|
||||
raise ValueError(six.text_type(e))
|
||||
return parsed
|
||||
|
||||
|
||||
|
Loading…
x
Reference in New Issue
Block a user