starlingx/integ
Code Issues Proposed changes

Integrate gnocchi for OpenStack Telemetry

Browse Source 
This is part of feature integrating gnocchi as metric storage backend
because of the deprecation of ceilometer metering backend.
With gnocchi integrated, ceilometer api and collector processes are
removed, ceilometer agent nofitication and polling processes
are still running to collect and normalise data from other openstack
services. Ceilometer notification agent also has the responsiblity
to send samples to gnocchi backend through its gnocchi publisher and
events to panko backend.
Two gnocchi processes are introduced, gnocchi api is REST API service
and gnocchi metricd is for metric computing and gnocchi CLI is
supported to query measures/metric from gnocchi backend. Ceilometer
service and user still need to be registered in keystone, but stop
creating endpoints. Gnocchi is a seperate user/service in keystone and
run on active controller licensing on port 8041

Changes in this commit:
- add uid gid for telemetry services
- remove the oslo cache configuration
- fix puppet gnocchi warning
- remove ceilometer and add gnocchi logs in syslog

Story: 2002825
Task: 22871
Depends-On: https://review.openstack.org/587279

Change-Id: I3294927c676bee771e719bb581e00266f161ec86
Signed-off-by: Don Penney <don.penney@windriver.com>
Signed-off-by: Jack Ding <jack.ding@windriver.com>
This commit is contained in:
Angie Wang 2018-06-13 01:21:21 -04:00 committed by Jack Ding
parent 53d3777a35
commit 27f421a078
19 changed files with 326 additions and 20 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
base/setup/centos/meta_patches/PATCH_ORDER
View File

@ -12,3 +12,4 @@ spec-include-add-fm-user-to-snmpd-group.patch
spec-add-magnum-uid-gid.patch
spec-add-ironic-uid-gid.patch
spec-add-murano-uid-gid.patch
spec-add-uid-gid-for-telemetry-services.patch

32
base/setup/centos/meta_patches/spec-add-uid-gid-for-telemetry-services.patch Normal file
View File

@ -0,0 +1,32 @@
From 8c24c1c8bc9b703714c52b9b45cd3ea90f4a6604 Mon Sep 17 00:00:00 2001
From: Angie Wang <angie.Wang@windriver.com>
Date: Thu, 7 Jun 2018 16:39:08 -0400
Subject: [PATCH] spec add uid gid for telemetry services
---
SPECS/setup.spec | 2 ++
1 file changed, 2 insertions(+)
diff --git a/SPECS/setup.spec b/SPECS/setup.spec
index b652b3c..76d4feb 100644
--- a/SPECS/setup.spec
+++ b/SPECS/setup.spec
@@ -30,6 +30,7 @@ Patch11: add-fm-user-to-snmpd-group.patch
Patch12: add-magnum-uid-gid.patch
Patch13: add-ironic-uid-gid.patch
Patch14: add-murano-uid-gid.patch
+Patch15: add-uid-gid-for-telemetry-services.patch
%description
The setup package contains a set of important system configuration and
@@ -52,6 +53,7 @@ setup files, such as passwd, group, and profile.
%patch12 -p1
%patch13 -p1
%patch14 -p1
+%patch15 -p1
./shadowconvert.sh
--
1.8.3.1

35
base/setup/centos/patches/add-uid-gid-for-telemetry-services.patch Normal file
View File

@ -0,0 +1,35 @@
From 5d0869348945228a40c60fc5316d53b1e93e8bce Mon Sep 17 00:00:00 2001
From: Angie Wang <angie.Wang@windriver.com>
Date: Thu, 7 Jun 2018 16:35:19 -0400
Subject: [PATCH] add uid gid for telemetry services
---
group | 3 +++
passwd | 3 +++
2 files changed, 6 insertions(+)
diff --git a/group b/group
index 22fa91d..551b1e4 100644
--- a/group
+++ b/group
@@ -29,3 +29,6 @@ libvirt:x:991:nova
magnum:x:1870:magnum
ironic:x:1874:ironic
murano:x:105:murano
+aodh:x:992:aodh
+gnocchi:x:990:gnocchi
+panko:x:989:panko,ceilometer
diff --git a/passwd b/passwd
index dfb96b7..bfd8ee0 100644
--- a/passwd
+++ b/passwd
@@ -19,3 +19,6 @@ fm:x:195:195:fm-mgr:/var/lib/fm:/sbin/nologin
magnum:x:1870:1870:OpenStack Magnum Daemons:/var/lib/magnum:/sbin/nologin
ironic:x:1874:1874:OpenStack Ironic Daemons:/var/lib/ironic:/sbin/nologin
murano:x:105:105:OpenStack Murano Daemons:/home/murano:/sbin/nologin
+aodh:x:997:992:OpenStack aodh Daemons:/var/lib/aodh:/sbin/nologin
+gnocchi:x:996:990:gnocchi Daemons:/var/lib/gnocchi:/sbin/nologin
+panko:x:995:989:OpenStack panko Daemons:/var/lib/panko:/sbin/nologin
--
1.8.3.1

2
centos_pkg_dirs
View File

@ -24,6 +24,7 @@ extended/lighttpd
extended/python-cephclient
extended/python-ryu
extended/python-smartpm
extended/python-voluptuous
extended/lldpd
extended/logrotate
extended/nova-utils
@ -76,6 +77,7 @@ devtools/puppet-modules/openstack/puppet-ceilometer-11.3.0
devtools/puppet-modules/openstack/puppet-ceph-2.2.0
devtools/puppet-modules/openstack/puppet-cinder-11.3.0
devtools/puppet-modules/openstack/puppet-glance-11.3.0
devtools/puppet-modules/openstack/puppet-gnocchi-11.3.0
devtools/puppet-modules/openstack/puppet-heat-11.3.0
devtools/puppet-modules/openstack/puppet-ironic-11.3.0
devtools/puppet-modules/openstack/puppet-keystone-11.3.0

32
devtools/puppet-modules/openstack/puppet-ceilometer-11.3.0/centos/meta_patches/0005-meta-remove-the-oslo-cache-configuration.patch Normal file
View File

@ -0,0 +1,32 @@
From 0a5156ce7ebf44b0013dfb3f9ef3c27c000e91f1 Mon Sep 17 00:00:00 2001
From: Angie Wang <angie.Wang@windriver.com>
Date: Mon, 4 Jun 2018 10:48:56 -0400
Subject: [PATCH 1/1] meta remove the oslo cache configuration
---
SPECS/puppet-ceilometer.spec | 2 ++
1 file changed, 2 insertions(+)
diff --git a/SPECS/puppet-ceilometer.spec b/SPECS/puppet-ceilometer.spec
index 9f291f9..df9d728 100644
--- a/SPECS/puppet-ceilometer.spec
+++ b/SPECS/puppet-ceilometer.spec
@@ -12,6 +12,7 @@ Source0: https://tarballs.openstack.org/%{name}/%{name}-%{upstream_versio
Patch0001: 0001-Roll-up-TIS-patches.patch
Patch0002: 0002-Fix-ceilometer-puppet-warnings.patch
Patch0003: 0003-remove-reseller-admin-role.patch
+Patch0004: 0004-remove-the-oslo-cache-configuration.patch
BuildArch: noarch
@@ -31,6 +32,7 @@ Puppet module for OpenStack Ceilometer
%patch0001 -p1
%patch0002 -p1
%patch0003 -p1
+%patch0004 -p1
find . -type f -name ".*" -exec rm {} +
find . -size 0 -exec rm {} +
--
1.8.3.1

1
devtools/puppet-modules/openstack/puppet-ceilometer-11.3.0/centos/meta_patches/PATCH_ORDER
View File

@ -2,3 +2,4 @@
0002-Add-Rebase-Patches.patch
0003-meta-fix-ceilometer-puppet-warnings.patch
0004-meta-remove-reseller-admin-role.patch
0005-meta-remove-the-oslo-cache-configuration.patch

25
devtools/puppet-modules/openstack/puppet-ceilometer-11.3.0/centos/patches/0004-remove-the-oslo-cache-configuration.patch Normal file
View File

@ -0,0 +1,25 @@
From 26ea115cb4af960dce65b9580178d69a99129448 Mon Sep 17 00:00:00 2001
From: Angie Wang <angie.Wang@windriver.com>
Date: Mon, 4 Jun 2018 10:45:48 -0400
Subject: [PATCH 1/1] remove the oslo cache configuration
---
manifests/init.pp | 4 ----
1 file changed, 4 deletions(-)
diff --git a/manifests/init.pp b/manifests/init.pp
index 9c48627..3598b73 100644
--- a/manifests/init.pp
+++ b/manifests/init.pp
@@ -453,8 +453,4 @@ please use memcache_servers instead.")
rpc_response_timeout => $rpc_response_timeout,
control_exchange => $control_exchange,
}
-
- oslo::cache { 'ceilometer_config':
- memcache_servers => $memcache_servers_real,
- }
}
--
1.8.3.1

1
devtools/puppet-modules/openstack/puppet-gnocchi-11.3.0/centos/build_srpm.data Normal file
View File

@ -0,0 +1 @@
TIS_PATCH_VER=1

43
devtools/puppet-modules/openstack/puppet-gnocchi-11.3.0/centos/meta_patches/0001-meta-fix-gnocchi-puppet-warning.patch Normal file
View File

@ -0,0 +1,43 @@
From 98e5c184bb31839bf32fe5b3b90efa6ec38e4df4 Mon Sep 17 00:00:00 2001
From: Angie Wang <Angie.Wang@windriver.com>
Date: Mon, 28 May 2018 17:37:43 -0400
Subject: [PATCH 1/1] meta fix gnocchi puppet warning
---
SPECS/puppet-gnocchi.spec | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/SPECS/puppet-gnocchi.spec b/SPECS/puppet-gnocchi.spec
index 4645ede..fb54ef3 100644
--- a/SPECS/puppet-gnocchi.spec
+++ b/SPECS/puppet-gnocchi.spec
@@ -1,7 +1,7 @@
%{!?upstream_version: %global upstream_version %{version}%{?milestone}}
Name: puppet-gnocchi
Version: 11.3.0
-Release: 1%{?dist}
+Release: 1.el7%{?_tis_dist}.%{tis_patch_ver}
Summary: Puppet module for OpenStack Gnocchi
License: ASL 2.0
@@ -9,6 +9,9 @@ URL: https://launchpad.net/puppet-gnocchi
Source0: https://tarballs.openstack.org/%{name}/%{name}-%{upstream_version}.tar.gz
+# WRS
+Patch0001: 0001-fix-puppet-gnocchi-warning.patch
+
BuildArch: noarch
Requires: puppet-inifile
@@ -22,6 +25,7 @@ Puppet module for OpenStack Gnocchi
%prep
%setup -q -n openstack-gnocchi-%{upstream_version}
+%patch0001 -p1
find . -type f -name ".*" -exec rm {} +
find . -size 0 -exec rm {} +
--
1.8.3.1

1
devtools/puppet-modules/openstack/puppet-gnocchi-11.3.0/centos/meta_patches/PATCH_ORDER Normal file
View File

@ -0,0 +1 @@
0001-meta-fix-gnocchi-puppet-warning.patch

45
devtools/puppet-modules/openstack/puppet-gnocchi-11.3.0/centos/patches/0001-fix-puppet-gnocchi-warning.patch Normal file
View File

@ -0,0 +1,45 @@
From 9a2fa13f3afc543ec0dcf1d309df094cf9b429b9 Mon Sep 17 00:00:00 2001
From: Angie Wang <Angie.Wang@windriver.com>
Date: Tue, 29 May 2018 12:53:58 -0400
Subject: [PATCH 1/1] fix puppet gnocchi warning
---
manifests/logging.pp | 22 +++++++++++-----------
1 file changed, 11 insertions(+), 11 deletions(-)
diff --git a/manifests/logging.pp b/manifests/logging.pp
index c192c9c..e0163c7 100644
--- a/manifests/logging.pp
+++ b/manifests/logging.pp
@@ -95,17 +95,17 @@ class gnocchi::logging(
$log_dir = '/var/log/gnocchi',
$debug = $::os_service_default,
# DEPRECATED
- $logging_context_format_string = $::os_service_default,
- $logging_default_format_string = $::os_service_default,
- $logging_debug_format_suffix = $::os_service_default,
- $logging_exception_prefix = $::os_service_default,
- $log_config_append = $::os_service_default,
- $default_log_levels = $::os_service_default,
- $publish_errors = $::os_service_default,
- $fatal_deprecations = $::os_service_default,
- $instance_format = $::os_service_default,
- $instance_uuid_format = $::os_service_default,
- $log_date_format = $::os_service_default,
+ $logging_context_format_string = undef,
+ $logging_default_format_string = undef,
+ $logging_debug_format_suffix = undef,
+ $logging_exception_prefix = undef,
+ $log_config_append = undef,
+ $default_log_levels = undef,
+ $publish_errors = undef,
+ $fatal_deprecations = undef,
+ $instance_format = undef,
+ $instance_uuid_format = undef,
+ $log_date_format = undef,
) {
include ::gnocchi::deps
--
1.8.3.1

1
devtools/puppet-modules/openstack/puppet-gnocchi-11.3.0/centos/srpm_path Normal file
View File

@ -0,0 +1 @@
mirror:Source/puppet-gnocchi-11.3.0-1.el7.src.rpm

1
extended/python-voluptuous/centos/build_srpm.data Normal file
View File

@ -0,0 +1 @@
TIS_PATCH_VER=1

44
extended/python-voluptuous/centos/meta_patches/0001-meta-add-NotIn-validation.patch Normal file
View File

@ -0,0 +1,44 @@
From a64cf29a67df13212e173c4aa74fe9de538ea97d Mon Sep 17 00:00:00 2001
From: Angie Wang <Angie.Wang@windriver.com>
Date: Mon, 28 May 2018 20:18:19 -0400
Subject: [PATCH 1/1] meta add NotIn validation
---
SPECS/python-voluptuous.spec | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/SPECS/python-voluptuous.spec b/SPECS/python-voluptuous.spec
index 2b1217c..9f82511 100644
--- a/SPECS/python-voluptuous.spec
+++ b/SPECS/python-voluptuous.spec
@@ -6,7 +6,7 @@
Name: python-%{upname}
Version: 0.8.9
-Release: 1%{?dist}
+Release: 1.el7%{?_tis_dist}.%{tis_patch_ver}
Summary: A Python data validation library
License: BSD
Group: Development/Languages
@@ -15,6 +15,10 @@ URL: http://github.com/alecthomas/voluptuous
# Use the tarball from github instead
#Source0: http://pypi.python.org/packages/source/v/voluptuous/%{upname}-%{version}.tar.gz
Source0: http://github.com/alecthomas/voluptuous/archive/%{version}.tar.gz#/%{upname}-%{version}.tar.gz
+
+# WRS
+Patch0001: 0001-add-NotIn-validation.patch
+
BuildRequires: python2-devel python-setuptools
BuildArch: noarch
@@ -38,6 +42,7 @@ primarily intended for validating data coming into Python as JSON, YAML, etc.
%prep
%setup -q -n %{upname}-%{version}
+%patch0001 -p1
rm -rf %{upname}.egg-info
%if 0%{?with_python3}
--
1.8.3.1

1
extended/python-voluptuous/centos/meta_patches/PATCH_ORDER Normal file
View File

@ -0,0 +1 @@
0001-meta-add-NotIn-validation.patch

47
extended/python-voluptuous/centos/patches/0001-add-NotIn-validation.patch Normal file
View File

@ -0,0 +1,47 @@
From afc63cafd4388310f865fb0fba3165a228796049 Mon Sep 17 00:00:00 2001
From: Angie Wang <Angie.Wang@windriver.com>
Date: Mon, 28 May 2018 20:12:43 -0400
Subject: [PATCH 1/1] add NotIn validation
---
voluptuous.py | 24 ++++++++++++++++++++++++
1 file changed, 24 insertions(+)
diff --git a/voluptuous.py b/voluptuous.py
index 5b561b1..b4ed666 100644
--- a/voluptuous.py
+++ b/voluptuous.py
@@ -1584,6 +1584,30 @@ def In(container, msg=None):
return validator
+class NotInInvalid(Invalid):
+ pass
+
+
+class NotIn(object):
+ """Validate that a value is not in a collection."""
+
+ def __init__(self, container, msg=None):
+ self.container = container
+ self.msg = msg
+
+ def __call__(self, v):
+ try:
+ check = v in self.container
+ except TypeError:
+ check = True
+ if check:
+ raise NotInInvalid(self.msg or 'value is not allowed')
+ return v
+
+ def __repr__(self):
+ return 'NotIn(%s)' % (self.container,)
+
+
def Lower(v):
"""Transform a string to lower case.
--
1.8.3.1

1
extended/python-voluptuous/centos/srpm_path Normal file
View File

@ -0,0 +1 @@
mirror:Source/python-voluptuous-0.8.9-1.el7.src.rpm

32
extended/syslog-ng/centos/files/syslog-ng.conf
View File

@ -176,12 +176,8 @@ destination d_glanceapi { file("/var/log/glance/glance-api.log" template(t
destination d_glanceregistry { file("/var/log/glance/glance-registry.log" template(t_openstack)); };
destination d_glanceregistryrest { file("/var/log/glance/glance-registry-api.log" template(t_openstack)); };
# Ceilometer is not using oslo_log so it does not use the t_openstack template
destination d_ceilometeragentcentral { file("/var/log/ceilometer/ceilometer-agent-central.log"); };
destination d_ceilometeragentnotification { file("/var/log/ceilometer/ceilometer-agent-notification.log"); };
destination d_ceilometeralarmevaluator { file("/var/log/ceilometer/ceilometer-alarm-evaluator.log"); };
destination d_ceilometeralarmnotifier { file("/var/log/ceilometer/ceilometer-alarm-notifier.log"); };
destination d_ceilometerapi { file("/var/log/ceilometer/ceilometer-api.log"); };
destination d_ceilometercollector { file("/var/log/ceilometer/ceilometer-collector.log"); };
destination d_ceilometerupgrade { file("/var/log/ceilometer/ceilometer-upgrade.log"); };
destination d_aodhapi { file("/var/log/aodh/aodh-api.log" template(t_openstack)); };
destination d_aodhdbsync { file("/var/log/aodh/aodh-dbsync.log" template(t_openstack)); };
destination d_aodhevaluator { file("/var/log/aodh/aodh-evaluator.log" template(t_openstack)); };
@ -197,6 +193,8 @@ destination d_ironicconductor { file("/var/log/ironic/ironic-conductor.log" temp
destination d_pankoapi { file("/var/log/panko/panko-api.log" template(t_openstack)); };
destination d_pankodbsync { file("/var/log/panko/panko-dbsync.log" template(t_openstack)); };
destination d_pankoexpirer { file("/var/log/panko/panko-expirer.log" template(t_openstack)); };
destination d_gnocchiapi { file("/var/log/gnocchi/api.log" template(t_openstack)); };
destination d_gnocchimetricd { file("/var/log/gnocchi/metricd.log" template(t_openstack)); };
# NFV-VIM Log destinations
destination d_vim { file("/var/log/nfv-vim.log" template(t_nfv)); };
@ -307,12 +305,8 @@ filter f_local1 { facility(local1)
filter f_local2 { facility(local2)
and not program(glance-api)
and not program(glance-registry)
and not program(ceilometer-agent-central)
and not program(ceilometer-agent-notification)
and not program(ceilometer-alarm-evaluator)
and not program(ceilometer-alarm-notifier)
and not program(ceilometer-api)
and not program(ceilometer-collector)
and not program(ceilometer-upgrade)
and not program(aodh-api)
and not program(aodh-dbsync)
and not program(aodh-evaluator)
@ -322,6 +316,8 @@ filter f_local2 { facility(local2)
and not program(panko-api)
and not program(panko-dbsync)
and not program(panko-expirer)
and not program(gnocchi-api)
and not program(gnocchi-metricd)
and not program(murano-api)
and not program(murano-engine)
and not program(magnum-api)
@ -405,12 +401,8 @@ filter f_glanceregistry { facility(local2) and program(glance-registry) and not
filter f_glanceregistryrest { facility(local2) and program(glance-registry) and match("eventlet.wsgi.server"); };
filter f_keystoneall { facility(local2) and message("keystone.*") and not match("keystone.common.wsgi"); };
filter f_keystoneapi { facility(local2) and match("keystone.common.wsgi"); };
filter f_ceilometeragentcentral { facility(local2) and program(ceilometer-agent-central); };
filter f_ceilometeragentnotification { facility(local2) and program(ceilometer-agent-notification); };
filter f_ceilometeralarmevaluator { facility(local2) and program(ceilometer-alarm-evaluator); };
filter f_ceilometeralarmnotifier { facility(local2) and program(ceilometer-alarm-notifier); };
filter f_ceilometerapi { facility(local2) and program(ceilometer-api); };
filter f_ceilometercollector { facility(local2) and program(ceilometer-collector); };
filter f_ceilometerupgrade { facility(local2) and program(ceilometer-upgrade); };
filter f_aodhapi { facility(local2) and program(aodh-api); };
filter f_aodhdbsync { facility(local2) and program(aodh-dbsync); };
filter f_aodhevaluator { facility(local2) and program(aodh-evaluator); };
@ -426,6 +418,8 @@ filter f_pankodbsync { facility(local2) and program(panko-dbsync); };
filter f_pankoexpirer { facility(local2) and program(panko-expirer); };
filter f_ironicapi { facility(local2) and program(ironic-api); };
filter f_ironicconductor { facility(local2) and program(ironic-conductor); };
filter f_gnocchiapi { facility(local2) and program(gnocchi-api); };
filter f_gnocchimetricd { facility(local2) and program(gnocchi-metricd); };
# NFV-VIM Log Filters
filter f_vim { facility(user) and program(VIM_); };
@ -525,12 +519,8 @@ log { source(s_src); filter(f_keystoneapi); destination(d_keystoneapi); };
log { source(s_src); filter(f_glanceapi); destination(d_glanceapi); };
log { source(s_src); filter(f_glanceregistry); destination(d_glanceregistry); };
log { source(s_src); filter(f_glanceregistryrest); destination(d_glanceregistryrest); };
log { source(s_src); filter(f_ceilometeragentcentral); destination(d_ceilometeragentcentral); };
log { source(s_src); filter(f_ceilometeragentnotification); destination(d_ceilometeragentnotification); };
log { source(s_src); filter(f_ceilometeralarmevaluator); destination(d_ceilometeralarmevaluator); };
log { source(s_src); filter(f_ceilometeralarmnotifier); destination(d_ceilometeralarmnotifier); };
log { source(s_src); filter(f_ceilometerapi); destination(d_ceilometerapi); };
log { source(s_src); filter(f_ceilometercollector); destination(d_ceilometercollector); };
log { source(s_src); filter(f_ceilometerupgrade); destination(d_ceilometerupgrade); };
log { source(s_src); filter(f_aodhapi); destination(d_aodhapi); };
log { source(s_src); filter(f_aodhdbsync); destination(d_aodhdbsync); };
log { source(s_src); filter(f_aodhevaluator); destination(d_aodhevaluator); };
@ -546,6 +536,8 @@ log { source(s_src); filter(f_pankodbsync); destination(d_pankodbsync); };
log { source(s_src); filter(f_pankoexpirer); destination(d_pankoexpirer); };
log { source(s_src); filter(f_ironicapi); destination(d_ironicapi); };
log { source(s_src); filter(f_ironicconductor); destination(d_ironicconductor); };
log { source(s_src); filter(f_gnocchiapi); destination(d_gnocchiapi); };
log { source(s_src); filter(f_gnocchimetricd); destination(d_gnocchimetricd); };
# NFV-VIM Log Paths
log {source(s_src); filter(f_vim); destination(d_vim); };

1
extended/syslog-ng/centos/files/syslog-ng.logrotate
View File

@ -28,6 +28,7 @@
/var/log/horizon.log
/var/log/ceilometer/*.log
/var/log/panko/*.log
/var/log/gnocchi/*.log
/var/log/ironic/*.log
/var/log/snmp-api.log
/var/log/dcmanager/*.log