airship/spyglass-plugin-xls
Code Issues Proposed changes

Initial commit

Browse Source 
Adds code for tugboat plugin from Airship Spyglass.
This commit is contained in:
Ian H. Pittwood 2019-04-26 16:01:24 -05:00
commit 6174c80f5d
20 changed files with 1750 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

116
.gitignore vendored Normal file
View File

@ -0,0 +1,116 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
env/
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
*.tgz
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
.hypothesis/
.testrepository/*
cover/*
results/*
.stestr/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# PyBuilder
target/
# Jupyter Notebook
.ipynb_checkpoints
# pyenv
.python-version
# celery beat schedule file
celerybeat-schedule
# SageMath parsed files
*.sage.py
# dotenv
.env
# virtualenv
.venv
venv/
ENV/
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
# mypy
.mypy_cache/
# pycharm-ide
.idea/
# osx
.DS_Store
# git
Changelog
AUTHORS
# Ansible
*.retry

18
.zuul.yaml Normal file
View File

@ -0,0 +1,18 @@
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
- project:
check:
jobs:
- openstack-tox-pep8
gate:
jobs:
- openstack-tox-pep8

201
LICENSE Normal file
View File

@ -0,0 +1,201 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

10
README.rst Normal file
View File

@ -0,0 +1,10 @@
Spyglass Excel Plugin
---------------------
Tugboat is a Spyglass plugin to generate airship-seaworthy site manifest files
from an excel based engineering spec. The plugin is configured with an Excel
sheet and its corresponding excel specification as inputs. Spyglass uses this
plugin to construct an intermediary yaml which is processed further using J2
templates to generate site manifests.
Spyglass: https://opendev.org/airship/spyglass

3
doc/requirements.txt Normal file
View File

@ -0,0 +1,3 @@
# Documentation
sphinx>=2.0.1
sphinx_rtd_theme==0.4.3

130
doc/source/conf.py Normal file
View File

@ -0,0 +1,130 @@
# -*- coding: utf-8 -*-
#
# shipyard documentation build configuration file, created by
# sphinx-quickstart on Sat Sep 16 03:40:50 2017.
#
# This file is execfile()d with the current directory set to its
# containing dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
import os
import sys
import sphinx_rtd_theme
sys.path.insert(0, os.path.abspath('../../'))
# -- General configuration ------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
#
# needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.todo',
'sphinx.ext.viewcode',
]
# Add any paths that contain templates here, relative to this directory.
# templates_path = []
# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
#
# source_suffix = ['.rst', '.md']
source_suffix = '.rst'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = u'spyglass-plugin-xls'
copyright = u'2019 AT&T Intellectual Property.'
author = u'The Airship Authors'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
version = u'0.0.1'
# The full version, including alpha/beta/rc tags.
release = u'0.0.1'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = None
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This patterns also effect to html_static_path and html_extra_path
exclude_patterns = []
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# If true, `todo` and `todoList` produce output, else they produce nothing.
todo_include_todos = False
# -- Options for HTML output ----------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = "sphinx_rtd_theme"
html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#
# html_theme_options = {}
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = []
# -- Options for HTMLHelp output ------------------------------------------
# Output file base name for HTML help builder.
htmlhelp_basename = 'ucpintdoc'
# -- Options for LaTeX output ---------------------------------------------
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
#
# 'papersize': 'letterpaper',
# The font size ('10pt', '11pt' or '12pt').
#
# 'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
#
# 'preamble': '',
# Latex figure (float) alignment
#
# 'figure_align': 'htbp',
}

96
doc/source/index.rst Normal file
View File

@ -0,0 +1,96 @@
..
Copyright 2018 AT&T Intellectual Property.
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
==============
Tugboat Plugin
==============
What is Tugboat Plugin?
-----------------------
Tugboat is a Spyglass plugin to generate airship-seaworthy site manifest files
from an excel based engineering spec. The plugin is configured with an Excel
sheet and its corresponding excel specification as inputs. Spyglass uses this
plugin to construct an intermediary yaml which is processed further using J2
templates to generate site manifests.
Excel specification
-------------------
Excel Spec is like an index to the Excel sheet to look for the data to be
collected by the tool. Excel Spec Sample specifies all the details that
need to be filled by the Deployment Engineer.
Below is the definition for each key in the Excel spec
::
ipmi_sheet_name - name of the sheet from where IPMI and host profile information is to be read
start_row - row number from where the IPMI and host profile information starts
end_row - row number from where the IPMI and host profile information ends
hostname_col - column number where the hostnames are to be read from
ipmi_address_col - column number from where the ipmi addresses are to be read
host_profile_col - column number from where the host profiles are to be read
ipmi_gateway_col - column number from where the ipmi gateways are to be read
private_ip_sheet - name of the sheet which has the private IP information
net_type_col - column number from where the network type is to be read
vlan_col - column number from where the network vlan is to be read
vlan_start_row - row number from where the vlan information starts
vlan_end_row - row number from where the vlan information ends
net_start_row - row number from where the network information starts
net_end_row - row number from where the network information ends
net_col - column number where the IP ranges for network is to be read
net_vlan_col - column number where the vlan information is present in the pod wise network section
public_ip_sheet - name of the sheet which has the public IP information
oam_vlan_col - column number from where the OAM vlan information is to be read from
oam_ip_row - row number from where the OAM network information is to be read from
oam_ip_col - column number from where the OAM network information is to be read from
oob_net_row - row number which has the OOB network subnet ranges
oob_net_start_col - column number from where the OOB network ranges start
oob_net_end_col - column number from where the OOB network ranges end
ingress_ip_row - row number from where the Ingress network information is to be read from
dns_ntp_ldap_sheet - name of the sheet which has the DNS, NTP and LDAP information
login_domain_row - row number which has the ldap login domain
ldap_col - column number which has the all ldap related information
global_group - row number which has the ldap group information
ldap_search_url_row - row number which has the ldap url
ntp_row - row number which has the ntp information
ntp_col - column number which has the ntp information
dns_row - row number which has the dns information
dns_col - column number which has the dns information
domain_row - row number which has the domain information
domain_col - column number which has the domain information
location_sheet - name of the sheet which has the location information
column - column number which has all the information
corridor_row - row number which has the corridor information
site_name_row - row number which has the site name
state_name_row - row number which has the state name
country_name_row - row number which has the country name
clli_name_row - row number which has CLLI information
Example: Tugboat Plugin Usage
-----------------------------
1. Required Input(Refer to 'spyglass/examples' folder to get these inputs)
a) Excel File: SiteDesignSpec_v0.1.xlsx
b) Excel Spec: excel_spec_upstream.yaml
c) Site Config: site_config.yaml
2. Spyglass CLI Command:
::
spyglass -mg -t tugboat -x SiteDesignSpec_v0.1.xlsx -e excel_spec_upstream.yaml -d site_config.yaml -s airship-seaworthy --template_dir=<relative path to '../examples/templates'

BIN
examples/SiteDesignSpec_v0.1.xlsx Normal file
View File

Binary file not shown.

63
examples/excel_spec.yaml Normal file
View File

@ -0,0 +1,63 @@
# Copyright 2018 The Openstack-Helm Authors.
# Copyright (c) 2018 AT&T Intellectual Property. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Important: Please modify the dictionary with appropriate
# design spec file.
---
specs:
# Design Spec file name: SiteDesignSpec_v0.1.xlsx
xl_spec:
ipmi_sheet_name: 'Site-Information'
start_row: 4
end_row: 15
hostname_col: 2
ipmi_address_col: 3
host_profile_col: 5
ipmi_gateway_col: 4
private_ip_sheet: 'Site-Information'
net_type_col: 1
vlan_col: 2
vlan_start_row: 19
vlan_end_row: 30
net_start_row: 33
net_end_row: 40
net_col: 2
net_vlan_col: 1
public_ip_sheet: 'Site-Information'
oam_vlan_col: 1
oam_ip_row: 43
oam_ip_col: 2
oob_net_row: 48
oob_net_start_col: 2
oob_net_end_col: 5
ingress_ip_row: 45
dns_ntp_ldap_sheet: 'Site-Information'
login_domain_row: 52
ldap_col: 2
global_group: 53
ldap_search_url_row: 54
ntp_row: 55
ntp_col: 2
dns_row: 56
dns_col: 2
domain_row: 51
domain_col: 2
location_sheet: 'Site-Information'
column: 2
corridor_row: 59
site_name_row: 58
state_name_row: 60
country_name_row: 61
clli_name_row: 62

33
examples/site_config.yaml Normal file
View File

@ -0,0 +1,33 @@
##################################
# Site Specific Tugboat Settings #
##################################
---
site_info:
ldap:
common_name: test
url: ldap://ldap.example.com
subdomain: test
ntp:
servers: 10.10.10.10,20.20.20.20,30.30.30.30
sitetype: foundry
domain: atlantafoundry.com
dns:
servers: 8.8.8.8,8.8.4.4,208.67.222.222
network:
vlan_network_data:
ingress:
subnet:
- 132.68.226.72/29
bgp :
peers:
- '172.29.0.2'
- '172.29.0.3'
asnumber: 64671
peer_asnumber: 64688
storage:
ceph:
controller:
osd_count: 6
...

4
requirements.txt Normal file
View File

@ -0,0 +1,4 @@
click==7.0
PyYAML==3.12
openpyxl==2.5.4
git+https://opendev.org/airship/spyglass.git

34
setup.py Normal file
View File

@ -0,0 +1,34 @@
# Copyright 2018 AT&T Intellectual Property. All other rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from setuptools import setup
from setuptools import find_packages
setup(
name='spyglass_plugin_xls',
version='0.0.1',
description='Excel data source plugin for Spyglass',
url='https://opendev.org/airship/spyglass-plugin-xls',
python_requires='>=3.5.0',
license='Apache 2.0',
packages=find_packages(),
install_requires=[
'PyYAML==3.12',
'openpyxl==2.5.4',
],
dependency_links=[
'git+https://opendev.org/airship/spyglass.git',
],
include_package_data=True,
)

1
spyglass-plugin-xls/__init__.py Normal file
View File

@ -0,0 +1 @@
__all__ = ['excel.py']

35
spyglass-plugin-xls/check_exceptions.py Normal file
View File

@ -0,0 +1,35 @@
# Copyright 2018 AT&T Intellectual Property. All other rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
class BaseError(Exception):
pass
class NotEnoughIp(BaseError):
def __init__(self, cidr, total_nodes):
self.cidr = cidr
self.total_nodes = total_nodes
def display_error(self):
print("{} can not handle {} nodes".format(self.cidr, self.total_nodes))
class NoSpecMatched(BaseError):
def __init__(self, excel_specs):
self.specs = excel_specs
def display_error(self):
print("No spec matched. Following are the available specs:\n".format(
self.specs))

151
spyglass-plugin-xls/cli.py Normal file
View File

@ -0,0 +1,151 @@
# Copyright 2018 AT&T Intellectual Property. All other rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import logging
import click
import spyglass
from spyglass.site_processors.site_processor import SiteProcessor
LOG = logging.getLogger(__name__)
LOG_FORMAT = '%(asctime)s %(levelname)-8s %(name)s:' \
'%(funcName)s [%(lineno)3d] %(message)s'
@click.option('-v',
'--verbose',
is_flag=True,
default=False,
help='Enable debug messages in log.')
@click.group()
def excel(*, verbose):
"""Plugin for extracting site data from Excel spreadsheets"""
if verbose:
log_level = logging.DEBUG
else:
log_level = logging.INFO
logging.basicConfig(format=LOG_FORMAT, level=log_level)
EXCEL_FILE_OPTION = click.option(
'-x',
'--excel-file',
'excel_file',
multiple=True,
type=click.Path(exists=True, readable=True, dir_okay=False),
required=True,
help='Path to the engineering Excel file. Required for '
'spyglass-plugin-xls plugin.',
)
EXCEL_SPEC_OPTION = click.option(
'-e',
'--excel-spec',
'excel_spec',
type=click.Path(exists=True, readable=True, dir_okay=False),
required=True,
help='Path to the Excel specification YAML file for the engineering '
'Excel file. Required for spyglass-plugin-xls plugin.',
)
SITE_CONFIGURATION_FILE_OPTION = click.option(
'-c',
'--site-configuration',
'site_configuration',
type=click.Path(exists=True, readable=True, dir_okay=False),
required=False,
help='Path to site specific configuration details YAML file.')
INTERMEDIARY_DIR_OPTION = click.option(
'-idir',
'--intermediary-dir',
'intermediary_dir',
type=click.Path(exists=True, file_okay=False, writable=True),
default='./',
help='Directory in which the intermediary file will be created.')
SITE_NAME_CONFIGURATION_OPTION = click.option(
'-s',
'--site-name',
'site_name',
type=click.STRING,
required=False,
help='Name of the site for which the intermediary is being generated.')
TEMPLATE_DIR_OPTION = click.option(
'-tdir',
'--template-dir',
'template_dir',
type=click.Path(exists=True, readable=True, file_okay=False),
required=True,
help='Path to the directory containing manifest J2 templates.')
MANIFEST_DIR_OPTION = click.option(
'-mdir',
'--manifest-dir',
'manifest_dir',
type=click.Path(exists=True, writable=True, file_okay=False),
required=False,
help='Path to place created manifest files.')
@excel.command('g',
short_help='generate intermediary',
help='Generates an intermediary file from passed excel data.')
@EXCEL_FILE_OPTION
@EXCEL_SPEC_OPTION
@SITE_CONFIGURATION_FILE_OPTION
@INTERMEDIARY_DIR_OPTION
@SITE_NAME_CONFIGURATION_OPTION
def generate_intermediary(*args, **kwargs):
process_input_ob = \
spyglass.cli.intermediary_processor('spyglass-plugin-xls', **kwargs)
LOG.info("Generate intermediary yaml")
process_input_ob.generate_intermediary_yaml()
process_input_ob.dump_intermediary_file(kwargs['intermediary_dir'])
@excel.command('m',
short_help='generates manifest and intermediary',
help='Generates manifest and intermediary files.')
@click.option(
'-g',
'--generate-intermediary',
'generate_intermediary',
is_flag=True,
default=False,
help='Flag to save the generated intermediary file used for the manifests.'
)
@EXCEL_FILE_OPTION
@EXCEL_SPEC_OPTION
@SITE_CONFIGURATION_FILE_OPTION
@INTERMEDIARY_DIR_OPTION
@SITE_NAME_CONFIGURATION_OPTION
@TEMPLATE_DIR_OPTION
@MANIFEST_DIR_OPTION
def generate_manifests_and_intermediary(*args, **kwargs):
process_input_ob = \
spyglass.cli.intermediary_processor('spyglass-plugin-xls', **kwargs)
LOG.info("Generate intermediary yaml")
intermediary_yaml = process_input_ob.generate_intermediary_yaml()
if generate_intermediary:
LOG.debug("Dumping intermediary yaml")
process_input_ob.dump_intermediary_file(kwargs['intermediary_dir'])
else:
LOG.debug("Skipping dump for intermediary yaml")
LOG.info("Generating site Manifests")
processor_engine = SiteProcessor(intermediary_yaml, kwargs['manifest_dir'])
processor_engine.render_template(kwargs['template_dir'])

358
spyglass-plugin-xls/excel.py Normal file
View File

@ -0,0 +1,358 @@
# Copyright 2018 AT&T Intellectual Property. All other rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import itertools
import logging
import pprint
import re
from spyglass.data_extractor.base import BaseDataSourcePlugin
from spyglass_plugin_xls.excel_parser import ExcelParser
LOG = logging.getLogger(__name__)
class ExcelPlugin(BaseDataSourcePlugin):
def __init__(self, region):
LOG.info("Excel Plugin Initializing")
self.source_type = "excel"
self.source_name = "spyglass-plugin-xls"
# Configuration parameters
self.excel_path = None
self.excel_spec = None
# Site related data
self.region = region
# Raw data from excel
self.parsed_xl_data = None
LOG.info("Initiated data extractor plugin:{}".format(self.source_name))
def set_config_opts(self, conf):
"""Placeholder to set configuration options specific to each plugin.
:param dict conf: Configuration options as dict
Example: conf = { 'excel_spec': 'spec1.yaml',
'excel_path': 'excel.xls' }
Each plugin will have their own config opts.
"""
self.excel_path = conf["excel_path"]
self.excel_spec = conf["excel_spec"]
# Extract raw data from excel sheets
self._get_excel_obj()
self._extract_raw_data_from_excel()
return
def get_plugin_conf(self, **kwargs):
"""Validates the plugin param from CLI and return if correct
Ideally the CLICK module shall report an error if excel file
and excel specs are not specified. The below code has been
written as an additional safeguard.
"""
if not kwargs["excel_file"]:
LOG.error("Engineering excel file not specified: Spyglass exited!")
exit()
excel_file_info = kwargs["excel_file"]
if not kwargs["excel_spec"]:
LOG.error("Engineering spec file not specified: Spyglass exited!")
exit()
excel_spec_info = kwargs["excel_spec"]
plugin_conf = {
"excel_path": excel_file_info,
"excel_spec": excel_spec_info,
}
return plugin_conf
def get_hosts(self, region, rack=None):
"""Return list of hosts in the region
:param string region: Region name
:param string rack: Rack name
:returns: list of hosts information
:rtype: list of dict
Example: [
{
'name': 'host01',
'type': 'controller',
'host_profile': 'hp_01'
},
{
'name': 'host02',
'type': 'compute',
'host_profile': 'hp_02'}
]
"""
LOG.info("Get Host Information")
ipmi_data = self.parsed_xl_data["ipmi_data"][0]
rackwise_hosts = self._get_rackwise_hosts()
host_list = []
for rack in rackwise_hosts.keys():
for host in rackwise_hosts[rack]:
host_list.append({
"rack_name":
rack,
"name":
host,
"host_profile":
ipmi_data[host]["host_profile"],
})
return host_list
def get_networks(self, region):
"""Extracts vlan network info from raw network data from excel"""
vlan_list = []
# Network data extracted from xl is formatted to have a predictable
# data type. For e.g VlAN 45 extracted from xl is formatted as 45
vlan_pattern = r"\d+"
private_net = self.parsed_xl_data["network_data"]["private"]
public_net = self.parsed_xl_data["network_data"]["public"]
# Extract network information from private and public network data
for net_type, net_val in itertools.chain(private_net.items(),
public_net.items()):
tmp_vlan = {}
# Ingress is special network that has no vlan, only a subnet string
# So treatment for ingress is different
if net_type != "ingress":
# standardize the network name as net_type may ne different.
# For e.g insteas of pxe it may be PXE or instead of calico
# it may be ksn. Valid network names are pxe, calico, oob, oam,
# overlay, storage, ingress
tmp_vlan["name"] = self._get_network_name_from_vlan_name(
net_type)
# extract vlan tag. It was extracted from xl file as 'VlAN 45'
# The code below extracts the numeric data fron net_val['vlan']
if net_val.get("vlan", "") != "":
value = re.findall(vlan_pattern, net_val["vlan"])
tmp_vlan["vlan"] = value[0]
else:
tmp_vlan["vlan"] = "#CHANGE_ME"
tmp_vlan["subnet"] = net_val.get("subnet", "#CHANGE_ME")
tmp_vlan["gateway"] = net_val.get("gateway", "#CHANGE_ME")
else:
tmp_vlan["name"] = "ingress"
tmp_vlan["subnet"] = net_val
vlan_list.append(tmp_vlan)
LOG.debug("vlan list extracted from spyglass-plugin-xls:\n{}".format(
pprint.pformat(vlan_list)))
return vlan_list
def get_ips(self, region, host=None):
"""Return list of IPs on the host
:param string region: Region name
:param string host: Host name
:returns: Dict of IPs per network on the host
:rtype: dict
Example: {'oob': {'ipv4': '192.168.1.10'},
'pxe': {'ipv4': '192.168.2.10'}}
The network name from get_networks is expected to be the keys of this
dict. In case some networks are missed, they are expected to be either
DHCP or internally generated n the next steps by the design rules.
"""
ip_ = {}
ipmi_data = self.parsed_xl_data["ipmi_data"][0]
ip_[host] = {
"oob": ipmi_data[host].get("ipmi_address", "#CHANGE_ME"),
"oam": ipmi_data[host].get("oam", "#CHANGE_ME"),
"calico": ipmi_data[host].get("calico", "#CHANGE_ME"),
"overlay": ipmi_data[host].get("overlay", "#CHANGE_ME"),
"pxe": ipmi_data[host].get("pxe", "#CHANGE_ME"),
"storage": ipmi_data[host].get("storage", "#CHANGE_ME"),
}
return ip_
def get_ldap_information(self, region):
"""Extract ldap information from excel"""
ldap_raw_data = self.parsed_xl_data["site_info"]["ldap"]
ldap_info = {}
# raw url is 'url: ldap://example.com' so we are converting to
# 'ldap://example.com'
url = ldap_raw_data.get("url", "#CHANGE_ME")
try:
ldap_info["url"] = url.split(" ")[1]
ldap_info["domain"] = url.split(".")[1]
except IndexError as e:
LOG.error("url.split:{}".format(e))
ldap_info["common_name"] = ldap_raw_data.get("common_name",
"#CHANGE_ME")
ldap_info["subdomain"] = ldap_raw_data.get("subdomain", "#CHANGE_ME")
return ldap_info
def get_ntp_servers(self, region):
"""Returns a comma separated list of ntp ip addresses"""
ntp_server_list = self._get_formatted_server_list(
self.parsed_xl_data["site_info"]["ntp"])
return ntp_server_list
def get_dns_servers(self, region):
"""Returns a comma separated list of dns ip addresses"""
dns_server_list = self._get_formatted_server_list(
self.parsed_xl_data["site_info"]["dns"])
return dns_server_list
def get_domain_name(self, region):
"""Returns domain name extracted from excel file"""
return self.parsed_xl_data["site_info"]["domain"]
def get_location_information(self, region):
"""Prepare location data from information extracted
by ExcelParser(i.e raw data)
"""
location_data = self.parsed_xl_data["site_info"]["location"]
corridor_pattern = r"\d+"
corridor_number = re.findall(corridor_pattern,
location_data["corridor"])[0]
name = location_data.get("name", "#CHANGE_ME")
state = location_data.get("state", "#CHANGE_ME")
country = location_data.get("country", "#CHANGE_ME")
physical_location_id = location_data.get("physical_location", "")
return {
"name": name,
"physical_location_id": physical_location_id,
"state": state,
"country": country,
"corridor": "c{}".format(corridor_number),
}
def get_racks(self, region):
# This function is not required since the excel plugin
# already provide rack information.
pass
def _get_excel_obj(self):
"""Creation of an ExcelParser object to store site information.
The information is obtained based on a excel spec yaml file.
This spec contains row, column and sheet information of
the excel file from where site specific data can be extracted.
"""
self.excel_obj = ExcelParser(self.excel_path, self.excel_spec)
def _extract_raw_data_from_excel(self):
"""Extracts raw information from excel file based on excel spec"""
self.parsed_xl_data = self.excel_obj.get_data()
def _get_network_name_from_vlan_name(self, vlan_name):
"""Network names are ksn, oam, oob, overlay, storage, pxe
This is a utility function to determine the vlan acceptable
vlan from the name extracted from excel file
The following mapping rules apply:
vlan_name contains "ksn or calico" the network name is "calico"
vlan_name contains "storage" the network name is "storage"
vlan_name contains "server" the network name is "oam"
vlan_name contains "ovs" the network name is "overlay"
vlan_name contains "oob" the network name is "oob"
vlan_name contains "pxe" the network name is "pxe"
"""
network_names = [
"ksn|calico",
"storage",
"oam|server",
"ovs|overlay",
"oob",
"pxe",
]
for name in network_names:
# Make a pattern that would ignore case.
# if name is 'ksn' pattern name is '(?i)(ksn)'
name_pattern = "(?i)({})".format(name)
if re.search(name_pattern, vlan_name):
if name == "ksn|calico":
return "calico"
if name == "storage":
return "storage"
if name == "oam|server":
return "oam"
if name == "ovs|overlay":
return "overlay"
if name == "oob":
return "oob"
if name == "pxe":
return "pxe"
# if nothing matches
LOG.error(
"Unable to recognize VLAN name extracted from Plugin data source")
return ""
def _get_formatted_server_list(self, server_list):
"""Format dns and ntp server list as comma separated string"""
# dns/ntp server info from excel is of the format
# 'xxx.xxx.xxx.xxx, (aaa.bbb.ccc.com)'
# The function returns a list of comma separated dns ip addresses
servers = []
for data in server_list:
if "(" not in data:
servers.append(data)
formatted_server_list = ",".join(servers)
return formatted_server_list
def _get_rack(self, host):
"""Get rack id from the rack string extracted from xl"""
rack_pattern = r"\w.*(r\d+)\w.*"
rack = re.findall(rack_pattern, host)[0]
if not self.region:
self.region = host.split(rack)[0]
return rack
def _get_rackwise_hosts(self):
"""Mapping hosts with rack ids"""
rackwise_hosts = {}
hostnames = self.parsed_xl_data["ipmi_data"][1]
racks = self._get_rack_data()
for rack in racks:
if rack not in rackwise_hosts:
rackwise_hosts[racks[rack]] = []
for host in hostnames:
if rack in host:
rackwise_hosts[racks[rack]].append(host)
LOG.debug("rackwise hosts:\n%s", pprint.pformat(rackwise_hosts))
return rackwise_hosts
def _get_rack_data(self):
"""Format rack name"""
LOG.info("Getting rack data")
racks = {}
hostnames = self.parsed_xl_data["ipmi_data"][1]
for host in hostnames:
rack = self._get_rack(host)
racks[rack] = rack.replace("r", "rack")
return racks

423
spyglass-plugin-xls/excel_parser.py Normal file
View File

@ -0,0 +1,423 @@
# Copyright 2018 AT&T Intellectual Property. All other rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import logging
from openpyxl import load_workbook
from openpyxl import Workbook
import pprint
import re
import sys
import yaml
from spyglass.data_extractor.custom_exceptions import NoSpecMatched
# from spyglass.data_extractor.custom_exceptions
LOG = logging.getLogger(__name__)
class ExcelParser(object):
"""Parse data from excel into a dict"""
def __init__(self, file_name, excel_specs):
self.file_name = file_name
with open(excel_specs, "r") as f:
spec_raw_data = f.read()
self.excel_specs = yaml.safe_load(spec_raw_data)
# A combined design spec, returns a workbok object after combining
# all the inputs excel specs
combined_design_spec = self.combine_excel_design_specs(file_name)
self.wb_combined = combined_design_spec
self.filenames = file_name
self.spec = "xl_spec"
@staticmethod
def sanitize(string):
"""Remove extra spaces and convert string to lower case"""
return string.replace(" ", "").lower()
def compare(self, string1, string2):
"""Compare the strings"""
return bool(re.search(self.sanitize(string1), self.sanitize(string2)))
def validate_sheet(self, spec, sheet):
"""Check if the sheet is correct or not"""
ws = self.wb_combined[sheet]
header_row = self.excel_specs["specs"][spec]["header_row"]
ipmi_header = self.excel_specs["specs"][spec]["ipmi_address_header"]
ipmi_column = self.excel_specs["specs"][spec]["ipmi_address_col"]
header_value = ws.cell(row=header_row, column=ipmi_column).value
return bool(self.compare(ipmi_header, header_value))
def find_correct_spec(self):
"""Find the correct spec"""
for spec in self.excel_specs["specs"]:
sheet_name = self.excel_specs["specs"][spec]["ipmi_sheet_name"]
for sheet in self.wb_combined.sheetnames:
if self.compare(sheet_name, sheet):
self.excel_specs["specs"][spec]["ipmi_sheet_name"] = sheet
if self.validate_sheet(spec, sheet):
return spec
raise NoSpecMatched(self.excel_specs)
def get_ipmi_data(self):
"""Read IPMI data from the sheet"""
ipmi_data = {}
hosts = []
provided_sheetname = self.excel_specs["specs"][
self.spec]["ipmi_sheet_name"]
workbook_object, extracted_sheetname = self.get_xl_obj_and_sheetname(
provided_sheetname)
if workbook_object is not None:
ws = workbook_object[extracted_sheetname]
else:
ws = self.wb_combined[provided_sheetname]
row = self.excel_specs["specs"][self.spec]["start_row"]
end_row = self.excel_specs["specs"][self.spec]["end_row"]
hostname_col = self.excel_specs["specs"][self.spec]["hostname_col"]
ipmi_address_col = self.excel_specs["specs"][
self.spec]["ipmi_address_col"]
host_profile_col = self.excel_specs["specs"][
self.spec]["host_profile_col"]
ipmi_gateway_col = self.excel_specs["specs"][
self.spec]["ipmi_gateway_col"]
previous_server_gateway = None
while row <= end_row:
hostname = self.sanitize(
ws.cell(row=row, column=hostname_col).value)
hosts.append(hostname)
ipmi_address = ws.cell(row=row, column=ipmi_address_col).value
if "/" in ipmi_address:
ipmi_address = ipmi_address.split("/")[0]
ipmi_gateway = ws.cell(row=row, column=ipmi_gateway_col).value
if ipmi_gateway:
previous_server_gateway = ipmi_gateway
else:
ipmi_gateway = previous_server_gateway
host_profile = ws.cell(row=row, column=host_profile_col).value
try:
if host_profile is None:
raise RuntimeError(
"No value read from {} ".format(self.file_name) +
"sheet:{} row:{}, col:{}".format(
self.spec, row, host_profile_col))
except RuntimeError as rerror:
LOG.critical(rerror)
sys.exit("Tugboat exited!!")
ipmi_data[hostname] = {
"ipmi_address": ipmi_address,
"ipmi_gateway": ipmi_gateway,
"host_profile": host_profile,
"type": type,
}
row += 1
LOG.debug("ipmi data extracted from excel:\n{}".format(
pprint.pformat(ipmi_data)))
LOG.debug("host data extracted from excel:\n{}".format(
pprint.pformat(hosts)))
return [ipmi_data, hosts]
def get_private_vlan_data(self, ws):
"""Get private vlan data from private IP sheet"""
vlan_data = {}
row = self.excel_specs["specs"][self.spec]["vlan_start_row"]
end_row = self.excel_specs["specs"][self.spec]["vlan_end_row"]
type_col = self.excel_specs["specs"][self.spec]["net_type_col"]
vlan_col = self.excel_specs["specs"][self.spec]["vlan_col"]
while row <= end_row:
cell_value = ws.cell(row=row, column=type_col).value
if cell_value:
vlan = ws.cell(row=row, column=vlan_col).value
if vlan:
vlan = vlan.lower()
vlan_data[vlan] = cell_value
row += 1
LOG.debug("vlan data extracted from excel:\n%s",
pprint.pformat(vlan_data))
return vlan_data
def get_private_network_data(self):
"""Read network data from the private ip sheet"""
provided_sheetname = self.excel_specs["specs"][
self.spec]["private_ip_sheet"]
workbook_object, extracted_sheetname = self.get_xl_obj_and_sheetname(
provided_sheetname)
if workbook_object is not None:
ws = workbook_object[extracted_sheetname]
else:
ws = self.wb_combined[provided_sheetname]
vlan_data = self.get_private_vlan_data(ws)
network_data = {}
row = self.excel_specs["specs"][self.spec]["net_start_row"]
end_row = self.excel_specs["specs"][self.spec]["net_end_row"]
col = self.excel_specs["specs"][self.spec]["net_col"]
vlan_col = self.excel_specs["specs"][self.spec]["net_vlan_col"]
old_vlan = ""
while row <= end_row:
vlan = ws.cell(row=row, column=vlan_col).value
if vlan:
vlan = vlan.lower()
network = ws.cell(row=row, column=col).value
if vlan and network:
net_type = vlan_data[vlan]
if "vlan" not in network_data:
network_data[net_type] = {"vlan": vlan, "subnet": []}
elif not vlan and network:
# If vlan is not present then assign old vlan to vlan as vlan
# value is spread over several rows
vlan = old_vlan
else:
row += 1
continue
network_data[vlan_data[vlan]]["subnet"].append(network)
old_vlan = vlan
row += 1
for network in network_data:
network_data[network]["is_common"] = True
"""
if len(network_data[network]['subnet']) > 1:
network_data[network]['is_common'] = False
else:
network_data[network]['is_common'] = True
LOG.debug(
"private network data extracted from\
excel:\n%s", pprint.pformat(network_data))
"""
return network_data
def get_public_network_data(self):
"""Read public network data from public ip data"""
network_data = {}
provided_sheetname = self.excel_specs["specs"][
self.spec]["public_ip_sheet"]
workbook_object, extracted_sheetname = self.get_xl_obj_and_sheetname(
provided_sheetname)
if workbook_object is not None:
ws = workbook_object[extracted_sheetname]
else:
ws = self.wb_combined[provided_sheetname]
oam_row = self.excel_specs["specs"][self.spec]["oam_ip_row"]
oam_col = self.excel_specs["specs"][self.spec]["oam_ip_col"]
oam_vlan_col = self.excel_specs["specs"][self.spec]["oam_vlan_col"]
ingress_row = self.excel_specs["specs"][self.spec]["ingress_ip_row"]
oob_row = self.excel_specs["specs"][self.spec]["oob_net_row"]
col = self.excel_specs["specs"][self.spec]["oob_net_start_col"]
end_col = self.excel_specs["specs"][self.spec]["oob_net_end_col"]
network_data = {
"oam": {
"subnet": [ws.cell(row=oam_row, column=oam_col).value],
"vlan": ws.cell(row=oam_row, column=oam_vlan_col).value,
},
"ingress": ws.cell(row=ingress_row, column=oam_col).value,
}
network_data["oob"] = {"subnet": []}
while col <= end_col:
cell_value = ws.cell(row=oob_row, column=col).value
if cell_value:
network_data["oob"]["subnet"].append(self.sanitize(cell_value))
col += 1
LOG.debug(
"public network data extracted from\
excel:\n%s",
pprint.pformat(network_data),
)
return network_data
def get_site_info(self):
"""Read location, dns, ntp and ldap data"""
site_info = {}
provided_sheetname = self.excel_specs["specs"][
self.spec]["dns_ntp_ldap_sheet"]
workbook_object, extracted_sheetname = self.get_xl_obj_and_sheetname(
provided_sheetname)
if workbook_object is not None:
ws = workbook_object[extracted_sheetname]
else:
ws = self.wb_combined[provided_sheetname]
dns_row = self.excel_specs["specs"][self.spec]["dns_row"]
dns_col = self.excel_specs["specs"][self.spec]["dns_col"]
ntp_row = self.excel_specs["specs"][self.spec]["ntp_row"]
ntp_col = self.excel_specs["specs"][self.spec]["ntp_col"]
domain_row = self.excel_specs["specs"][self.spec]["domain_row"]
domain_col = self.excel_specs["specs"][self.spec]["domain_col"]
login_domain_row = self.excel_specs["specs"][
self.spec]["login_domain_row"]
ldap_col = self.excel_specs["specs"][self.spec]["ldap_col"]
global_group = self.excel_specs["specs"][self.spec]["global_group"]
ldap_search_url_row = self.excel_specs["specs"][
self.spec]["ldap_search_url_row"]
dns_servers = ws.cell(row=dns_row, column=dns_col).value
ntp_servers = ws.cell(row=ntp_row, column=ntp_col).value
try:
if dns_servers is None:
raise RuntimeError((
"No value for dns_server from:{} Sheet:'{}' ",
"Row:{} Col:{}",
).format(self.file_name, provided_sheetname, dns_row, dns_col))
except RuntimeError as rerror:
LOG.critical(rerror)
sys.exit("Tugboat exited!!")
dns_servers = dns_servers.replace("\n", " ")
ntp_servers = ntp_servers.replace("\n", " ")
if "," in dns_servers:
dns_servers = dns_servers.split(",")
else:
dns_servers = dns_servers.split()
if "," in ntp_servers:
ntp_servers = ntp_servers.split(",")
else:
ntp_servers = ntp_servers.split()
site_info = {
"location": self.get_location_data(),
"dns": dns_servers,
"ntp": ntp_servers,
"domain": ws.cell(row=domain_row, column=domain_col).value,
"ldap": {
"subdomain": ws.cell(row=login_domain_row,
column=ldap_col).value,
"common_name": ws.cell(row=global_group,
column=ldap_col).value,
"url": ws.cell(row=ldap_search_url_row, column=ldap_col).value,
},
}
LOG.debug(
"Site Info extracted from\
excel:\n%s",
pprint.pformat(site_info),
)
return site_info
def get_location_data(self):
"""Read location data from the site and zone sheet"""
provided_sheetname = self.excel_specs["specs"][
self.spec]["location_sheet"]
workbook_object, extracted_sheetname = self.get_xl_obj_and_sheetname(
provided_sheetname)
if workbook_object is not None:
ws = workbook_object[extracted_sheetname]
else:
ws = self.wb_combined[provided_sheetname]
corridor_row = self.excel_specs["specs"][self.spec]["corridor_row"]
column = self.excel_specs["specs"][self.spec]["column"]
site_name_row = self.excel_specs["specs"][self.spec]["site_name_row"]
state_name_row = self.excel_specs["specs"][self.spec]["state_name_row"]
country_name_row = self.excel_specs["specs"][
self.spec]["country_name_row"]
clli_name_row = self.excel_specs["specs"][self.spec]["clli_name_row"]
return {
"corridor": ws.cell(row=corridor_row, column=column).value,
"name": ws.cell(row=site_name_row, column=column).value,
"state": ws.cell(row=state_name_row, column=column).value,
"country": ws.cell(row=country_name_row, column=column).value,
"physical_location": ws.cell(row=clli_name_row,
column=column).value,
}
def validate_sheet_names_with_spec(self):
"""Checks is sheet name in spec file matches with excel file"""
spec = list(self.excel_specs["specs"].keys())[0]
spec_item = self.excel_specs["specs"][spec]
sheet_name_list = []
ipmi_header_sheet_name = spec_item["ipmi_sheet_name"]
sheet_name_list.append(ipmi_header_sheet_name)
private_ip_sheet_name = spec_item["private_ip_sheet"]
sheet_name_list.append(private_ip_sheet_name)
public_ip_sheet_name = spec_item["public_ip_sheet"]
sheet_name_list.append(public_ip_sheet_name)
dns_ntp_ldap_sheet_name = spec_item["dns_ntp_ldap_sheet"]
sheet_name_list.append(dns_ntp_ldap_sheet_name)
location_sheet_name = spec_item["location_sheet"]
sheet_name_list.append(location_sheet_name)
try:
for sheetname in sheet_name_list:
workbook_object, extracted_sheetname = (
self.get_xl_obj_and_sheetname(sheetname))
if workbook_object is not None:
wb = workbook_object
sheetname = extracted_sheetname
else:
wb = self.wb_combined
if sheetname not in wb.sheetnames:
raise RuntimeError(
"SheetName '{}' not found ".format(sheetname))
except RuntimeError as rerror:
LOG.critical(rerror)
sys.exit("Tugboat exited!!")
LOG.info("Sheet names in excel spec validated")
def get_data(self):
"""Create a dict with combined data"""
self.validate_sheet_names_with_spec()
ipmi_data = self.get_ipmi_data()
network_data = self.get_private_network_data()
public_network_data = self.get_public_network_data()
site_info_data = self.get_site_info()
data = {
"ipmi_data": ipmi_data,
"network_data": {
"private": network_data,
"public": public_network_data,
},
"site_info": site_info_data,
}
LOG.debug(
"Location data extracted from\
excel:\n%s",
pprint.pformat(data),
)
return data
def combine_excel_design_specs(self, filenames):
"""Combines multiple excel file to a single design spec"""
design_spec = Workbook()
for exel_file in filenames:
loaded_workbook = load_workbook(exel_file, data_only=True)
for names in loaded_workbook.sheetnames:
design_spec_worksheet = design_spec.create_sheet(names)
loaded_workbook_ws = loaded_workbook[names]
for row in loaded_workbook_ws:
for cell in row:
design_spec_worksheet[
cell.coordinate].value = cell.value
return design_spec
def get_xl_obj_and_sheetname(self, sheetname):
"""The logic confirms if the sheetname is specified for example as:
'MTN57a_AEC_Network_Design_v1.6.xlsx:Public IPs'
"""
if re.search(".xlsx", sheetname) or re.search(".xls", sheetname):
""" Extract file name """
source_xl_file = sheetname.split(":")[0]
wb = load_workbook(source_xl_file, data_only=True)
return [wb, sheetname.split(":")[1]]
else:
return [None, sheetname]

8
test-requirements.txt Normal file
View File

@ -0,0 +1,8 @@
# Formatting
yapf==0.27.0
# Linting
hacking>=1.1.0,<1.2.0 # Apache-2.0
# Security
bandit>=1.5.0

9
tools/gate/whitespace-linter.sh Executable file
View File

@ -0,0 +1,9 @@
#!/usr/bin/env bash
set -x
RES=$(git grep -E -l " +$")
if [[ -n $RES ]]; then
exit 1
fi

57
tox.ini Normal file
View File

@ -0,0 +1,57 @@
[tox]
envlist = pep8, bandit, docs
minversion = 2.3.1
skipsdist = True
[testenv]
usedevelop = True
setenv =
VIRTUAL_ENV={envdir}
LANGUAGE=en_US
LC_ALL=en_US.utf-8
deps =
-r{toxinidir}/requirements.txt
-r{toxinidir}/test-requirements.txt
passenv = http_proxy https_proxy HTTP_PROXY HTTPS_PROXY no_proxy NO_PROXY PBR_VERSION
whitelist_externals =
find
commands =
find . -type f -name "*.pyc" -delete
{toxinidir}/tools/gate/run-unit-tests.sh '{posargs}'
[testenv:fmt]
basepython = python3
deps =
-r{toxinidir}/test-requirements.txt
commands =
yapf -ir {toxinidir}/spyglass-plugin-xls
[testenv:pep8]
basepython = python3
deps =
-r{toxinidir}/test-requirements.txt
commands =
bash -c "{toxinidir}/tools/gate/whitespace-linter.sh"
yapf -dr {toxinidir}/spyglass-plugin-xls {toxinidir}/setup.py
flake8 {toxinidir}/spyglass-plugin-xls
bandit -r spyglass-plugin-xls -n 5
whitelist_externals =
bash
[testenv:bandit]
deps =
bandit
commands = bandit -r spyglass-plugin-xls -n 5
[flake8]
ignore = E125,E251,W503
[testenv:docs]
basepython = python3
deps =
-r{toxinidir}/requirements.txt
-r{toxinidir}/doc/requirements.txt
commands =
rm -rf doc/build
sphinx-build -b html doc/source doc/build -n -W -v
whitelist_externals = rm