diff --git a/doc/source/getting_started.rst b/doc/source/getting_started.rst index ca0645d..07e546a 100644 --- a/doc/source/getting_started.rst +++ b/doc/source/getting_started.rst @@ -65,12 +65,7 @@ Architecture Supported Features ------------------ -1. Tugboat Plugin: Supports extracting site data from Excel files and - then generate site manifests for sitetype:airship-seaworthy. - Find more documentation for Tugboat, see :ref:`tugboatinfo`. - -2. Remote Data Source Plugin: Supports extracting site data from a REST - endpoint. +1. Spyglass XLS Plugin: https://opendev.org/airship/spyglass-plugin-xls Future Work ----------- @@ -135,4 +130,4 @@ Before using Spyglass you must: .. code-block:: console - pip3 install -r airship-spyglass/requirements.txt + pip3 install -r spyglass/requirements.txt diff --git a/doc/source/index.rst b/doc/source/index.rst index 5a528c1..e802f35 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -34,4 +34,3 @@ fed to Shipyard for site deployment / updates. getting_started developer_quickstart cli - tugboat diff --git a/doc/source/tugboat.rst b/doc/source/tugboat.rst deleted file mode 100644 index 2adf9f1..0000000 --- a/doc/source/tugboat.rst +++ /dev/null @@ -1,108 +0,0 @@ -.. - Copyright 2019 AT&T Intellectual Property. - All Rights Reserved. - - Licensed under the Apache License, Version 2.0 (the "License"); you may - not use this file except in compliance with the License. You may obtain - a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, WITHOUT - WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the - License for the specific language governing permissions and limitations - under the License. - -.. _tugboatinfo: - -======= -Tugboat -======= - -What is Tugboat? ----------------- - -Tugboat is a Spyglass plugin to generate airship-seaworthy site manifest files -from an excel based engineering spec. The plugin is configured with an Excel -sheet and its corresponding excel specification as inputs. Spyglass uses this -plugin to construct an intermediary yaml which is processed further using J2 -templates to generate site manifests. - -Excel specification -------------------- -Excel Spec is like an index to the Excel sheet to look for the data to be -collected by the tool. Excel Spec Sample specifies all the details that -need to be filled by the Deployment Engineer. - -Below is the definition for each key in the Excel spec - -* ipmi_sheet_name - name of the sheet from where IPMI and host profile - information is to be read -* start_row - row number from where the IPMI and host profile information - starts -* end_row - row number from where the IPMI and host profile information ends -* hostname_col - column number where the hostnames are to be read from -* ipmi_address_col - column number from where the ipmi addresses are to be read -* host_profile_col - column number from where the host profiles are to be read -* ipmi_gateway_col - column number from where the ipmi gateways are to be read -* private_ip_sheet - name of the sheet which has the private IP information -* net_type_col - column number from where the network type is to be read -* vlan_col - column number from where the network vlan is to be read -* vlan_start_row - row number from where the vlan information starts -* vlan_end_row - row number from where the vlan information ends -* net_start_row - row number from where the network information starts -* net_end_row - row number from where the network information ends -* net_col - column number where the IP ranges for network is to be read -* net_vlan_col - column number where the vlan information is present in the - pod wise network section -* public_ip_sheet - name of the sheet which has the public IP information -* oam_vlan_col - column number from where the OAM vlan information is to be - read from -* oam_ip_row - row number from where the OAM network information is to be read - from -* oam_ip_col - column number from where the OAM network information is to be - read from -* oob_net_row - row number which has the OOB network subnet ranges -* oob_net_start_col - column number from where the OOB network ranges start -* oob_net_end_col - column number from where the OOB network ranges end -* ingress_ip_row - row number from where the Ingress network information is to - be read from -* dns_ntp_ldap_sheet - name of the sheet which has the DNS, NTP and LDAP - information -* login_domain_row - row number which has the ldap login domain -* ldap_col - column number which has the all ldap related information -* global_group - row number which has the ldap group information -* ldap_search_url_row - row number which has the ldap url -* ntp_row - row number which has the ntp information -* ntp_col - column number which has the ntp information -* dns_row - row number which has the dns information -* dns_col - column number which has the dns information -* domain_row - row number which has the domain information -* domain_col - column number which has the domain information -* location_sheet - name of the sheet which has the location information -* column - column number which has all the information -* corridor_row - row number which has the corridor information -* site_name_row - row number which has the site name -* state_name_row - row number which has the state name -* country_name_row - row number which has the country name -* clli_name_row - row number which has CLLI information - -Example: Tugboat Plugin Usage ------------------------------ - -1. Required Input(Refer to 'spyglass/examples' folder to get these inputs) - - a) Excel File: SiteDesignSpec_v0.1.xlsx - b) Excel Spec: excel_spec_upstream.yaml - c) Site Config: site_config.yaml - d) Template_dir: '../examples/templates' - e) Site name: airship-seaworthy - -2. Spyglass CLI Command: - -.. code-block:: bash - - spyglass m -i -p tugboat -x SiteDesignSpec_v0.1.xlsx \ - -e excel_spec_upstream.yaml -c site_config.yaml \ - -s airship-seaworthy -t \ No newline at end of file diff --git a/requirements.txt b/requirements.txt index f759109..a8b1656 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,7 +1,10 @@ click==7.0 +click-plugins==1.1.1 jinja2==2.10 jsonschema==3.0.1 openpyxl==2.5.4 netaddr==0.7.19 pyyaml==5.1 requests==2.21.0 + +git+https://opendev.org/airship/spyglass-plugin-xls.git#egg=spyglass-plugin-xls diff --git a/setup.cfg b/setup.cfg index 528f1dc..ebf9d75 100644 --- a/setup.cfg +++ b/setup.cfg @@ -27,8 +27,9 @@ packages = console_scripts = spyglass = spyglass.cli:main data_extractor_plugins = - tugboat = spyglass.data_extractor.plugins.tugboat.tugboat:TugboatPlugin - formation = spyglass.data_extractor.plugins.formation.formation:FormationPlugin + excel = spyglass_plugin_xls.excel:ExcelPlugin +cli_plugins = + excel = spyglass_plugin_xls.cli:excel [yapf] based_on_style = pep8 diff --git a/spyglass/cli.py b/spyglass/cli.py index a473ec1..81369d8 100644 --- a/spyglass/cli.py +++ b/spyglass/cli.py @@ -16,6 +16,7 @@ import logging import pprint import click +from click_plugins import with_plugins import pkg_resources import yaml @@ -31,96 +32,6 @@ CONTEXT_SETTINGS = { 'help_option_names': ['-h', '--help'], } - -def tugboat_required_callback(ctx, param, value): - LOG.debug('Evaluating %s: %s', param.name, value) - if 'plugin_type' not in ctx.params or \ - ctx.params['plugin_type'] == 'tugboat': - if not value: - raise click.UsageError( - '%s is required for the tugboat ' - 'plugin.' % str(param.name), - ctx=ctx) - return value - - -def formation_required_callback(ctx, param, value): - LOG.debug('Evaluating %s: %s', param.name, value) - if 'plugin_type' in ctx.params: - if ctx.params['plugin_type'] == 'formation': - if not value: - raise click.UsageError( - '%s is required for the ' - 'formation plugin.' % str(param.name), - ctx=ctx) - return value - return ['', '', ''] - - -PLUGIN_TYPE_OPTION = click.option( - '-p', - '--plugin-type', - 'plugin_type', - type=click.Choice(['formation', 'tugboat']), - default='tugboat', - show_default=True, - help='The plugin type to use.') - -# TODO(ianp): Either provide a prompt for passwords or use environment -# variable so passwords are no longer plain text -FORMATION_TARGET_OPTION = click.option( - '-f', - '--formation-target', - 'formation_target', - nargs=3, - help=( - 'Target URL, username, and password for formation plugin. Required ' - 'for formation plugin.'), - callback=formation_required_callback) - -INTERMEDIARY_DIR_OPTION = click.option( - '-d', - '--intermediary-dir', - 'intermediary_dir', - type=click.Path(exists=True, file_okay=False, writable=True), - default='./', - help='Directory in which the intermediary file will be created.') - -EXCEL_FILE_OPTION = click.option( - '-x', - '--excel-file', - 'excel_file', - multiple=True, - type=click.Path(exists=True, readable=True, dir_okay=False), - help='Path to the engineering Excel file. Required for tugboat plugin.', - callback=tugboat_required_callback) - -EXCEL_SPEC_OPTION = click.option( - '-e', - '--excel-spec', - 'excel_spec', - type=click.Path(exists=True, readable=True, dir_okay=False), - help=( - 'Path to the Excel specification YAML file for the engineering ' - 'Excel file. Required for tugboat plugin.'), - callback=tugboat_required_callback) - -SITE_CONFIGURATION_FILE_OPTION = click.option( - '-c', - '--site-configuration', - 'site_configuration', - type=click.Path(exists=True, readable=True, dir_okay=False), - required=False, - help='Path to site specific configuration details YAML file.') - -SITE_NAME_CONFIGURATION_OPTION = click.option( - '-s', - '--site-name', - 'site_name', - type=click.STRING, - required=False, - help='Name of the site for which the intermediary is being generated.') - TEMPLATE_DIR_OPTION = click.option( '-t', '--template-dir', @@ -138,13 +49,14 @@ MANIFEST_DIR_OPTION = click.option( help='Path to place created manifest files.') -@click.group(context_settings=CONTEXT_SETTINGS) @click.option( '-v', '--verbose', is_flag=True, default=False, help='Enable debug messages in log.') +@with_plugins(pkg_resources.iter_entry_points('cli_plugins')) +@click.group() def main(*, verbose): """CLI for Airship Spyglass""" if verbose: @@ -154,9 +66,7 @@ def main(*, verbose): logging.basicConfig(format=LOG_FORMAT, level=log_level) -def _intermediary_helper( - plugin_type, formation_data, site, excel_file, excel_spec, - additional_configuration): +def intermediary_processor(plugin_type, **kwargs): LOG.info("Generating Intermediary yaml") plugin_type = plugin_type plugin_class = None @@ -165,6 +75,7 @@ def _intermediary_helper( LOG.info("Load the plugin class") for entry_point in \ pkg_resources.iter_entry_points('data_extractor_plugins'): + LOG.debug("Entry point '%s' found", entry_point.name) if entry_point.name == plugin_type: plugin_class = entry_point.load() @@ -175,20 +86,13 @@ def _intermediary_helper( # Extract data from plugin data source LOG.info("Extract data from plugin data source") - data_extractor = plugin_class(site) - plugin_conf = data_extractor.get_plugin_conf( - { - 'excel': excel_file, - 'excel_spec': excel_spec, - 'formation_url': formation_data[0], - 'formation_user': formation_data[1], - 'formation_password': formation_data[2] - }) + data_extractor = plugin_class(kwargs['site_name']) + plugin_conf = data_extractor.get_plugin_conf(**kwargs) data_extractor.set_config_opts(plugin_conf) data_extractor.extract_data() # Apply any additional_config provided by user - additional_config = additional_configuration + additional_config = kwargs.get('site_configuration', None) if additional_config is not None: with open(additional_config, 'r') as config: raw_data = config.read() @@ -204,75 +108,12 @@ def _intermediary_helper( # Apply design rules to the data LOG.info("Apply design rules to the extracted data") - process_input_ob = ProcessDataSource(site) + process_input_ob = ProcessDataSource(kwargs['site_name']) process_input_ob.load_extracted_data_from_data_source( data_extractor.site_data) return process_input_ob -@main.command( - 'i', - short_help='generate intermediary', - help='Generates an intermediary file from passed excel data.') -@PLUGIN_TYPE_OPTION -@FORMATION_TARGET_OPTION -@INTERMEDIARY_DIR_OPTION -@EXCEL_FILE_OPTION -@EXCEL_SPEC_OPTION -@SITE_CONFIGURATION_FILE_OPTION -@SITE_NAME_CONFIGURATION_OPTION -def generate_intermediary( - *, plugin_type, formation_target, intermediary_dir, excel_file, - excel_spec, site_configuration, site_name): - process_input_ob = _intermediary_helper( - plugin_type, formation_target, site_name, excel_file, excel_spec, - site_configuration) - LOG.info("Generate intermediary yaml") - process_input_ob.generate_intermediary_yaml() - process_input_ob.dump_intermediary_file(intermediary_dir) - - -@main.command( - 'm', - short_help='generates manifest and intermediary', - help='Generates manifest and intermediary files.') -@click.option( - '-i', - '--save-intermediary', - 'save_intermediary', - is_flag=True, - default=False, - help='Flag to save the generated intermediary file used for the manifests.' -) -@PLUGIN_TYPE_OPTION -@FORMATION_TARGET_OPTION -@INTERMEDIARY_DIR_OPTION -@EXCEL_FILE_OPTION -@EXCEL_SPEC_OPTION -@SITE_CONFIGURATION_FILE_OPTION -@SITE_NAME_CONFIGURATION_OPTION -@TEMPLATE_DIR_OPTION -@MANIFEST_DIR_OPTION -def generate_manifests_and_intermediary( - *, save_intermediary, plugin_type, formation_target, intermediary_dir, - excel_file, excel_spec, site_configuration, site_name, template_dir, - manifest_dir): - process_input_ob = _intermediary_helper( - plugin_type, formation_target, site_name, excel_file, excel_spec, - site_configuration) - LOG.info("Generate intermediary yaml") - intermediary_yaml = process_input_ob.generate_intermediary_yaml() - if save_intermediary: - LOG.debug("Dumping intermediary yaml") - process_input_ob.dump_intermediary_file(intermediary_dir) - else: - LOG.debug("Skipping dump for intermediary yaml") - - LOG.info("Generating site Manifests") - processor_engine = SiteProcessor(intermediary_yaml, manifest_dir) - processor_engine.render_template(template_dir) - - @main.command( 'mi', short_help='generates manifest from intermediary', diff --git a/spyglass/data_extractor/plugins/__init__.py b/spyglass/data_extractor/plugins/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/spyglass/data_extractor/plugins/formation/__init__.py b/spyglass/data_extractor/plugins/formation/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/spyglass/data_extractor/plugins/formation/formation.py b/spyglass/data_extractor/plugins/formation/formation.py deleted file mode 100755 index 1f80379..0000000 --- a/spyglass/data_extractor/plugins/formation/formation.py +++ /dev/null @@ -1,508 +0,0 @@ -# Copyright 2018 AT&T Intellectual Property. All other rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -import logging -import pprint -import re - -import formation_client -import requests -import urllib3 - -from spyglass.data_extractor.base import BaseDataSourcePlugin -import spyglass.data_extractor.custom_exceptions as exceptions - -urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) - -LOG = logging.getLogger(__name__) - - -class FormationPlugin(BaseDataSourcePlugin): - - def __init__(self, region): - # Save site name is valid - if not region: - LOG.error("Site: None! Spyglass exited!") - LOG.info("Check spyglass --help for details") - exit() - super().__init__(region) - - self.source_type = "rest" - self.source_name = "formation" - - # Configuration parameters - self.formation_api_url = None - self.user = None - self.password = None - self.token = None - - # Formation objects - self.client_config = None - self.formation_api_client = None - - # Site related data - self.region_zone_map = {} - self.site_name_id_mapping = {} - self.zone_name_id_mapping = {} - self.region_name_id_mapping = {} - self.rack_name_id_mapping = {} - self.device_name_id_mapping = {} - LOG.info("Initiated data extractor plugin:{}".format(self.source_name)) - - def set_config_opts(self, conf): - """Sets the config params passed by CLI""" - - LOG.info("Plugin params passed:\n{}".format(pprint.pformat(conf))) - self._validate_config_options(conf) - self.formation_api_url = conf["url"] - self.user = conf["user"] - self.password = conf["password"] - self.token = conf.get("token", None) - - self._get_formation_client() - self._update_site_and_zone(self.region) - - def get_plugin_conf(self, kwargs): - """Validates the plugin param and return if success""" - - if not kwargs["formation_url"]: - LOG.error("formation_url not specified! Spyglass exited!") - exit() - url = kwargs["formation_url"] - - if not kwargs["formation_user"]: - LOG.error("formation_user not specified! Spyglass exited!") - exit() - user = kwargs["formation_user"] - - if not kwargs["formation_password"]: - LOG.error("formation_password not specified! Spyglass exited!") - exit() - password = kwargs['formation_password'] - - plugin_conf = {"url": url, "user": user, "password": password} - return plugin_conf - - def _validate_config_options(self, conf): - """Validate the CLI params passed - - The method checks for missing parameters and terminates - Spyglass execution if found so. - """ - - missing_params = [] - for key in conf.keys(): - if conf[key] is None: - missing_params.append(key) - if len(missing_params) != 0: - LOG.error("Missing Plugin Params{}:".format(missing_params)) - exit() - - # Implement helper classes - - def _generate_token(self): - """Generate token for Formation - - Formation API does not provide separate resource to generate - token. This is a workaround to call directly Formation API - to get token instead of using Formation client. - """ - - # Create formation client config object - self.client_config = formation_client.Configuration() - self.client_config.host = self.formation_api_url - self.client_config.username = self.user - self.client_config.password = self.password - self.client_config.verify_ssl = False - - # Assumes token is never expired in the execution of this tool - if self.token: - return self.token - - url = self.formation_api_url + "/zones" - try: - token_response = requests.get( - url, - auth=(self.user, self.password), - verify=self.client_config.verify_ssl, - ) - except requests.exceptions.ConnectionError: - raise exceptions.FormationConnectionError( - "Incorrect URL: {}".format(url)) - - if token_response.status_code == 200: - self.token = token_response.json().get("X-Subject-Token", None) - else: - raise exceptions.TokenGenerationError( - "Unable to generate token because {}".format( - token_response.reason)) - - return self.token - - def _get_formation_client(self): - """Create formation client object - - Formation uses X-Auth-Token for authentication and should be in - format "user|token". - Generate the token and add it formation config object. - """ - - token = self._generate_token() - self.client_config.api_key = {"X-Auth-Token": self.user + "|" + token} - self.formation_api_client = \ - formation_client.ApiClient(self.client_config) - - def _update_site_and_zone(self, region): - """Get Zone name and Site name from region""" - - zone = self._get_zone_by_region_name(region) - site = self._get_site_by_zone_name(zone) - - # zone = region[:-1] - # site = zone[:-1] - - self.region_zone_map[region] = {} - self.region_zone_map[region]["zone"] = zone - self.region_zone_map[region]["site"] = site - - def _get_zone_by_region_name(self, region_name): - zone_api = formation_client.ZonesApi(self.formation_api_client) - zones = zone_api.zones_get() - - # Walk through each zone and get regions - # Return when region name matches - for zone in zones: - self.zone_name_id_mapping[zone.name] = zone.id - zone_regions = self.get_regions(zone.name) - if region_name in zone_regions: - return zone.name - - return None - - def _get_site_by_zone_name(self, zone_name): - site_api = formation_client.SitesApi(self.formation_api_client) - sites = site_api.sites_get() - - # Walk through each site and get zones - # Return when site name matches - for site in sites: - self.site_name_id_mapping[site.name] = site.id - site_zones = self.get_zones(site.name) - if zone_name in site_zones: - return site.name - - return None - - def _get_site_id_by_name(self, site_name): - if site_name in self.site_name_id_mapping: - return self.site_name_id_mapping.get(site_name) - - site_api = formation_client.SitesApi(self.formation_api_client) - sites = site_api.sites_get() - for site in sites: - self.site_name_id_mapping[site.name] = site.id - if site.name == site_name: - return site.id - - def _get_zone_id_by_name(self, zone_name): - if zone_name in self.zone_name_id_mapping: - return self.zone_name_id_mapping.get(zone_name) - - zone_api = formation_client.ZonesApi(self.formation_api_client) - zones = zone_api.zones_get() - for zone in zones: - if zone.name == zone_name: - self.zone_name_id_mapping[zone.name] = zone.id - return zone.id - - def _get_region_id_by_name(self, region_name): - if region_name in self.region_name_id_mapping: - return self.region_name_id_mapping.get(region_name) - - for zone in self.zone_name_id_mapping: - self.get_regions(zone) - - return self.region_name_id_mapping.get(region_name, None) - - def _get_rack_id_by_name(self, rack_name): - if rack_name in self.rack_name_id_mapping: - return self.rack_name_id_mapping.get(rack_name) - - for zone in self.zone_name_id_mapping: - self.get_racks(zone) - - return self.rack_name_id_mapping.get(rack_name, None) - - def _get_device_id_by_name(self, device_name): - if device_name in self.device_name_id_mapping: - return self.device_name_id_mapping.get(device_name) - - self.get_hosts(self.zone) - - return self.device_name_id_mapping.get(device_name, None) - - def _get_racks(self, zone, rack_type="compute"): - zone_id = self._get_zone_id_by_name(zone) - rack_api = formation_client.RacksApi(self.formation_api_client) - racks = rack_api.zones_zone_id_racks_get(zone_id) - - racks_list = [] - for rack in racks: - rack_name = rack.name - self.rack_name_id_mapping[rack_name] = rack.id - if rack.rack_type.name == rack_type: - racks_list.append(rack_name) - - return racks_list - - # Functions that will be used internally within this plugin - - def get_zones(self, site=None): - zone_api = formation_client.ZonesApi(self.formation_api_client) - - if site is None: - zones = zone_api.zones_get() - else: - site_id = self._get_site_id_by_name(site) - zones = zone_api.sites_site_id_zones_get(site_id) - - zones_list = [] - for zone in zones: - zone_name = zone.name - self.zone_name_id_mapping[zone_name] = zone.id - zones_list.append(zone_name) - - return zones_list - - def get_regions(self, zone): - zone_id = self._get_zone_id_by_name(zone) - region_api = formation_client.RegionApi(self.formation_api_client) - regions = region_api.zones_zone_id_regions_get(zone_id) - regions_list = [] - for region in regions: - region_name = region.name - self.region_name_id_mapping[region_name] = region.id - regions_list.append(region_name) - - return regions_list - - # Implement Abstract functions - - def get_racks(self, region): - zone = self.region_zone_map[region]["zone"] - return self._get_racks(zone, rack_type="compute") - - def get_hosts(self, region, rack=None): - zone = self.region_zone_map[region]["zone"] - zone_id = self._get_zone_id_by_name(zone) - device_api = formation_client.DevicesApi(self.formation_api_client) - control_hosts = device_api.zones_zone_id_control_nodes_get(zone_id) - compute_hosts = device_api.zones_zone_id_devices_get( - zone_id, type="KVM") - - hosts_list = [] - for host in control_hosts: - self.device_name_id_mapping[host.aic_standard_name] = host.id - hosts_list.append( - { - "name": host.aic_standard_name, - "type": "controller", - "rack_name": host.rack_name, - "host_profile": host.host_profile_name, - }) - - for host in compute_hosts: - self.device_name_id_mapping[host.aic_standard_name] = host.id - hosts_list.append( - { - "name": host.aic_standard_name, - "type": "compute", - "rack_name": host.rack_name, - "host_profile": host.host_profile_name, - }) - """ - for host in itertools.chain(control_hosts, compute_hosts): - self.device_name_id_mapping[host.aic_standard_name] = host.id - hosts_list.append({ - 'name': host.aic_standard_name, - 'type': host.categories[0], - 'rack_name': host.rack_name, - 'host_profile': host.host_profile_name - }) - """ - - return hosts_list - - def get_networks(self, region): - zone = self.region_zone_map[region]["zone"] - zone_id = self._get_zone_id_by_name(zone) - region_id = self._get_region_id_by_name(region) - vlan_api = formation_client.VlansApi(self.formation_api_client) - vlans = vlan_api.zones_zone_id_regions_region_id_vlans_get( - zone_id, region_id) - - # Case when vlans list is empty from - # zones_zone_id_regions_region_id_vlans_get - if len(vlans) == 0: - # get device-id from the first host and get the network details - hosts = self.get_hosts(self.region) - host = hosts[0]["name"] - device_id = self._get_device_id_by_name(host) - vlans = \ - vlan_api.zones_zone_id_devices_device_id_vlans_get(zone_id, - device_id) - - LOG.debug("Extracted region network information\n{}".format(vlans)) - vlans_list = [] - for vlan_ in vlans: - if len(vlan_.vlan.ipv4) != 0: - tmp_vlan = { - "name": self._get_network_name_from_vlan_name( - vlan_.vlan.name), - "vlan": vlan_.vlan.vlan_id, - "subnet": vlan_.vlan.subnet_range, - "gateway": vlan_.ipv4_gateway, - "subnet_level": vlan_.vlan.subnet_level - } - vlans_list.append(tmp_vlan) - - return vlans_list - - def get_ips(self, region, host=None): - zone = self.region_zone_map[region]["zone"] - zone_id = self._get_zone_id_by_name(zone) - - if host: - hosts = [host] - else: - hosts = [] - hosts_dict = self.get_hosts(zone) - for host in hosts_dict: - hosts.append(host["name"]) - - vlan_api = formation_client.VlansApi(self.formation_api_client) - ip_ = {} - - for host in hosts: - device_id = self._get_device_id_by_name(host) - vlans = \ - vlan_api.zones_zone_id_devices_device_id_vlans_get(zone_id, - device_id) - LOG.debug("Received VLAN Network Information\n{}".format(vlans)) - ip_[host] = {} - for vlan_ in vlans: - # TODO(pg710r) We need to handle the case when incoming ipv4 - # list is empty - if len(vlan_.vlan.ipv4) != 0: - name = self._get_network_name_from_vlan_name( - vlan_.vlan.name) - ipv4 = vlan_.vlan.ipv4[0].ip - LOG.debug( - "vlan:{},name:{},ip:{},vlan_name:{}".format( - vlan_.vlan.vlan_id, name, ipv4, vlan_.vlan.name)) - # TODD(pg710r) This code needs to extended to support ipv4 - # and ipv6 - # ip_[host][name] = {'ipv4': ipv4} - ip_[host][name] = ipv4 - - return ip_ - - def _get_network_name_from_vlan_name(self, vlan_name): - """Network names are ksn, oam, oob, overlay, storage, pxe - - The following mapping rules apply: - vlan_name contains "ksn" the network name is "calico" - vlan_name contains "storage" the network name is "storage" - vlan_name contains "server" the network name is "oam" - vlan_name contains "ovs" the network name is "overlay" - vlan_name contains "ILO" the network name is "oob" - """ - - network_names = { - "ksn": "calico", - "storage": "storage", - "server": "oam", - "ovs": "overlay", - "ILO": "oob", - "pxe": "pxe", - } - - for name in network_names: - # Make a pattern that would ignore case. - # if name is 'ksn' pattern name is '(?i)(ksn)' - name_pattern = "(?i)({})".format(name) - if re.search(name_pattern, vlan_name): - return network_names[name] - # Return empty string is vlan_name is not matched with network_names - return "" - - def get_dns_servers(self, region): - try: - zone = self.region_zone_map[region]["zone"] - zone_id = self._get_zone_id_by_name(zone) - zone_api = formation_client.ZonesApi(self.formation_api_client) - zone_ = zone_api.zones_zone_id_get(zone_id) - except formation_client.rest.ApiException as e: - raise exceptions.ApiClientError(e.msg) - - if not zone_.ipv4_dns: - LOG.warning("No dns server") - return [] - - dns_list = [] - for dns in zone_.ipv4_dns: - dns_list.append(dns.ip) - - return dns_list - - def get_ntp_servers(self, region): - return [] - - def get_ldap_information(self, region): - return {} - - def get_location_information(self, region): - """Get location information for a zone and return""" - - site = self.region_zone_map[region]["site"] - site_id = self._get_site_id_by_name(site) - site_api = formation_client.SitesApi(self.formation_api_client) - site_info = site_api.sites_site_id_get(site_id) - - try: - return { - # 'corridor': site_info.corridor, - "name": site_info.city, - "state": site_info.state, - "country": site_info.country, - "physical_location_id": site_info.clli, - } - except AttributeError as e: - raise exceptions.MissingAttributeError( - "Missing {} information in {}".format(e, site_info.city)) - - def get_domain_name(self, region): - try: - zone = self.region_zone_map[region]["zone"] - zone_id = self._get_zone_id_by_name(zone) - zone_api = formation_client.ZonesApi(self.formation_api_client) - zone_ = zone_api.zones_zone_id_get(zone_id) - except formation_client.rest.ApiException as e: - raise exceptions.ApiClientError(e.msg) - - if not zone_.dns: - LOG.warning("Got None while running get domain name") - return None - - return zone_.dns diff --git a/spyglass/data_extractor/plugins/tugboat/__init__.py b/spyglass/data_extractor/plugins/tugboat/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/spyglass/data_extractor/plugins/tugboat/check_exceptions.py b/spyglass/data_extractor/plugins/tugboat/check_exceptions.py index 49ac3b4..e69de29 100644 --- a/spyglass/data_extractor/plugins/tugboat/check_exceptions.py +++ b/spyglass/data_extractor/plugins/tugboat/check_exceptions.py @@ -1,38 +0,0 @@ -# Copyright 2018 AT&T Intellectual Property. All other rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - - -class BaseError(Exception): - pass - - -class NotEnoughIp(BaseError): - - def __init__(self, cidr, total_nodes): - self.cidr = cidr - self.total_nodes = total_nodes - - def display_error(self): - print("{} can not handle {} nodes".format(self.cidr, self.total_nodes)) - - -class NoSpecMatched(BaseError): - - def __init__(self, excel_specs): - self.specs = excel_specs - - def display_error(self): - print( - "No spec matched. Following are the available specs:\n".format( - self.specs)) diff --git a/spyglass/data_extractor/plugins/tugboat/excel_parser.py b/spyglass/data_extractor/plugins/tugboat/excel_parser.py deleted file mode 100755 index 3f4d1c7..0000000 --- a/spyglass/data_extractor/plugins/tugboat/excel_parser.py +++ /dev/null @@ -1,417 +0,0 @@ -# Copyright 2018 AT&T Intellectual Property. All other rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -import logging -import pprint -import re -import sys - -from openpyxl import load_workbook -from openpyxl import Workbook -import yaml - -from spyglass.data_extractor.custom_exceptions import NoSpecMatched - -LOG = logging.getLogger(__name__) - - -class ExcelParser(object): - """Parse data from excel into a dict""" - - def __init__(self, file_name, excel_specs): - self.file_name = file_name - with open(excel_specs, "r") as f: - spec_raw_data = f.read() - self.excel_specs = yaml.safe_load(spec_raw_data) - # A combined design spec, returns a workbook object after combining - # all the inputs excel specs - combined_design_spec = self.combine_excel_design_specs(file_name) - self.wb_combined = combined_design_spec - self.filenames = file_name - self.spec = "xl_spec" - - @staticmethod - def sanitize(string): - """Remove extra spaces and convert string to lower case""" - - return string.replace(" ", "").lower() - - def compare(self, string1, string2): - """Compare the strings""" - - return bool(re.search(self.sanitize(string1), self.sanitize(string2))) - - def validate_sheet(self, spec, sheet): - """Check if the sheet is correct or not""" - - ws = self.wb_combined[sheet] - header_row = self.excel_specs["specs"][spec]["header_row"] - ipmi_header = self.excel_specs["specs"][spec]["ipmi_address_header"] - ipmi_column = self.excel_specs["specs"][spec]["ipmi_address_col"] - header_value = ws.cell(row=header_row, column=ipmi_column).value - return bool(self.compare(ipmi_header, header_value)) - - def find_correct_spec(self): - """Find the correct spec""" - - for spec in self.excel_specs["specs"]: - sheet_name = self.excel_specs["specs"][spec]["ipmi_sheet_name"] - for sheet in self.wb_combined.sheetnames: - if self.compare(sheet_name, sheet): - self.excel_specs["specs"][spec]["ipmi_sheet_name"] = sheet - if self.validate_sheet(spec, sheet): - return spec - raise NoSpecMatched(self.excel_specs) - - def get_ipmi_data(self): - """Read IPMI data from the sheet""" - - ipmi_data = {} - hosts = [] - spec_ = self.excel_specs["specs"][self.spec] - provided_sheetname = spec_["ipmi_sheet_name"] - workbook_object, extracted_sheetname = \ - self.get_xl_obj_and_sheetname(provided_sheetname) - if workbook_object is not None: - ws = workbook_object[extracted_sheetname] - else: - ws = self.wb_combined[provided_sheetname] - row = spec_["start_row"] - end_row = spec_["end_row"] - hostname_col = spec_["hostname_col"] - ipmi_address_col = spec_["ipmi_address_col"] - host_profile_col = spec_["host_profile_col"] - ipmi_gateway_col = spec_["ipmi_gateway_col"] - previous_server_gateway = None - while row <= end_row: - hostname = \ - self.sanitize(ws.cell(row=row, column=hostname_col).value) - hosts.append(hostname) - ipmi_address = ws.cell(row=row, column=ipmi_address_col).value - if "/" in ipmi_address: - ipmi_address = ipmi_address.split("/")[0] - ipmi_gateway = ws.cell(row=row, column=ipmi_gateway_col).value - if ipmi_gateway: - previous_server_gateway = ipmi_gateway - else: - ipmi_gateway = previous_server_gateway - host_profile = ws.cell(row=row, column=host_profile_col).value - try: - if host_profile is None: - raise RuntimeError( - "No value read from " - "{} sheet:{} row:{}, col:{}".format( - self.file_name, self.spec, row, host_profile_col)) - except RuntimeError as rerror: - LOG.critical(rerror) - sys.exit("Tugboat exited!!") - ipmi_data[hostname] = { - "ipmi_address": ipmi_address, - "ipmi_gateway": ipmi_gateway, - "host_profile": host_profile, - "type": type, # FIXME (Ian Pittwood): shadows type built-in - } - row += 1 - LOG.debug( - "ipmi data extracted from excel:\n{}".format( - pprint.pformat(ipmi_data))) - LOG.debug( - "host data extracted from excel:\n{}".format( - pprint.pformat(hosts))) - return [ipmi_data, hosts] - - def get_private_vlan_data(self, ws): - """Get private vlan data from private IP sheet""" - - vlan_data = {} - row = self.excel_specs["specs"][self.spec]["vlan_start_row"] - end_row = self.excel_specs["specs"][self.spec]["vlan_end_row"] - type_col = self.excel_specs["specs"][self.spec]["net_type_col"] - vlan_col = self.excel_specs["specs"][self.spec]["vlan_col"] - while row <= end_row: - cell_value = ws.cell(row=row, column=type_col).value - if cell_value: - vlan = ws.cell(row=row, column=vlan_col).value - if vlan: - vlan = vlan.lower() - vlan_data[vlan] = cell_value - row += 1 - LOG.debug( - "vlan data extracted from excel:\n%s" % pprint.pformat(vlan_data)) - return vlan_data - - def get_private_network_data(self): - """Read network data from the private ip sheet""" - - spec_ = self.excel_specs["specs"][self.spec] - provided_sheetname = spec_["private_ip_sheet"] - workbook_object, extracted_sheetname = \ - self.get_xl_obj_and_sheetname(provided_sheetname) - if workbook_object is not None: - ws = workbook_object[extracted_sheetname] - else: - ws = self.wb_combined[provided_sheetname] - vlan_data = self.get_private_vlan_data(ws) - network_data = {} - row = spec_["net_start_row"] - end_row = spec_["net_end_row"] - col = spec_["net_col"] - vlan_col = spec_["net_vlan_col"] - old_vlan = "" - while row <= end_row: - vlan = ws.cell(row=row, column=vlan_col).value - if vlan: - vlan = vlan.lower() - network = ws.cell(row=row, column=col).value - if vlan and network: - net_type = vlan_data[vlan] - if "vlan" not in network_data: - network_data[net_type] = {"vlan": vlan, "subnet": []} - elif not vlan and network: - # If vlan is not present then assign old vlan to vlan as vlan - # value is spread over several rows - vlan = old_vlan - else: - row += 1 - continue - network_data[vlan_data[vlan]]["subnet"].append(network) - old_vlan = vlan - row += 1 - for network in network_data: - network_data[network]["is_common"] = True - """ - if len(network_data[network]['subnet']) > 1: - network_data[network]['is_common'] = False - else: - network_data[network]['is_common'] = True - LOG.debug("private network data extracted from excel:\n%s" - % pprint.pformat(network_data)) - """ - return network_data - - def get_public_network_data(self): - """Read public network data from public ip data""" - - spec_ = self.excel_specs["specs"][self.spec] - provided_sheetname = spec_["public_ip_sheet"] - workbook_object, extracted_sheetname = self.get_xl_obj_and_sheetname( - provided_sheetname) - if workbook_object is not None: - ws = workbook_object[extracted_sheetname] - else: - ws = self.wb_combined[provided_sheetname] - oam_row = spec_["oam_ip_row"] - oam_col = spec_["oam_ip_col"] - oam_vlan_col = spec_["oam_vlan_col"] - ingress_row = spec_["ingress_ip_row"] - oob_row = spec_["oob_net_row"] - col = spec_["oob_net_start_col"] - end_col = spec_["oob_net_end_col"] - network_data = { - "oam": { - "subnet": [ws.cell(row=oam_row, column=oam_col).value], - "vlan": ws.cell(row=oam_row, column=oam_vlan_col).value, - }, - "ingress": ws.cell(row=ingress_row, column=oam_col).value, - "oob": { - "subnet": [], - } - } - while col <= end_col: - cell_value = ws.cell(row=oob_row, column=col).value - if cell_value: - network_data["oob"]["subnet"].append(self.sanitize(cell_value)) - col += 1 - LOG.debug( - "public network data extracted from excel:\n%s" % - pprint.pformat(network_data)) - return network_data - - def get_site_info(self): - """Read location, dns, ntp and ldap data""" - - spec_ = self.excel_specs["specs"][self.spec] - provided_sheetname = spec_["dns_ntp_ldap_sheet"] - workbook_object, extracted_sheetname = \ - self.get_xl_obj_and_sheetname(provided_sheetname) - if workbook_object is not None: - ws = workbook_object[extracted_sheetname] - else: - ws = self.wb_combined[provided_sheetname] - dns_row = spec_["dns_row"] - dns_col = spec_["dns_col"] - ntp_row = spec_["ntp_row"] - ntp_col = spec_["ntp_col"] - domain_row = spec_["domain_row"] - domain_col = spec_["domain_col"] - login_domain_row = spec_["login_domain_row"] - ldap_col = spec_["ldap_col"] - global_group = spec_["global_group"] - ldap_search_url_row = spec_["ldap_search_url_row"] - dns_servers = ws.cell(row=dns_row, column=dns_col).value - ntp_servers = ws.cell(row=ntp_row, column=ntp_col).value - try: - if dns_servers is None: - raise RuntimeError( - "No value for dns_server from: " - "{} Sheet:'{}' Row:{} Col:{}".format( - self.file_name, provided_sheetname, dns_row, dns_col)) - if ntp_servers is None: - raise RuntimeError( - "No value for ntp_server from: " - "{} Sheet:'{}' Row:{} Col:{}".format( - self.file_name, provided_sheetname, ntp_row, ntp_col)) - except RuntimeError as rerror: - LOG.critical(rerror) - sys.exit("Tugboat exited!!") - - dns_servers = dns_servers.replace("\n", " ") - ntp_servers = ntp_servers.replace("\n", " ") - if "," in dns_servers: - dns_servers = dns_servers.split(",") - else: - dns_servers = dns_servers.split() - if "," in ntp_servers: - ntp_servers = ntp_servers.split(",") - else: - ntp_servers = ntp_servers.split() - site_info = { - "location": self.get_location_data(), - "dns": dns_servers, - "ntp": ntp_servers, - "domain": ws.cell(row=domain_row, column=domain_col).value, - "ldap": { - "subdomain": ws.cell(row=login_domain_row, - column=ldap_col).value, - "common_name": ws.cell(row=global_group, - column=ldap_col).value, - "url": ws.cell(row=ldap_search_url_row, column=ldap_col).value, - }, - } - LOG.debug( - "Site Info extracted from\ - excel:\n%s", - pprint.pformat(site_info), - ) - return site_info - - def get_location_data(self): - """Read location data from the site and zone sheet""" - - spec_ = self.excel_specs["specs"][self.spec] - provided_sheetname = spec_["location_sheet"] - workbook_object, extracted_sheetname = \ - self.get_xl_obj_and_sheetname(provided_sheetname) - if workbook_object is not None: - ws = workbook_object[extracted_sheetname] - else: - ws = self.wb_combined[provided_sheetname] - corridor_row = spec_["corridor_row"] - column = spec_["column"] - site_name_row = spec_["site_name_row"] - state_name_row = spec_["state_name_row"] - country_name_row = spec_["country_name_row"] - clli_name_row = spec_["clli_name_row"] - return { - "corridor": ws.cell(row=corridor_row, column=column).value, - "name": ws.cell(row=site_name_row, column=column).value, - "state": ws.cell(row=state_name_row, column=column).value, - "country": ws.cell(row=country_name_row, column=column).value, - "physical_location": ws.cell(row=clli_name_row, - column=column).value, - } - - def validate_sheet_names_with_spec(self): - """Checks is sheet name in spec file matches with excel file""" - - spec = list(self.excel_specs["specs"].keys())[0] - spec_item = self.excel_specs["specs"][spec] - sheet_name_list = [] - ipmi_header_sheet_name = spec_item["ipmi_sheet_name"] - sheet_name_list.append(ipmi_header_sheet_name) - private_ip_sheet_name = spec_item["private_ip_sheet"] - sheet_name_list.append(private_ip_sheet_name) - public_ip_sheet_name = spec_item["public_ip_sheet"] - sheet_name_list.append(public_ip_sheet_name) - dns_ntp_ldap_sheet_name = spec_item["dns_ntp_ldap_sheet"] - sheet_name_list.append(dns_ntp_ldap_sheet_name) - location_sheet_name = spec_item["location_sheet"] - sheet_name_list.append(location_sheet_name) - try: - for sheetname in sheet_name_list: - workbook_object, extracted_sheetname = \ - self.get_xl_obj_and_sheetname(sheetname) - if workbook_object is not None: - wb = workbook_object - sheetname = extracted_sheetname - else: - wb = self.wb_combined - - if sheetname not in wb.sheetnames: - raise RuntimeError( - "SheetName '{}' not found ".format(sheetname)) - except RuntimeError as rerror: - LOG.critical(rerror) - sys.exit("Tugboat exited!!") - - LOG.info("Sheet names in excel spec validated") - - def get_data(self): - """Create a dict with combined data""" - - self.validate_sheet_names_with_spec() - ipmi_data = self.get_ipmi_data() - network_data = self.get_private_network_data() - public_network_data = self.get_public_network_data() - site_info_data = self.get_site_info() - data = { - "ipmi_data": ipmi_data, - "network_data": { - "private": network_data, - "public": public_network_data, - }, - "site_info": site_info_data, - } - LOG.debug( - "Location data extracted from excel:\n%s" % pprint.pformat(data)) - return data - - def combine_excel_design_specs(self, filenames): - """Combines multiple excel file to a single design spec""" - - design_spec = Workbook() - for exel_file in filenames: - loaded_workbook = load_workbook(exel_file, data_only=True) - for names in loaded_workbook.sheetnames: - design_spec_worksheet = design_spec.create_sheet(names) - loaded_workbook_ws = loaded_workbook[names] - for row in loaded_workbook_ws: - for cell in row: - design_spec_worksheet[cell.coordinate].value = \ - cell.value - return design_spec - - def get_xl_obj_and_sheetname(self, sheetname): - """The logic confirms if the sheetname is specified for example as: - - 'MTN57a_AEC_Network_Design_v1.6.xlsx:Public IPs' - """ - - if re.search(".xlsx", sheetname) or re.search(".xls", sheetname): - # Extract file name - source_xl_file = sheetname.split(":")[0] - wb = load_workbook(source_xl_file, data_only=True) - return [wb, sheetname.split(":")[1]] - else: - return [None, sheetname] diff --git a/spyglass/data_extractor/plugins/tugboat/tugboat.py b/spyglass/data_extractor/plugins/tugboat/tugboat.py deleted file mode 100755 index a9e231c..0000000 --- a/spyglass/data_extractor/plugins/tugboat/tugboat.py +++ /dev/null @@ -1,357 +0,0 @@ -# Copyright 2018 AT&T Intellectual Property. All other rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -import itertools -import logging -import pprint -import re - -from spyglass.data_extractor.base import BaseDataSourcePlugin -from spyglass.data_extractor.plugins.tugboat.excel_parser import ExcelParser - -LOG = logging.getLogger(__name__) - - -class TugboatPlugin(BaseDataSourcePlugin): - - def __init__(self, region): - LOG.info("Tugboat Initializing") - self.source_type = "excel" - self.source_name = "tugboat" - - # Configuration parameters - self.excel_path = None - self.excel_spec = None - - # Site related data - self.region = region - - # Raw data from excel - self.parsed_xl_data = None - - LOG.info("Initiated data extractor plugin:{}".format(self.source_name)) - - def set_config_opts(self, conf): - """Placeholder to set configuration options specific to each plugin. - - :param dict conf: Configuration options as dict - - Example: conf = { 'excel_spec': 'spec1.yaml', - 'excel_path': 'excel.xls' } - - Each plugin will have their own config opts. - """ - - self.excel_path = conf["excel_path"] - self.excel_spec = conf["excel_spec"] - - # Extract raw data from excel sheets - self._get_excel_obj() - self._extract_raw_data_from_excel() - return - - def get_plugin_conf(self, kwargs): - """Validates the plugin param from CLI and return if correct - - Ideally the CLICK module shall report an error if excel file - and excel specs are not specified. The below code has been - written as an additional safeguard. - """ - if not kwargs["excel"]: - LOG.error("Engineering excel file not specified: Spyglass exited!") - exit() - excel_file_info = kwargs["excel"] - if not kwargs["excel_spec"]: - LOG.error("Engineering spec file not specified: Spyglass exited!") - exit() - excel_spec_info = kwargs["excel_spec"] - plugin_conf = { - "excel_path": excel_file_info, - "excel_spec": excel_spec_info, - } - return plugin_conf - - def get_hosts(self, region, rack=None): - """Return list of hosts in the region - - :param string region: Region name - :param string rack: Rack name - :returns: list of hosts information - :rtype: list of dict - Example: [ - { - 'name': 'host01', - 'type': 'controller', - 'host_profile': 'hp_01' - }, - { - 'name': 'host02', - 'type': 'compute', - 'host_profile': 'hp_02'} - ] - """ - - LOG.info("Get Host Information") - ipmi_data = self.parsed_xl_data["ipmi_data"][0] - rackwise_hosts = self._get_rackwise_hosts() - host_list = [] - for rack in rackwise_hosts.keys(): - for host in rackwise_hosts[rack]: - host_list.append( - { - "rack_name": rack, - "name": host, - "host_profile": ipmi_data[host]["host_profile"], - }) - return host_list - - def get_networks(self, region): - """Extracts vlan network info from raw network data from excel""" - - vlan_list = [] - # Network data extracted from xl is formatted to have a predictable - # data type. For e.g VlAN 45 extracted from xl is formatted as 45 - vlan_pattern = r"\d+" - private_net = self.parsed_xl_data["network_data"]["private"] - public_net = self.parsed_xl_data["network_data"]["public"] - # Extract network information from private and public network data - for net_type, net_val in itertools.chain(private_net.items(), - public_net.items()): - tmp_vlan = {} - # Ingress is special network that has no vlan, only a subnet string - # So treatment for ingress is different - if net_type != "ingress": - # standardize the network name as net_type may ne different. - # For e.g instead of pxe it may be PXE or instead of calico - # it may be ksn. Valid network names are pxe, calico, oob, oam, - # overlay, storage, ingress - tmp_vlan["name"] = \ - self._get_network_name_from_vlan_name(net_type) - - # extract vlan tag. It was extracted from xl file as 'VlAN 45' - # The code below extracts the numeric data fron net_val['vlan'] - if net_val.get("vlan", "") != "": - value = re.findall(vlan_pattern, net_val["vlan"]) - tmp_vlan["vlan"] = value[0] - else: - tmp_vlan["vlan"] = "#CHANGE_ME" - - tmp_vlan["subnet"] = net_val.get("subnet", "#CHANGE_ME") - tmp_vlan["gateway"] = net_val.get("gateway", "#CHANGE_ME") - else: - tmp_vlan["name"] = "ingress" - tmp_vlan["subnet"] = net_val - vlan_list.append(tmp_vlan) - LOG.debug( - "vlan list extracted from tugboat:\n{}".format( - pprint.pformat(vlan_list))) - return vlan_list - - def get_ips(self, region, host=None): - """Return list of IPs on the host - - :param string region: Region name - :param string host: Host name - :returns: Dict of IPs per network on the host - :rtype: dict - Example: {'oob': {'ipv4': '192.168.1.10'}, - 'pxe': {'ipv4': '192.168.2.10'}} - The network name from get_networks is expected to be the keys of this - dict. In case some networks are missed, they are expected to be either - DHCP or internally generated n the next steps by the design rules. - """ - - ip_ = {} - ipmi_data = self.parsed_xl_data["ipmi_data"][0] - ip_[host] = { - "oob": ipmi_data[host].get("ipmi_address", "#CHANGE_ME"), - "oam": ipmi_data[host].get("oam", "#CHANGE_ME"), - "calico": ipmi_data[host].get("calico", "#CHANGE_ME"), - "overlay": ipmi_data[host].get("overlay", "#CHANGE_ME"), - "pxe": ipmi_data[host].get("pxe", "#CHANGE_ME"), - "storage": ipmi_data[host].get("storage", "#CHANGE_ME"), - } - return ip_ - - def get_ldap_information(self, region): - """Extract ldap information from excel""" - - ldap_raw_data = self.parsed_xl_data["site_info"]["ldap"] - ldap_info = {} - # raw url is 'url: ldap://example.com' so we are converting to - # 'ldap://example.com' - url = ldap_raw_data.get("url", "#CHANGE_ME") - try: - ldap_info["url"] = url.split(" ")[1] - ldap_info["domain"] = url.split(".")[1] - except IndexError as e: - LOG.error("url.split:{}".format(e)) - ldap_info["common_name"] = \ - ldap_raw_data.get("common_name", "#CHANGE_ME") - ldap_info["subdomain"] = ldap_raw_data.get("subdomain", "#CHANGE_ME") - - return ldap_info - - def get_ntp_servers(self, region): - """Returns a comma separated list of ntp ip addresses""" - - ntp_server_list = \ - self._get_formatted_server_list(self.parsed_xl_data["site_info"] - ["ntp"]) - return ntp_server_list - - def get_dns_servers(self, region): - """Returns a comma separated list of dns ip addresses""" - dns_server_list = \ - self._get_formatted_server_list(self.parsed_xl_data["site_info"] - ["dns"]) - return dns_server_list - - def get_domain_name(self, region): - """Returns domain name extracted from excel file""" - - return self.parsed_xl_data["site_info"]["domain"] - - def get_location_information(self, region): - """Prepare location data from information extracted by ExcelParser""" - - location_data = self.parsed_xl_data["site_info"]["location"] - - corridor_pattern = r"\d+" - corridor_number = \ - re.findall(corridor_pattern, location_data["corridor"])[0] - name = location_data.get("name", "#CHANGE_ME") - state = location_data.get("state", "#CHANGE_ME") - country = location_data.get("country", "#CHANGE_ME") - physical_location_id = location_data.get("physical_location", "") - - return { - "name": name, - "physical_location_id": physical_location_id, - "state": state, - "country": country, - "corridor": "c{}".format(corridor_number), - } - - def get_racks(self, region): - # This function is not required since the excel plugin - # already provide rack information. - pass - - def _get_excel_obj(self): - """Creation of an ExcelParser object to store site information. - - The information is obtained based on a excel spec yaml file. - This spec contains row, column and sheet information of - the excel file from where site specific data can be extracted. - """ - - self.excel_obj = ExcelParser(self.excel_path, self.excel_spec) - - def _extract_raw_data_from_excel(self): - """Extracts raw information from excel file based on excel spec""" - self.parsed_xl_data = self.excel_obj.get_data() - - def _get_network_name_from_vlan_name(self, vlan_name): - """Network names are ksn, oam, oob, overlay, storage, pxe - - This is a utility function to determine the vlan acceptable - vlan from the name extracted from excel file - - The following mapping rules apply: - vlan_name contains "ksn or calico" the network name is "calico" - vlan_name contains "storage" the network name is "storage" - vlan_name contains "server" the network name is "oam" - vlan_name contains "ovs" the network name is "overlay" - vlan_name contains "oob" the network name is "oob" - vlan_name contains "pxe" the network name is "pxe" - """ - - network_names = [ - "ksn|calico", - "storage", - "oam|server", - "ovs|overlay", - "oob", - "pxe", - ] - for name in network_names: - # Make a pattern that would ignore case. - # if name is 'ksn' pattern name is '(?i)(ksn)' - name_pattern = "(?i)({})".format(name) - if re.search(name_pattern, vlan_name): - if name == "ksn|calico": - return "calico" - if name == "storage": - return "storage" - if name == "oam|server": - return "oam" - if name == "ovs|overlay": - return "overlay" - if name == "oob": - return "oob" - if name == "pxe": - return "pxe" - # if nothing matches - LOG.error( - "Unable to recognize VLAN name extracted from Plugin data source") - return "" - - def _get_formatted_server_list(self, server_list): - """Format dns and ntp server list as comma separated string""" - - # dns/ntp server info from excel is of the format - # 'xxx.xxx.xxx.xxx, (aaa.bbb.ccc.com)' - # The function returns a list of comma separated dns ip addresses - servers = [] - for data in server_list: - if "(" not in data: - servers.append(data) - formatted_server_list = ",".join(servers) - return formatted_server_list - - def _get_rack(self, host): - """Get rack id from the rack string extracted from xl""" - - rack_pattern = r"\w.*(r\d+)\w.*" - rack = re.findall(rack_pattern, host)[0] - if not self.region: - self.region = host.split(rack)[0] - return rack - - def _get_rackwise_hosts(self): - """Mapping hosts with rack ids""" - - rackwise_hosts = {} - hostnames = self.parsed_xl_data["ipmi_data"][1] - racks = self._get_rack_data() - for rack in racks: - if rack not in rackwise_hosts: - rackwise_hosts[racks[rack]] = [] - for host in hostnames: - if rack in host: - rackwise_hosts[racks[rack]].append(host) - LOG.debug("rackwise hosts:\n%s", pprint.pformat(rackwise_hosts)) - return rackwise_hosts - - def _get_rack_data(self): - """Format rack name""" - - LOG.info("Getting rack data") - racks = {} - hostnames = self.parsed_xl_data["ipmi_data"][1] - for host in hostnames: - rack = self._get_rack(host) - racks[rack] = rack.replace("r", "rack") - return racks diff --git a/spyglass/examples/SiteDesignSpec_v0.1.xlsx b/spyglass/examples/SiteDesignSpec_v0.1.xlsx deleted file mode 100644 index cdf8278..0000000 Binary files a/spyglass/examples/SiteDesignSpec_v0.1.xlsx and /dev/null differ diff --git a/spyglass/examples/excel_spec.yaml b/spyglass/examples/excel_spec.yaml deleted file mode 100644 index 694738e..0000000 --- a/spyglass/examples/excel_spec.yaml +++ /dev/null @@ -1,63 +0,0 @@ -# Copyright 2018 The Openstack-Helm Authors. -# Copyright (c) 2018 AT&T Intellectual Property. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# Important: Please modify the dictionary with appropriate -# design spec file. ---- -specs: - # Design Spec file name: SiteDesignSpec_v0.1.xlsx - xl_spec: - ipmi_sheet_name: 'Site-Information' - start_row: 4 - end_row: 15 - hostname_col: 2 - ipmi_address_col: 3 - host_profile_col: 5 - ipmi_gateway_col: 4 - private_ip_sheet: 'Site-Information' - net_type_col: 1 - vlan_col: 2 - vlan_start_row: 19 - vlan_end_row: 30 - net_start_row: 33 - net_end_row: 40 - net_col: 2 - net_vlan_col: 1 - public_ip_sheet: 'Site-Information' - oam_vlan_col: 1 - oam_ip_row: 43 - oam_ip_col: 2 - oob_net_row: 48 - oob_net_start_col: 2 - oob_net_end_col: 5 - ingress_ip_row: 45 - dns_ntp_ldap_sheet: 'Site-Information' - login_domain_row: 52 - ldap_col: 2 - global_group: 53 - ldap_search_url_row: 54 - ntp_row: 55 - ntp_col: 2 - dns_row: 56 - dns_col: 2 - domain_row: 51 - domain_col: 2 - location_sheet: 'Site-Information' - column: 2 - corridor_row: 59 - site_name_row: 58 - state_name_row: 60 - country_name_row: 61 - clli_name_row: 62 diff --git a/spyglass/examples/site_config.yaml b/spyglass/examples/site_config.yaml deleted file mode 100644 index 25fa990..0000000 --- a/spyglass/examples/site_config.yaml +++ /dev/null @@ -1,33 +0,0 @@ -################################## -# Site Specific Tugboat Settings # -################################## ---- -site_info: - ldap: - common_name: test - url: ldap://ldap.example.com - subdomain: test - ntp: - servers: 10.10.10.10,20.20.20.20,30.30.30.30 - sitetype: foundry - domain: atlantafoundry.com - dns: - servers: 8.8.8.8,8.8.4.4,208.67.222.222 -network: - vlan_network_data: - ingress: - subnet: - - 132.68.226.72/29 - bgp : - peers: - - '172.29.0.2' - - '172.29.0.3' - asnumber: 64671 - peer_asnumber: 64688 -storage: - ceph: - controller: - osd_count: 6 -... - -