From a002e4203de6480a5afa19195e353c913393e044 Mon Sep 17 00:00:00 2001 From: Ian Pittwood Date: Wed, 17 Apr 2019 15:55:21 -0500 Subject: [PATCH] Separate plugins from Spyglass This change removes plugins from Spyglass and places them in separate repositories. Formation, a proprietary plugin, will be removed by this change and Tugboat will become its own OpenDev maintained repo, spyglass-plugin-xls. By creating more streamlined plugin management, end users should be able to more easily create their own plugins for different data sources. Related change https://review.opendev.org/#/c/659116/ Depends-On: Ib2f75878b1a29e835cb8e2323aebe9d431c479e7 Change-Id: Ie0eb2e5aefe6bb764e1aa608e53371adaabb9a17 --- doc/source/getting_started.rst | 9 +- doc/source/index.rst | 1 - doc/source/tugboat.rst | 108 ---- requirements.txt | 3 + setup.cfg | 5 +- spyglass/cli.py | 177 +----- spyglass/data_extractor/plugins/__init__.py | 0 .../plugins/formation/__init__.py | 0 .../plugins/formation/formation.py | 508 ------------------ .../plugins/tugboat/__init__.py | 0 .../plugins/tugboat/check_exceptions.py | 38 -- .../plugins/tugboat/excel_parser.py | 417 -------------- .../data_extractor/plugins/tugboat/tugboat.py | 357 ------------ spyglass/examples/SiteDesignSpec_v0.1.xlsx | Bin 17291 -> 0 bytes spyglass/examples/excel_spec.yaml | 63 --- spyglass/examples/site_config.yaml | 33 -- 16 files changed, 17 insertions(+), 1702 deletions(-) delete mode 100644 doc/source/tugboat.rst delete mode 100644 spyglass/data_extractor/plugins/__init__.py delete mode 100644 spyglass/data_extractor/plugins/formation/__init__.py delete mode 100755 spyglass/data_extractor/plugins/formation/formation.py delete mode 100644 spyglass/data_extractor/plugins/tugboat/__init__.py delete mode 100755 spyglass/data_extractor/plugins/tugboat/excel_parser.py delete mode 100755 spyglass/data_extractor/plugins/tugboat/tugboat.py delete mode 100644 spyglass/examples/SiteDesignSpec_v0.1.xlsx delete mode 100644 spyglass/examples/excel_spec.yaml delete mode 100644 spyglass/examples/site_config.yaml diff --git a/doc/source/getting_started.rst b/doc/source/getting_started.rst index ca0645d..07e546a 100644 --- a/doc/source/getting_started.rst +++ b/doc/source/getting_started.rst @@ -65,12 +65,7 @@ Architecture Supported Features ------------------ -1. Tugboat Plugin: Supports extracting site data from Excel files and - then generate site manifests for sitetype:airship-seaworthy. - Find more documentation for Tugboat, see :ref:`tugboatinfo`. - -2. Remote Data Source Plugin: Supports extracting site data from a REST - endpoint. +1. Spyglass XLS Plugin: https://opendev.org/airship/spyglass-plugin-xls Future Work ----------- @@ -135,4 +130,4 @@ Before using Spyglass you must: .. code-block:: console - pip3 install -r airship-spyglass/requirements.txt + pip3 install -r spyglass/requirements.txt diff --git a/doc/source/index.rst b/doc/source/index.rst index 5a528c1..e802f35 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -34,4 +34,3 @@ fed to Shipyard for site deployment / updates. getting_started developer_quickstart cli - tugboat diff --git a/doc/source/tugboat.rst b/doc/source/tugboat.rst deleted file mode 100644 index 2adf9f1..0000000 --- a/doc/source/tugboat.rst +++ /dev/null @@ -1,108 +0,0 @@ -.. - Copyright 2019 AT&T Intellectual Property. - All Rights Reserved. - - Licensed under the Apache License, Version 2.0 (the "License"); you may - not use this file except in compliance with the License. You may obtain - a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, WITHOUT - WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the - License for the specific language governing permissions and limitations - under the License. - -.. _tugboatinfo: - -======= -Tugboat -======= - -What is Tugboat? ----------------- - -Tugboat is a Spyglass plugin to generate airship-seaworthy site manifest files -from an excel based engineering spec. The plugin is configured with an Excel -sheet and its corresponding excel specification as inputs. Spyglass uses this -plugin to construct an intermediary yaml which is processed further using J2 -templates to generate site manifests. - -Excel specification -------------------- -Excel Spec is like an index to the Excel sheet to look for the data to be -collected by the tool. Excel Spec Sample specifies all the details that -need to be filled by the Deployment Engineer. - -Below is the definition for each key in the Excel spec - -* ipmi_sheet_name - name of the sheet from where IPMI and host profile - information is to be read -* start_row - row number from where the IPMI and host profile information - starts -* end_row - row number from where the IPMI and host profile information ends -* hostname_col - column number where the hostnames are to be read from -* ipmi_address_col - column number from where the ipmi addresses are to be read -* host_profile_col - column number from where the host profiles are to be read -* ipmi_gateway_col - column number from where the ipmi gateways are to be read -* private_ip_sheet - name of the sheet which has the private IP information -* net_type_col - column number from where the network type is to be read -* vlan_col - column number from where the network vlan is to be read -* vlan_start_row - row number from where the vlan information starts -* vlan_end_row - row number from where the vlan information ends -* net_start_row - row number from where the network information starts -* net_end_row - row number from where the network information ends -* net_col - column number where the IP ranges for network is to be read -* net_vlan_col - column number where the vlan information is present in the - pod wise network section -* public_ip_sheet - name of the sheet which has the public IP information -* oam_vlan_col - column number from where the OAM vlan information is to be - read from -* oam_ip_row - row number from where the OAM network information is to be read - from -* oam_ip_col - column number from where the OAM network information is to be - read from -* oob_net_row - row number which has the OOB network subnet ranges -* oob_net_start_col - column number from where the OOB network ranges start -* oob_net_end_col - column number from where the OOB network ranges end -* ingress_ip_row - row number from where the Ingress network information is to - be read from -* dns_ntp_ldap_sheet - name of the sheet which has the DNS, NTP and LDAP - information -* login_domain_row - row number which has the ldap login domain -* ldap_col - column number which has the all ldap related information -* global_group - row number which has the ldap group information -* ldap_search_url_row - row number which has the ldap url -* ntp_row - row number which has the ntp information -* ntp_col - column number which has the ntp information -* dns_row - row number which has the dns information -* dns_col - column number which has the dns information -* domain_row - row number which has the domain information -* domain_col - column number which has the domain information -* location_sheet - name of the sheet which has the location information -* column - column number which has all the information -* corridor_row - row number which has the corridor information -* site_name_row - row number which has the site name -* state_name_row - row number which has the state name -* country_name_row - row number which has the country name -* clli_name_row - row number which has CLLI information - -Example: Tugboat Plugin Usage ------------------------------ - -1. Required Input(Refer to 'spyglass/examples' folder to get these inputs) - - a) Excel File: SiteDesignSpec_v0.1.xlsx - b) Excel Spec: excel_spec_upstream.yaml - c) Site Config: site_config.yaml - d) Template_dir: '../examples/templates' - e) Site name: airship-seaworthy - -2. Spyglass CLI Command: - -.. code-block:: bash - - spyglass m -i -p tugboat -x SiteDesignSpec_v0.1.xlsx \ - -e excel_spec_upstream.yaml -c site_config.yaml \ - -s airship-seaworthy -t \ No newline at end of file diff --git a/requirements.txt b/requirements.txt index f759109..a8b1656 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,7 +1,10 @@ click==7.0 +click-plugins==1.1.1 jinja2==2.10 jsonschema==3.0.1 openpyxl==2.5.4 netaddr==0.7.19 pyyaml==5.1 requests==2.21.0 + +git+https://opendev.org/airship/spyglass-plugin-xls.git#egg=spyglass-plugin-xls diff --git a/setup.cfg b/setup.cfg index 528f1dc..ebf9d75 100644 --- a/setup.cfg +++ b/setup.cfg @@ -27,8 +27,9 @@ packages = console_scripts = spyglass = spyglass.cli:main data_extractor_plugins = - tugboat = spyglass.data_extractor.plugins.tugboat.tugboat:TugboatPlugin - formation = spyglass.data_extractor.plugins.formation.formation:FormationPlugin + excel = spyglass_plugin_xls.excel:ExcelPlugin +cli_plugins = + excel = spyglass_plugin_xls.cli:excel [yapf] based_on_style = pep8 diff --git a/spyglass/cli.py b/spyglass/cli.py index a473ec1..81369d8 100644 --- a/spyglass/cli.py +++ b/spyglass/cli.py @@ -16,6 +16,7 @@ import logging import pprint import click +from click_plugins import with_plugins import pkg_resources import yaml @@ -31,96 +32,6 @@ CONTEXT_SETTINGS = { 'help_option_names': ['-h', '--help'], } - -def tugboat_required_callback(ctx, param, value): - LOG.debug('Evaluating %s: %s', param.name, value) - if 'plugin_type' not in ctx.params or \ - ctx.params['plugin_type'] == 'tugboat': - if not value: - raise click.UsageError( - '%s is required for the tugboat ' - 'plugin.' % str(param.name), - ctx=ctx) - return value - - -def formation_required_callback(ctx, param, value): - LOG.debug('Evaluating %s: %s', param.name, value) - if 'plugin_type' in ctx.params: - if ctx.params['plugin_type'] == 'formation': - if not value: - raise click.UsageError( - '%s is required for the ' - 'formation plugin.' % str(param.name), - ctx=ctx) - return value - return ['', '', ''] - - -PLUGIN_TYPE_OPTION = click.option( - '-p', - '--plugin-type', - 'plugin_type', - type=click.Choice(['formation', 'tugboat']), - default='tugboat', - show_default=True, - help='The plugin type to use.') - -# TODO(ianp): Either provide a prompt for passwords or use environment -# variable so passwords are no longer plain text -FORMATION_TARGET_OPTION = click.option( - '-f', - '--formation-target', - 'formation_target', - nargs=3, - help=( - 'Target URL, username, and password for formation plugin. Required ' - 'for formation plugin.'), - callback=formation_required_callback) - -INTERMEDIARY_DIR_OPTION = click.option( - '-d', - '--intermediary-dir', - 'intermediary_dir', - type=click.Path(exists=True, file_okay=False, writable=True), - default='./', - help='Directory in which the intermediary file will be created.') - -EXCEL_FILE_OPTION = click.option( - '-x', - '--excel-file', - 'excel_file', - multiple=True, - type=click.Path(exists=True, readable=True, dir_okay=False), - help='Path to the engineering Excel file. Required for tugboat plugin.', - callback=tugboat_required_callback) - -EXCEL_SPEC_OPTION = click.option( - '-e', - '--excel-spec', - 'excel_spec', - type=click.Path(exists=True, readable=True, dir_okay=False), - help=( - 'Path to the Excel specification YAML file for the engineering ' - 'Excel file. Required for tugboat plugin.'), - callback=tugboat_required_callback) - -SITE_CONFIGURATION_FILE_OPTION = click.option( - '-c', - '--site-configuration', - 'site_configuration', - type=click.Path(exists=True, readable=True, dir_okay=False), - required=False, - help='Path to site specific configuration details YAML file.') - -SITE_NAME_CONFIGURATION_OPTION = click.option( - '-s', - '--site-name', - 'site_name', - type=click.STRING, - required=False, - help='Name of the site for which the intermediary is being generated.') - TEMPLATE_DIR_OPTION = click.option( '-t', '--template-dir', @@ -138,13 +49,14 @@ MANIFEST_DIR_OPTION = click.option( help='Path to place created manifest files.') -@click.group(context_settings=CONTEXT_SETTINGS) @click.option( '-v', '--verbose', is_flag=True, default=False, help='Enable debug messages in log.') +@with_plugins(pkg_resources.iter_entry_points('cli_plugins')) +@click.group() def main(*, verbose): """CLI for Airship Spyglass""" if verbose: @@ -154,9 +66,7 @@ def main(*, verbose): logging.basicConfig(format=LOG_FORMAT, level=log_level) -def _intermediary_helper( - plugin_type, formation_data, site, excel_file, excel_spec, - additional_configuration): +def intermediary_processor(plugin_type, **kwargs): LOG.info("Generating Intermediary yaml") plugin_type = plugin_type plugin_class = None @@ -165,6 +75,7 @@ def _intermediary_helper( LOG.info("Load the plugin class") for entry_point in \ pkg_resources.iter_entry_points('data_extractor_plugins'): + LOG.debug("Entry point '%s' found", entry_point.name) if entry_point.name == plugin_type: plugin_class = entry_point.load() @@ -175,20 +86,13 @@ def _intermediary_helper( # Extract data from plugin data source LOG.info("Extract data from plugin data source") - data_extractor = plugin_class(site) - plugin_conf = data_extractor.get_plugin_conf( - { - 'excel': excel_file, - 'excel_spec': excel_spec, - 'formation_url': formation_data[0], - 'formation_user': formation_data[1], - 'formation_password': formation_data[2] - }) + data_extractor = plugin_class(kwargs['site_name']) + plugin_conf = data_extractor.get_plugin_conf(**kwargs) data_extractor.set_config_opts(plugin_conf) data_extractor.extract_data() # Apply any additional_config provided by user - additional_config = additional_configuration + additional_config = kwargs.get('site_configuration', None) if additional_config is not None: with open(additional_config, 'r') as config: raw_data = config.read() @@ -204,75 +108,12 @@ def _intermediary_helper( # Apply design rules to the data LOG.info("Apply design rules to the extracted data") - process_input_ob = ProcessDataSource(site) + process_input_ob = ProcessDataSource(kwargs['site_name']) process_input_ob.load_extracted_data_from_data_source( data_extractor.site_data) return process_input_ob -@main.command( - 'i', - short_help='generate intermediary', - help='Generates an intermediary file from passed excel data.') -@PLUGIN_TYPE_OPTION -@FORMATION_TARGET_OPTION -@INTERMEDIARY_DIR_OPTION -@EXCEL_FILE_OPTION -@EXCEL_SPEC_OPTION -@SITE_CONFIGURATION_FILE_OPTION -@SITE_NAME_CONFIGURATION_OPTION -def generate_intermediary( - *, plugin_type, formation_target, intermediary_dir, excel_file, - excel_spec, site_configuration, site_name): - process_input_ob = _intermediary_helper( - plugin_type, formation_target, site_name, excel_file, excel_spec, - site_configuration) - LOG.info("Generate intermediary yaml") - process_input_ob.generate_intermediary_yaml() - process_input_ob.dump_intermediary_file(intermediary_dir) - - -@main.command( - 'm', - short_help='generates manifest and intermediary', - help='Generates manifest and intermediary files.') -@click.option( - '-i', - '--save-intermediary', - 'save_intermediary', - is_flag=True, - default=False, - help='Flag to save the generated intermediary file used for the manifests.' -) -@PLUGIN_TYPE_OPTION -@FORMATION_TARGET_OPTION -@INTERMEDIARY_DIR_OPTION -@EXCEL_FILE_OPTION -@EXCEL_SPEC_OPTION -@SITE_CONFIGURATION_FILE_OPTION -@SITE_NAME_CONFIGURATION_OPTION -@TEMPLATE_DIR_OPTION -@MANIFEST_DIR_OPTION -def generate_manifests_and_intermediary( - *, save_intermediary, plugin_type, formation_target, intermediary_dir, - excel_file, excel_spec, site_configuration, site_name, template_dir, - manifest_dir): - process_input_ob = _intermediary_helper( - plugin_type, formation_target, site_name, excel_file, excel_spec, - site_configuration) - LOG.info("Generate intermediary yaml") - intermediary_yaml = process_input_ob.generate_intermediary_yaml() - if save_intermediary: - LOG.debug("Dumping intermediary yaml") - process_input_ob.dump_intermediary_file(intermediary_dir) - else: - LOG.debug("Skipping dump for intermediary yaml") - - LOG.info("Generating site Manifests") - processor_engine = SiteProcessor(intermediary_yaml, manifest_dir) - processor_engine.render_template(template_dir) - - @main.command( 'mi', short_help='generates manifest from intermediary', diff --git a/spyglass/data_extractor/plugins/__init__.py b/spyglass/data_extractor/plugins/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/spyglass/data_extractor/plugins/formation/__init__.py b/spyglass/data_extractor/plugins/formation/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/spyglass/data_extractor/plugins/formation/formation.py b/spyglass/data_extractor/plugins/formation/formation.py deleted file mode 100755 index 1f80379..0000000 --- a/spyglass/data_extractor/plugins/formation/formation.py +++ /dev/null @@ -1,508 +0,0 @@ -# Copyright 2018 AT&T Intellectual Property. All other rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -import logging -import pprint -import re - -import formation_client -import requests -import urllib3 - -from spyglass.data_extractor.base import BaseDataSourcePlugin -import spyglass.data_extractor.custom_exceptions as exceptions - -urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) - -LOG = logging.getLogger(__name__) - - -class FormationPlugin(BaseDataSourcePlugin): - - def __init__(self, region): - # Save site name is valid - if not region: - LOG.error("Site: None! Spyglass exited!") - LOG.info("Check spyglass --help for details") - exit() - super().__init__(region) - - self.source_type = "rest" - self.source_name = "formation" - - # Configuration parameters - self.formation_api_url = None - self.user = None - self.password = None - self.token = None - - # Formation objects - self.client_config = None - self.formation_api_client = None - - # Site related data - self.region_zone_map = {} - self.site_name_id_mapping = {} - self.zone_name_id_mapping = {} - self.region_name_id_mapping = {} - self.rack_name_id_mapping = {} - self.device_name_id_mapping = {} - LOG.info("Initiated data extractor plugin:{}".format(self.source_name)) - - def set_config_opts(self, conf): - """Sets the config params passed by CLI""" - - LOG.info("Plugin params passed:\n{}".format(pprint.pformat(conf))) - self._validate_config_options(conf) - self.formation_api_url = conf["url"] - self.user = conf["user"] - self.password = conf["password"] - self.token = conf.get("token", None) - - self._get_formation_client() - self._update_site_and_zone(self.region) - - def get_plugin_conf(self, kwargs): - """Validates the plugin param and return if success""" - - if not kwargs["formation_url"]: - LOG.error("formation_url not specified! Spyglass exited!") - exit() - url = kwargs["formation_url"] - - if not kwargs["formation_user"]: - LOG.error("formation_user not specified! Spyglass exited!") - exit() - user = kwargs["formation_user"] - - if not kwargs["formation_password"]: - LOG.error("formation_password not specified! Spyglass exited!") - exit() - password = kwargs['formation_password'] - - plugin_conf = {"url": url, "user": user, "password": password} - return plugin_conf - - def _validate_config_options(self, conf): - """Validate the CLI params passed - - The method checks for missing parameters and terminates - Spyglass execution if found so. - """ - - missing_params = [] - for key in conf.keys(): - if conf[key] is None: - missing_params.append(key) - if len(missing_params) != 0: - LOG.error("Missing Plugin Params{}:".format(missing_params)) - exit() - - # Implement helper classes - - def _generate_token(self): - """Generate token for Formation - - Formation API does not provide separate resource to generate - token. This is a workaround to call directly Formation API - to get token instead of using Formation client. - """ - - # Create formation client config object - self.client_config = formation_client.Configuration() - self.client_config.host = self.formation_api_url - self.client_config.username = self.user - self.client_config.password = self.password - self.client_config.verify_ssl = False - - # Assumes token is never expired in the execution of this tool - if self.token: - return self.token - - url = self.formation_api_url + "/zones" - try: - token_response = requests.get( - url, - auth=(self.user, self.password), - verify=self.client_config.verify_ssl, - ) - except requests.exceptions.ConnectionError: - raise exceptions.FormationConnectionError( - "Incorrect URL: {}".format(url)) - - if token_response.status_code == 200: - self.token = token_response.json().get("X-Subject-Token", None) - else: - raise exceptions.TokenGenerationError( - "Unable to generate token because {}".format( - token_response.reason)) - - return self.token - - def _get_formation_client(self): - """Create formation client object - - Formation uses X-Auth-Token for authentication and should be in - format "user|token". - Generate the token and add it formation config object. - """ - - token = self._generate_token() - self.client_config.api_key = {"X-Auth-Token": self.user + "|" + token} - self.formation_api_client = \ - formation_client.ApiClient(self.client_config) - - def _update_site_and_zone(self, region): - """Get Zone name and Site name from region""" - - zone = self._get_zone_by_region_name(region) - site = self._get_site_by_zone_name(zone) - - # zone = region[:-1] - # site = zone[:-1] - - self.region_zone_map[region] = {} - self.region_zone_map[region]["zone"] = zone - self.region_zone_map[region]["site"] = site - - def _get_zone_by_region_name(self, region_name): - zone_api = formation_client.ZonesApi(self.formation_api_client) - zones = zone_api.zones_get() - - # Walk through each zone and get regions - # Return when region name matches - for zone in zones: - self.zone_name_id_mapping[zone.name] = zone.id - zone_regions = self.get_regions(zone.name) - if region_name in zone_regions: - return zone.name - - return None - - def _get_site_by_zone_name(self, zone_name): - site_api = formation_client.SitesApi(self.formation_api_client) - sites = site_api.sites_get() - - # Walk through each site and get zones - # Return when site name matches - for site in sites: - self.site_name_id_mapping[site.name] = site.id - site_zones = self.get_zones(site.name) - if zone_name in site_zones: - return site.name - - return None - - def _get_site_id_by_name(self, site_name): - if site_name in self.site_name_id_mapping: - return self.site_name_id_mapping.get(site_name) - - site_api = formation_client.SitesApi(self.formation_api_client) - sites = site_api.sites_get() - for site in sites: - self.site_name_id_mapping[site.name] = site.id - if site.name == site_name: - return site.id - - def _get_zone_id_by_name(self, zone_name): - if zone_name in self.zone_name_id_mapping: - return self.zone_name_id_mapping.get(zone_name) - - zone_api = formation_client.ZonesApi(self.formation_api_client) - zones = zone_api.zones_get() - for zone in zones: - if zone.name == zone_name: - self.zone_name_id_mapping[zone.name] = zone.id - return zone.id - - def _get_region_id_by_name(self, region_name): - if region_name in self.region_name_id_mapping: - return self.region_name_id_mapping.get(region_name) - - for zone in self.zone_name_id_mapping: - self.get_regions(zone) - - return self.region_name_id_mapping.get(region_name, None) - - def _get_rack_id_by_name(self, rack_name): - if rack_name in self.rack_name_id_mapping: - return self.rack_name_id_mapping.get(rack_name) - - for zone in self.zone_name_id_mapping: - self.get_racks(zone) - - return self.rack_name_id_mapping.get(rack_name, None) - - def _get_device_id_by_name(self, device_name): - if device_name in self.device_name_id_mapping: - return self.device_name_id_mapping.get(device_name) - - self.get_hosts(self.zone) - - return self.device_name_id_mapping.get(device_name, None) - - def _get_racks(self, zone, rack_type="compute"): - zone_id = self._get_zone_id_by_name(zone) - rack_api = formation_client.RacksApi(self.formation_api_client) - racks = rack_api.zones_zone_id_racks_get(zone_id) - - racks_list = [] - for rack in racks: - rack_name = rack.name - self.rack_name_id_mapping[rack_name] = rack.id - if rack.rack_type.name == rack_type: - racks_list.append(rack_name) - - return racks_list - - # Functions that will be used internally within this plugin - - def get_zones(self, site=None): - zone_api = formation_client.ZonesApi(self.formation_api_client) - - if site is None: - zones = zone_api.zones_get() - else: - site_id = self._get_site_id_by_name(site) - zones = zone_api.sites_site_id_zones_get(site_id) - - zones_list = [] - for zone in zones: - zone_name = zone.name - self.zone_name_id_mapping[zone_name] = zone.id - zones_list.append(zone_name) - - return zones_list - - def get_regions(self, zone): - zone_id = self._get_zone_id_by_name(zone) - region_api = formation_client.RegionApi(self.formation_api_client) - regions = region_api.zones_zone_id_regions_get(zone_id) - regions_list = [] - for region in regions: - region_name = region.name - self.region_name_id_mapping[region_name] = region.id - regions_list.append(region_name) - - return regions_list - - # Implement Abstract functions - - def get_racks(self, region): - zone = self.region_zone_map[region]["zone"] - return self._get_racks(zone, rack_type="compute") - - def get_hosts(self, region, rack=None): - zone = self.region_zone_map[region]["zone"] - zone_id = self._get_zone_id_by_name(zone) - device_api = formation_client.DevicesApi(self.formation_api_client) - control_hosts = device_api.zones_zone_id_control_nodes_get(zone_id) - compute_hosts = device_api.zones_zone_id_devices_get( - zone_id, type="KVM") - - hosts_list = [] - for host in control_hosts: - self.device_name_id_mapping[host.aic_standard_name] = host.id - hosts_list.append( - { - "name": host.aic_standard_name, - "type": "controller", - "rack_name": host.rack_name, - "host_profile": host.host_profile_name, - }) - - for host in compute_hosts: - self.device_name_id_mapping[host.aic_standard_name] = host.id - hosts_list.append( - { - "name": host.aic_standard_name, - "type": "compute", - "rack_name": host.rack_name, - "host_profile": host.host_profile_name, - }) - """ - for host in itertools.chain(control_hosts, compute_hosts): - self.device_name_id_mapping[host.aic_standard_name] = host.id - hosts_list.append({ - 'name': host.aic_standard_name, - 'type': host.categories[0], - 'rack_name': host.rack_name, - 'host_profile': host.host_profile_name - }) - """ - - return hosts_list - - def get_networks(self, region): - zone = self.region_zone_map[region]["zone"] - zone_id = self._get_zone_id_by_name(zone) - region_id = self._get_region_id_by_name(region) - vlan_api = formation_client.VlansApi(self.formation_api_client) - vlans = vlan_api.zones_zone_id_regions_region_id_vlans_get( - zone_id, region_id) - - # Case when vlans list is empty from - # zones_zone_id_regions_region_id_vlans_get - if len(vlans) == 0: - # get device-id from the first host and get the network details - hosts = self.get_hosts(self.region) - host = hosts[0]["name"] - device_id = self._get_device_id_by_name(host) - vlans = \ - vlan_api.zones_zone_id_devices_device_id_vlans_get(zone_id, - device_id) - - LOG.debug("Extracted region network information\n{}".format(vlans)) - vlans_list = [] - for vlan_ in vlans: - if len(vlan_.vlan.ipv4) != 0: - tmp_vlan = { - "name": self._get_network_name_from_vlan_name( - vlan_.vlan.name), - "vlan": vlan_.vlan.vlan_id, - "subnet": vlan_.vlan.subnet_range, - "gateway": vlan_.ipv4_gateway, - "subnet_level": vlan_.vlan.subnet_level - } - vlans_list.append(tmp_vlan) - - return vlans_list - - def get_ips(self, region, host=None): - zone = self.region_zone_map[region]["zone"] - zone_id = self._get_zone_id_by_name(zone) - - if host: - hosts = [host] - else: - hosts = [] - hosts_dict = self.get_hosts(zone) - for host in hosts_dict: - hosts.append(host["name"]) - - vlan_api = formation_client.VlansApi(self.formation_api_client) - ip_ = {} - - for host in hosts: - device_id = self._get_device_id_by_name(host) - vlans = \ - vlan_api.zones_zone_id_devices_device_id_vlans_get(zone_id, - device_id) - LOG.debug("Received VLAN Network Information\n{}".format(vlans)) - ip_[host] = {} - for vlan_ in vlans: - # TODO(pg710r) We need to handle the case when incoming ipv4 - # list is empty - if len(vlan_.vlan.ipv4) != 0: - name = self._get_network_name_from_vlan_name( - vlan_.vlan.name) - ipv4 = vlan_.vlan.ipv4[0].ip - LOG.debug( - "vlan:{},name:{},ip:{},vlan_name:{}".format( - vlan_.vlan.vlan_id, name, ipv4, vlan_.vlan.name)) - # TODD(pg710r) This code needs to extended to support ipv4 - # and ipv6 - # ip_[host][name] = {'ipv4': ipv4} - ip_[host][name] = ipv4 - - return ip_ - - def _get_network_name_from_vlan_name(self, vlan_name): - """Network names are ksn, oam, oob, overlay, storage, pxe - - The following mapping rules apply: - vlan_name contains "ksn" the network name is "calico" - vlan_name contains "storage" the network name is "storage" - vlan_name contains "server" the network name is "oam" - vlan_name contains "ovs" the network name is "overlay" - vlan_name contains "ILO" the network name is "oob" - """ - - network_names = { - "ksn": "calico", - "storage": "storage", - "server": "oam", - "ovs": "overlay", - "ILO": "oob", - "pxe": "pxe", - } - - for name in network_names: - # Make a pattern that would ignore case. - # if name is 'ksn' pattern name is '(?i)(ksn)' - name_pattern = "(?i)({})".format(name) - if re.search(name_pattern, vlan_name): - return network_names[name] - # Return empty string is vlan_name is not matched with network_names - return "" - - def get_dns_servers(self, region): - try: - zone = self.region_zone_map[region]["zone"] - zone_id = self._get_zone_id_by_name(zone) - zone_api = formation_client.ZonesApi(self.formation_api_client) - zone_ = zone_api.zones_zone_id_get(zone_id) - except formation_client.rest.ApiException as e: - raise exceptions.ApiClientError(e.msg) - - if not zone_.ipv4_dns: - LOG.warning("No dns server") - return [] - - dns_list = [] - for dns in zone_.ipv4_dns: - dns_list.append(dns.ip) - - return dns_list - - def get_ntp_servers(self, region): - return [] - - def get_ldap_information(self, region): - return {} - - def get_location_information(self, region): - """Get location information for a zone and return""" - - site = self.region_zone_map[region]["site"] - site_id = self._get_site_id_by_name(site) - site_api = formation_client.SitesApi(self.formation_api_client) - site_info = site_api.sites_site_id_get(site_id) - - try: - return { - # 'corridor': site_info.corridor, - "name": site_info.city, - "state": site_info.state, - "country": site_info.country, - "physical_location_id": site_info.clli, - } - except AttributeError as e: - raise exceptions.MissingAttributeError( - "Missing {} information in {}".format(e, site_info.city)) - - def get_domain_name(self, region): - try: - zone = self.region_zone_map[region]["zone"] - zone_id = self._get_zone_id_by_name(zone) - zone_api = formation_client.ZonesApi(self.formation_api_client) - zone_ = zone_api.zones_zone_id_get(zone_id) - except formation_client.rest.ApiException as e: - raise exceptions.ApiClientError(e.msg) - - if not zone_.dns: - LOG.warning("Got None while running get domain name") - return None - - return zone_.dns diff --git a/spyglass/data_extractor/plugins/tugboat/__init__.py b/spyglass/data_extractor/plugins/tugboat/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/spyglass/data_extractor/plugins/tugboat/check_exceptions.py b/spyglass/data_extractor/plugins/tugboat/check_exceptions.py index 49ac3b4..e69de29 100644 --- a/spyglass/data_extractor/plugins/tugboat/check_exceptions.py +++ b/spyglass/data_extractor/plugins/tugboat/check_exceptions.py @@ -1,38 +0,0 @@ -# Copyright 2018 AT&T Intellectual Property. All other rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - - -class BaseError(Exception): - pass - - -class NotEnoughIp(BaseError): - - def __init__(self, cidr, total_nodes): - self.cidr = cidr - self.total_nodes = total_nodes - - def display_error(self): - print("{} can not handle {} nodes".format(self.cidr, self.total_nodes)) - - -class NoSpecMatched(BaseError): - - def __init__(self, excel_specs): - self.specs = excel_specs - - def display_error(self): - print( - "No spec matched. Following are the available specs:\n".format( - self.specs)) diff --git a/spyglass/data_extractor/plugins/tugboat/excel_parser.py b/spyglass/data_extractor/plugins/tugboat/excel_parser.py deleted file mode 100755 index 3f4d1c7..0000000 --- a/spyglass/data_extractor/plugins/tugboat/excel_parser.py +++ /dev/null @@ -1,417 +0,0 @@ -# Copyright 2018 AT&T Intellectual Property. All other rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -import logging -import pprint -import re -import sys - -from openpyxl import load_workbook -from openpyxl import Workbook -import yaml - -from spyglass.data_extractor.custom_exceptions import NoSpecMatched - -LOG = logging.getLogger(__name__) - - -class ExcelParser(object): - """Parse data from excel into a dict""" - - def __init__(self, file_name, excel_specs): - self.file_name = file_name - with open(excel_specs, "r") as f: - spec_raw_data = f.read() - self.excel_specs = yaml.safe_load(spec_raw_data) - # A combined design spec, returns a workbook object after combining - # all the inputs excel specs - combined_design_spec = self.combine_excel_design_specs(file_name) - self.wb_combined = combined_design_spec - self.filenames = file_name - self.spec = "xl_spec" - - @staticmethod - def sanitize(string): - """Remove extra spaces and convert string to lower case""" - - return string.replace(" ", "").lower() - - def compare(self, string1, string2): - """Compare the strings""" - - return bool(re.search(self.sanitize(string1), self.sanitize(string2))) - - def validate_sheet(self, spec, sheet): - """Check if the sheet is correct or not""" - - ws = self.wb_combined[sheet] - header_row = self.excel_specs["specs"][spec]["header_row"] - ipmi_header = self.excel_specs["specs"][spec]["ipmi_address_header"] - ipmi_column = self.excel_specs["specs"][spec]["ipmi_address_col"] - header_value = ws.cell(row=header_row, column=ipmi_column).value - return bool(self.compare(ipmi_header, header_value)) - - def find_correct_spec(self): - """Find the correct spec""" - - for spec in self.excel_specs["specs"]: - sheet_name = self.excel_specs["specs"][spec]["ipmi_sheet_name"] - for sheet in self.wb_combined.sheetnames: - if self.compare(sheet_name, sheet): - self.excel_specs["specs"][spec]["ipmi_sheet_name"] = sheet - if self.validate_sheet(spec, sheet): - return spec - raise NoSpecMatched(self.excel_specs) - - def get_ipmi_data(self): - """Read IPMI data from the sheet""" - - ipmi_data = {} - hosts = [] - spec_ = self.excel_specs["specs"][self.spec] - provided_sheetname = spec_["ipmi_sheet_name"] - workbook_object, extracted_sheetname = \ - self.get_xl_obj_and_sheetname(provided_sheetname) - if workbook_object is not None: - ws = workbook_object[extracted_sheetname] - else: - ws = self.wb_combined[provided_sheetname] - row = spec_["start_row"] - end_row = spec_["end_row"] - hostname_col = spec_["hostname_col"] - ipmi_address_col = spec_["ipmi_address_col"] - host_profile_col = spec_["host_profile_col"] - ipmi_gateway_col = spec_["ipmi_gateway_col"] - previous_server_gateway = None - while row <= end_row: - hostname = \ - self.sanitize(ws.cell(row=row, column=hostname_col).value) - hosts.append(hostname) - ipmi_address = ws.cell(row=row, column=ipmi_address_col).value - if "/" in ipmi_address: - ipmi_address = ipmi_address.split("/")[0] - ipmi_gateway = ws.cell(row=row, column=ipmi_gateway_col).value - if ipmi_gateway: - previous_server_gateway = ipmi_gateway - else: - ipmi_gateway = previous_server_gateway - host_profile = ws.cell(row=row, column=host_profile_col).value - try: - if host_profile is None: - raise RuntimeError( - "No value read from " - "{} sheet:{} row:{}, col:{}".format( - self.file_name, self.spec, row, host_profile_col)) - except RuntimeError as rerror: - LOG.critical(rerror) - sys.exit("Tugboat exited!!") - ipmi_data[hostname] = { - "ipmi_address": ipmi_address, - "ipmi_gateway": ipmi_gateway, - "host_profile": host_profile, - "type": type, # FIXME (Ian Pittwood): shadows type built-in - } - row += 1 - LOG.debug( - "ipmi data extracted from excel:\n{}".format( - pprint.pformat(ipmi_data))) - LOG.debug( - "host data extracted from excel:\n{}".format( - pprint.pformat(hosts))) - return [ipmi_data, hosts] - - def get_private_vlan_data(self, ws): - """Get private vlan data from private IP sheet""" - - vlan_data = {} - row = self.excel_specs["specs"][self.spec]["vlan_start_row"] - end_row = self.excel_specs["specs"][self.spec]["vlan_end_row"] - type_col = self.excel_specs["specs"][self.spec]["net_type_col"] - vlan_col = self.excel_specs["specs"][self.spec]["vlan_col"] - while row <= end_row: - cell_value = ws.cell(row=row, column=type_col).value - if cell_value: - vlan = ws.cell(row=row, column=vlan_col).value - if vlan: - vlan = vlan.lower() - vlan_data[vlan] = cell_value - row += 1 - LOG.debug( - "vlan data extracted from excel:\n%s" % pprint.pformat(vlan_data)) - return vlan_data - - def get_private_network_data(self): - """Read network data from the private ip sheet""" - - spec_ = self.excel_specs["specs"][self.spec] - provided_sheetname = spec_["private_ip_sheet"] - workbook_object, extracted_sheetname = \ - self.get_xl_obj_and_sheetname(provided_sheetname) - if workbook_object is not None: - ws = workbook_object[extracted_sheetname] - else: - ws = self.wb_combined[provided_sheetname] - vlan_data = self.get_private_vlan_data(ws) - network_data = {} - row = spec_["net_start_row"] - end_row = spec_["net_end_row"] - col = spec_["net_col"] - vlan_col = spec_["net_vlan_col"] - old_vlan = "" - while row <= end_row: - vlan = ws.cell(row=row, column=vlan_col).value - if vlan: - vlan = vlan.lower() - network = ws.cell(row=row, column=col).value - if vlan and network: - net_type = vlan_data[vlan] - if "vlan" not in network_data: - network_data[net_type] = {"vlan": vlan, "subnet": []} - elif not vlan and network: - # If vlan is not present then assign old vlan to vlan as vlan - # value is spread over several rows - vlan = old_vlan - else: - row += 1 - continue - network_data[vlan_data[vlan]]["subnet"].append(network) - old_vlan = vlan - row += 1 - for network in network_data: - network_data[network]["is_common"] = True - """ - if len(network_data[network]['subnet']) > 1: - network_data[network]['is_common'] = False - else: - network_data[network]['is_common'] = True - LOG.debug("private network data extracted from excel:\n%s" - % pprint.pformat(network_data)) - """ - return network_data - - def get_public_network_data(self): - """Read public network data from public ip data""" - - spec_ = self.excel_specs["specs"][self.spec] - provided_sheetname = spec_["public_ip_sheet"] - workbook_object, extracted_sheetname = self.get_xl_obj_and_sheetname( - provided_sheetname) - if workbook_object is not None: - ws = workbook_object[extracted_sheetname] - else: - ws = self.wb_combined[provided_sheetname] - oam_row = spec_["oam_ip_row"] - oam_col = spec_["oam_ip_col"] - oam_vlan_col = spec_["oam_vlan_col"] - ingress_row = spec_["ingress_ip_row"] - oob_row = spec_["oob_net_row"] - col = spec_["oob_net_start_col"] - end_col = spec_["oob_net_end_col"] - network_data = { - "oam": { - "subnet": [ws.cell(row=oam_row, column=oam_col).value], - "vlan": ws.cell(row=oam_row, column=oam_vlan_col).value, - }, - "ingress": ws.cell(row=ingress_row, column=oam_col).value, - "oob": { - "subnet": [], - } - } - while col <= end_col: - cell_value = ws.cell(row=oob_row, column=col).value - if cell_value: - network_data["oob"]["subnet"].append(self.sanitize(cell_value)) - col += 1 - LOG.debug( - "public network data extracted from excel:\n%s" % - pprint.pformat(network_data)) - return network_data - - def get_site_info(self): - """Read location, dns, ntp and ldap data""" - - spec_ = self.excel_specs["specs"][self.spec] - provided_sheetname = spec_["dns_ntp_ldap_sheet"] - workbook_object, extracted_sheetname = \ - self.get_xl_obj_and_sheetname(provided_sheetname) - if workbook_object is not None: - ws = workbook_object[extracted_sheetname] - else: - ws = self.wb_combined[provided_sheetname] - dns_row = spec_["dns_row"] - dns_col = spec_["dns_col"] - ntp_row = spec_["ntp_row"] - ntp_col = spec_["ntp_col"] - domain_row = spec_["domain_row"] - domain_col = spec_["domain_col"] - login_domain_row = spec_["login_domain_row"] - ldap_col = spec_["ldap_col"] - global_group = spec_["global_group"] - ldap_search_url_row = spec_["ldap_search_url_row"] - dns_servers = ws.cell(row=dns_row, column=dns_col).value - ntp_servers = ws.cell(row=ntp_row, column=ntp_col).value - try: - if dns_servers is None: - raise RuntimeError( - "No value for dns_server from: " - "{} Sheet:'{}' Row:{} Col:{}".format( - self.file_name, provided_sheetname, dns_row, dns_col)) - if ntp_servers is None: - raise RuntimeError( - "No value for ntp_server from: " - "{} Sheet:'{}' Row:{} Col:{}".format( - self.file_name, provided_sheetname, ntp_row, ntp_col)) - except RuntimeError as rerror: - LOG.critical(rerror) - sys.exit("Tugboat exited!!") - - dns_servers = dns_servers.replace("\n", " ") - ntp_servers = ntp_servers.replace("\n", " ") - if "," in dns_servers: - dns_servers = dns_servers.split(",") - else: - dns_servers = dns_servers.split() - if "," in ntp_servers: - ntp_servers = ntp_servers.split(",") - else: - ntp_servers = ntp_servers.split() - site_info = { - "location": self.get_location_data(), - "dns": dns_servers, - "ntp": ntp_servers, - "domain": ws.cell(row=domain_row, column=domain_col).value, - "ldap": { - "subdomain": ws.cell(row=login_domain_row, - column=ldap_col).value, - "common_name": ws.cell(row=global_group, - column=ldap_col).value, - "url": ws.cell(row=ldap_search_url_row, column=ldap_col).value, - }, - } - LOG.debug( - "Site Info extracted from\ - excel:\n%s", - pprint.pformat(site_info), - ) - return site_info - - def get_location_data(self): - """Read location data from the site and zone sheet""" - - spec_ = self.excel_specs["specs"][self.spec] - provided_sheetname = spec_["location_sheet"] - workbook_object, extracted_sheetname = \ - self.get_xl_obj_and_sheetname(provided_sheetname) - if workbook_object is not None: - ws = workbook_object[extracted_sheetname] - else: - ws = self.wb_combined[provided_sheetname] - corridor_row = spec_["corridor_row"] - column = spec_["column"] - site_name_row = spec_["site_name_row"] - state_name_row = spec_["state_name_row"] - country_name_row = spec_["country_name_row"] - clli_name_row = spec_["clli_name_row"] - return { - "corridor": ws.cell(row=corridor_row, column=column).value, - "name": ws.cell(row=site_name_row, column=column).value, - "state": ws.cell(row=state_name_row, column=column).value, - "country": ws.cell(row=country_name_row, column=column).value, - "physical_location": ws.cell(row=clli_name_row, - column=column).value, - } - - def validate_sheet_names_with_spec(self): - """Checks is sheet name in spec file matches with excel file""" - - spec = list(self.excel_specs["specs"].keys())[0] - spec_item = self.excel_specs["specs"][spec] - sheet_name_list = [] - ipmi_header_sheet_name = spec_item["ipmi_sheet_name"] - sheet_name_list.append(ipmi_header_sheet_name) - private_ip_sheet_name = spec_item["private_ip_sheet"] - sheet_name_list.append(private_ip_sheet_name) - public_ip_sheet_name = spec_item["public_ip_sheet"] - sheet_name_list.append(public_ip_sheet_name) - dns_ntp_ldap_sheet_name = spec_item["dns_ntp_ldap_sheet"] - sheet_name_list.append(dns_ntp_ldap_sheet_name) - location_sheet_name = spec_item["location_sheet"] - sheet_name_list.append(location_sheet_name) - try: - for sheetname in sheet_name_list: - workbook_object, extracted_sheetname = \ - self.get_xl_obj_and_sheetname(sheetname) - if workbook_object is not None: - wb = workbook_object - sheetname = extracted_sheetname - else: - wb = self.wb_combined - - if sheetname not in wb.sheetnames: - raise RuntimeError( - "SheetName '{}' not found ".format(sheetname)) - except RuntimeError as rerror: - LOG.critical(rerror) - sys.exit("Tugboat exited!!") - - LOG.info("Sheet names in excel spec validated") - - def get_data(self): - """Create a dict with combined data""" - - self.validate_sheet_names_with_spec() - ipmi_data = self.get_ipmi_data() - network_data = self.get_private_network_data() - public_network_data = self.get_public_network_data() - site_info_data = self.get_site_info() - data = { - "ipmi_data": ipmi_data, - "network_data": { - "private": network_data, - "public": public_network_data, - }, - "site_info": site_info_data, - } - LOG.debug( - "Location data extracted from excel:\n%s" % pprint.pformat(data)) - return data - - def combine_excel_design_specs(self, filenames): - """Combines multiple excel file to a single design spec""" - - design_spec = Workbook() - for exel_file in filenames: - loaded_workbook = load_workbook(exel_file, data_only=True) - for names in loaded_workbook.sheetnames: - design_spec_worksheet = design_spec.create_sheet(names) - loaded_workbook_ws = loaded_workbook[names] - for row in loaded_workbook_ws: - for cell in row: - design_spec_worksheet[cell.coordinate].value = \ - cell.value - return design_spec - - def get_xl_obj_and_sheetname(self, sheetname): - """The logic confirms if the sheetname is specified for example as: - - 'MTN57a_AEC_Network_Design_v1.6.xlsx:Public IPs' - """ - - if re.search(".xlsx", sheetname) or re.search(".xls", sheetname): - # Extract file name - source_xl_file = sheetname.split(":")[0] - wb = load_workbook(source_xl_file, data_only=True) - return [wb, sheetname.split(":")[1]] - else: - return [None, sheetname] diff --git a/spyglass/data_extractor/plugins/tugboat/tugboat.py b/spyglass/data_extractor/plugins/tugboat/tugboat.py deleted file mode 100755 index a9e231c..0000000 --- a/spyglass/data_extractor/plugins/tugboat/tugboat.py +++ /dev/null @@ -1,357 +0,0 @@ -# Copyright 2018 AT&T Intellectual Property. All other rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -import itertools -import logging -import pprint -import re - -from spyglass.data_extractor.base import BaseDataSourcePlugin -from spyglass.data_extractor.plugins.tugboat.excel_parser import ExcelParser - -LOG = logging.getLogger(__name__) - - -class TugboatPlugin(BaseDataSourcePlugin): - - def __init__(self, region): - LOG.info("Tugboat Initializing") - self.source_type = "excel" - self.source_name = "tugboat" - - # Configuration parameters - self.excel_path = None - self.excel_spec = None - - # Site related data - self.region = region - - # Raw data from excel - self.parsed_xl_data = None - - LOG.info("Initiated data extractor plugin:{}".format(self.source_name)) - - def set_config_opts(self, conf): - """Placeholder to set configuration options specific to each plugin. - - :param dict conf: Configuration options as dict - - Example: conf = { 'excel_spec': 'spec1.yaml', - 'excel_path': 'excel.xls' } - - Each plugin will have their own config opts. - """ - - self.excel_path = conf["excel_path"] - self.excel_spec = conf["excel_spec"] - - # Extract raw data from excel sheets - self._get_excel_obj() - self._extract_raw_data_from_excel() - return - - def get_plugin_conf(self, kwargs): - """Validates the plugin param from CLI and return if correct - - Ideally the CLICK module shall report an error if excel file - and excel specs are not specified. The below code has been - written as an additional safeguard. - """ - if not kwargs["excel"]: - LOG.error("Engineering excel file not specified: Spyglass exited!") - exit() - excel_file_info = kwargs["excel"] - if not kwargs["excel_spec"]: - LOG.error("Engineering spec file not specified: Spyglass exited!") - exit() - excel_spec_info = kwargs["excel_spec"] - plugin_conf = { - "excel_path": excel_file_info, - "excel_spec": excel_spec_info, - } - return plugin_conf - - def get_hosts(self, region, rack=None): - """Return list of hosts in the region - - :param string region: Region name - :param string rack: Rack name - :returns: list of hosts information - :rtype: list of dict - Example: [ - { - 'name': 'host01', - 'type': 'controller', - 'host_profile': 'hp_01' - }, - { - 'name': 'host02', - 'type': 'compute', - 'host_profile': 'hp_02'} - ] - """ - - LOG.info("Get Host Information") - ipmi_data = self.parsed_xl_data["ipmi_data"][0] - rackwise_hosts = self._get_rackwise_hosts() - host_list = [] - for rack in rackwise_hosts.keys(): - for host in rackwise_hosts[rack]: - host_list.append( - { - "rack_name": rack, - "name": host, - "host_profile": ipmi_data[host]["host_profile"], - }) - return host_list - - def get_networks(self, region): - """Extracts vlan network info from raw network data from excel""" - - vlan_list = [] - # Network data extracted from xl is formatted to have a predictable - # data type. For e.g VlAN 45 extracted from xl is formatted as 45 - vlan_pattern = r"\d+" - private_net = self.parsed_xl_data["network_data"]["private"] - public_net = self.parsed_xl_data["network_data"]["public"] - # Extract network information from private and public network data - for net_type, net_val in itertools.chain(private_net.items(), - public_net.items()): - tmp_vlan = {} - # Ingress is special network that has no vlan, only a subnet string - # So treatment for ingress is different - if net_type != "ingress": - # standardize the network name as net_type may ne different. - # For e.g instead of pxe it may be PXE or instead of calico - # it may be ksn. Valid network names are pxe, calico, oob, oam, - # overlay, storage, ingress - tmp_vlan["name"] = \ - self._get_network_name_from_vlan_name(net_type) - - # extract vlan tag. It was extracted from xl file as 'VlAN 45' - # The code below extracts the numeric data fron net_val['vlan'] - if net_val.get("vlan", "") != "": - value = re.findall(vlan_pattern, net_val["vlan"]) - tmp_vlan["vlan"] = value[0] - else: - tmp_vlan["vlan"] = "#CHANGE_ME" - - tmp_vlan["subnet"] = net_val.get("subnet", "#CHANGE_ME") - tmp_vlan["gateway"] = net_val.get("gateway", "#CHANGE_ME") - else: - tmp_vlan["name"] = "ingress" - tmp_vlan["subnet"] = net_val - vlan_list.append(tmp_vlan) - LOG.debug( - "vlan list extracted from tugboat:\n{}".format( - pprint.pformat(vlan_list))) - return vlan_list - - def get_ips(self, region, host=None): - """Return list of IPs on the host - - :param string region: Region name - :param string host: Host name - :returns: Dict of IPs per network on the host - :rtype: dict - Example: {'oob': {'ipv4': '192.168.1.10'}, - 'pxe': {'ipv4': '192.168.2.10'}} - The network name from get_networks is expected to be the keys of this - dict. In case some networks are missed, they are expected to be either - DHCP or internally generated n the next steps by the design rules. - """ - - ip_ = {} - ipmi_data = self.parsed_xl_data["ipmi_data"][0] - ip_[host] = { - "oob": ipmi_data[host].get("ipmi_address", "#CHANGE_ME"), - "oam": ipmi_data[host].get("oam", "#CHANGE_ME"), - "calico": ipmi_data[host].get("calico", "#CHANGE_ME"), - "overlay": ipmi_data[host].get("overlay", "#CHANGE_ME"), - "pxe": ipmi_data[host].get("pxe", "#CHANGE_ME"), - "storage": ipmi_data[host].get("storage", "#CHANGE_ME"), - } - return ip_ - - def get_ldap_information(self, region): - """Extract ldap information from excel""" - - ldap_raw_data = self.parsed_xl_data["site_info"]["ldap"] - ldap_info = {} - # raw url is 'url: ldap://example.com' so we are converting to - # 'ldap://example.com' - url = ldap_raw_data.get("url", "#CHANGE_ME") - try: - ldap_info["url"] = url.split(" ")[1] - ldap_info["domain"] = url.split(".")[1] - except IndexError as e: - LOG.error("url.split:{}".format(e)) - ldap_info["common_name"] = \ - ldap_raw_data.get("common_name", "#CHANGE_ME") - ldap_info["subdomain"] = ldap_raw_data.get("subdomain", "#CHANGE_ME") - - return ldap_info - - def get_ntp_servers(self, region): - """Returns a comma separated list of ntp ip addresses""" - - ntp_server_list = \ - self._get_formatted_server_list(self.parsed_xl_data["site_info"] - ["ntp"]) - return ntp_server_list - - def get_dns_servers(self, region): - """Returns a comma separated list of dns ip addresses""" - dns_server_list = \ - self._get_formatted_server_list(self.parsed_xl_data["site_info"] - ["dns"]) - return dns_server_list - - def get_domain_name(self, region): - """Returns domain name extracted from excel file""" - - return self.parsed_xl_data["site_info"]["domain"] - - def get_location_information(self, region): - """Prepare location data from information extracted by ExcelParser""" - - location_data = self.parsed_xl_data["site_info"]["location"] - - corridor_pattern = r"\d+" - corridor_number = \ - re.findall(corridor_pattern, location_data["corridor"])[0] - name = location_data.get("name", "#CHANGE_ME") - state = location_data.get("state", "#CHANGE_ME") - country = location_data.get("country", "#CHANGE_ME") - physical_location_id = location_data.get("physical_location", "") - - return { - "name": name, - "physical_location_id": physical_location_id, - "state": state, - "country": country, - "corridor": "c{}".format(corridor_number), - } - - def get_racks(self, region): - # This function is not required since the excel plugin - # already provide rack information. - pass - - def _get_excel_obj(self): - """Creation of an ExcelParser object to store site information. - - The information is obtained based on a excel spec yaml file. - This spec contains row, column and sheet information of - the excel file from where site specific data can be extracted. - """ - - self.excel_obj = ExcelParser(self.excel_path, self.excel_spec) - - def _extract_raw_data_from_excel(self): - """Extracts raw information from excel file based on excel spec""" - self.parsed_xl_data = self.excel_obj.get_data() - - def _get_network_name_from_vlan_name(self, vlan_name): - """Network names are ksn, oam, oob, overlay, storage, pxe - - This is a utility function to determine the vlan acceptable - vlan from the name extracted from excel file - - The following mapping rules apply: - vlan_name contains "ksn or calico" the network name is "calico" - vlan_name contains "storage" the network name is "storage" - vlan_name contains "server" the network name is "oam" - vlan_name contains "ovs" the network name is "overlay" - vlan_name contains "oob" the network name is "oob" - vlan_name contains "pxe" the network name is "pxe" - """ - - network_names = [ - "ksn|calico", - "storage", - "oam|server", - "ovs|overlay", - "oob", - "pxe", - ] - for name in network_names: - # Make a pattern that would ignore case. - # if name is 'ksn' pattern name is '(?i)(ksn)' - name_pattern = "(?i)({})".format(name) - if re.search(name_pattern, vlan_name): - if name == "ksn|calico": - return "calico" - if name == "storage": - return "storage" - if name == "oam|server": - return "oam" - if name == "ovs|overlay": - return "overlay" - if name == "oob": - return "oob" - if name == "pxe": - return "pxe" - # if nothing matches - LOG.error( - "Unable to recognize VLAN name extracted from Plugin data source") - return "" - - def _get_formatted_server_list(self, server_list): - """Format dns and ntp server list as comma separated string""" - - # dns/ntp server info from excel is of the format - # 'xxx.xxx.xxx.xxx, (aaa.bbb.ccc.com)' - # The function returns a list of comma separated dns ip addresses - servers = [] - for data in server_list: - if "(" not in data: - servers.append(data) - formatted_server_list = ",".join(servers) - return formatted_server_list - - def _get_rack(self, host): - """Get rack id from the rack string extracted from xl""" - - rack_pattern = r"\w.*(r\d+)\w.*" - rack = re.findall(rack_pattern, host)[0] - if not self.region: - self.region = host.split(rack)[0] - return rack - - def _get_rackwise_hosts(self): - """Mapping hosts with rack ids""" - - rackwise_hosts = {} - hostnames = self.parsed_xl_data["ipmi_data"][1] - racks = self._get_rack_data() - for rack in racks: - if rack not in rackwise_hosts: - rackwise_hosts[racks[rack]] = [] - for host in hostnames: - if rack in host: - rackwise_hosts[racks[rack]].append(host) - LOG.debug("rackwise hosts:\n%s", pprint.pformat(rackwise_hosts)) - return rackwise_hosts - - def _get_rack_data(self): - """Format rack name""" - - LOG.info("Getting rack data") - racks = {} - hostnames = self.parsed_xl_data["ipmi_data"][1] - for host in hostnames: - rack = self._get_rack(host) - racks[rack] = rack.replace("r", "rack") - return racks diff --git a/spyglass/examples/SiteDesignSpec_v0.1.xlsx b/spyglass/examples/SiteDesignSpec_v0.1.xlsx deleted file mode 100644 index cdf827808bc2f887f635c8985579449e8234fb03..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 17291 zcmeHvbyQtT(k}#ecXtc!5Zv9}-6goYySqEVHMj=}?hxGFIXL8zJ9pll$=vU)_y0G0 zoppM3?XF+NRrp>wmbCM*E`K$!>h;l2HTum8m}FrYkQ+sA;|sdmgK+@%`sFQlRZ3a=AG zi+l?r{fP)Y+{PUo+IJ&dLs{jcAR!=uvhKjg!WlF-q} z_|Qm$4ku)G2tpZ+2E^*TA80Vh08Y#nB8a~WE5Bn>q@{+XCTgIL6^mf~3Mtg?usYv- zo)}Al2{$4R_9PGInOdRg)6S5~#Xcz;sEsWXMQN2($Gt}Juw$BZHQ!kmx1a~wB`x8)B9CzjJjf%7%^|{Cy&h2C;Ht!&f28};9yxrWz5xBcf?r3uXAM7B$RLpNwX{pIl zE87D+fFJyB+~aWO0?YkY8;`rOK_-y5hj!;9-pl(k`}q?DNdDhswMm769SG|H|n9V!`~EN3Tqfk?UuG4Y?412_3$lUyDN&lyVgiZzodr@sn6Z zXo@Z*#aZj7z(rKX4g?YN?euy1zOwcy`goY=Zkwet5*d}7q}jD9B=zaDGXyocLyG9< z%B=w;mwCYaUAma0JEco!EOkXoS-#ZB28rnGm2e&6B%K;I6iP8}2s%%?zviHf#=7A{ zHSoNk(pgnV?N|1^-uVHH z7qPpZv31XlaAr5ks~4?o*03Tm3-SZgq}UKy9>7<#k>zSM+kJo?s=s3RawH&GDEgmVN5Z(} z00V5$MQ~T(bm!-UH~H^eo>ak|%_|TLb;zMI)<)%g?#~%X=WMk_sZ)idEH{i}rq7(W z+ju0->r|BWxS(x6GAEj#b?@cW?g_)1nA`5*(W1m_tLW(2CYP*a<&4|;iOV?@_0#(U zla#e^Sxk8Iq*4cm1RSxAXacJd+C((=AG4#nS@SY1BC-__UH7pEd|*c&?QAw!bfROs zd~l|&uwX{QVG{FVXzK$kUUbnlK7hIBSxm=T_%@B6D2_uUjicwZ9l zE9YO=Or`Ru?Fs{82gV^E;-}k^LnKaF?NNU{5jaqrIIa;9j0{bUAQ45f@+E;O-wVkI z&96s9BA!Tk+o$JaABR#QG!h^fRLzs8E=lu1Q5VzYSAy27`P>eY)Q6M@C94SYWA%^r zcF!`ebTijPPw)hQOi(mA)S|sMu0N1AP_%srrf6kL;MC5jvZDd{*Z}62SWE!cn@*oD z-zbEczNmlr2?G~BV-Y#&gda9(%;2dRG5Kto)E6$x*i=mof3aRLESfUd(^`Nk-mCEk zwarq=Ks~}%p4i|Kj5$@ywCce*X#NY`2nHa;pCCkMB=IwyWXp*;MdcS>#${nDDynqn zQE850WUhhY^?rJsl4{c?$}1M!kmkfiNormGPs)B)8M1)q1qx1&Ftc_~YX~b>52Th0_ReifwT0(F8 zBJ-rt+ZP5wh_8X&TQwgcZFc1J8Z&bI%GEiOm8@o+$Q7reskV@xZtOt{?OFbyH{9pe z9~%qmEaG&tF_IC5Gr9q_w36G%F6GdIK|T8%WC+58KYo99*Ey^mKSdcvf7!9}3~8vv zf$DA_;TgqU=`+qosg^mSkk@R&pwEpwuerC(O*8L|`GerO(V|La2Uc=Sd~=|(9{ucL zALBOV{=jp3y0dJ&03mwUu~Ch=5je!aZ=FP&Mi(cLEn`PQ=~DtEgsX`Tcfk^IzMKKB zbZD>0DcI<&KSr*e@VdJfw)>465-mjko*^fW*K_x4a1IVYPC3!7H%bd<(Dk8qji1*D z{kh5r`$nYcx$^aGVVC=TCI07Ij%JSSUK|1l=z$ps2<{(i)zQh_+QjkK+igxuD>jn@ z)kn|bC&+g87V>b9b%L!~$+YTm47IcBkI*heLaYiu68j%T^KbV&&8-|zpb0>;F{m>3 zUXK&64@=VP)`jv<*rZf_SIP@^EE>>;2+x$dGJC!`2NASVRG*DD;6CF@@9D&Sw=?bj z$tE#E2g}b<``Qeb|awVwYTx&FWkK^bay~e?egvYdVdHj`sTsOp0@2&@!zxCmxH6SQHXH ze^xwNWFq=rD%}K}H05ujWY+;kl`+R<)oK%=G}j_q;4IG8)@nUOl{MA6~pw3l1B$pM77bDjLEw#+free*P%U4&IRj=c;G;LYp(Y z@j%?TVx8cV+SNwXc<6+T5Xun`8HTW+Pg{or)-e=_Qy>yx{Zo zGM8wZ2DW`7pa3h3ybV2IQdD~uKPQ&*F3<+ZvTwnbg0V@Ff*U-7;@zwJ*ANaQAth7ef48iyxKE%!KFpXF;CLf&286B_kNyHuQIyCu}P7J#Yf`_Rw z+mx`plGsavk^tFTX*3sehd|}}4?ttNP6Vn65~?pwDGobLFygbGSZ5o7!^v9vqYyN+ zxGSFd{oIskkih&1r7jboN`K4OYh@{vhB*9UhCP04Qa zau2_RcHPkSQkh1`1WuY|M*LdUjN4SX`;RRtlXgOl5ruwaJ50IzMgbC=NUDm)0@`TH zlfgvax0nRPS4;|p2@}iX#o4eQhAlu_l`tl9K{F1~AuaW?gpa@|4!NC*MVOC|OfWtPq6kZh8bp$saP-hH#4 z@((YWx0XK~1LH|3p3(JRewHp~ol>0AZgSk$85om39f~0ImRbck)|!k?M_m`G&LQ#B zv`9}*PDd<~Ny(=#D8EV@jvSwTLR?*NixaGKV8t8LB;u&o9F%bW5~x$AbWt07R)F4; z$58e8f<{^_MGA>6vkC~~?l}2sFKis-(3ShLD#??BmJ_zX#;J(cfLbwtxzA1P?HK@f zj;wlTV{78#)NC;wMI}{2L`F&C!B;YmSrJ;B*>5c|hV401g97zqzU?!|JwD7yR7>$m zxg4}e37}a#GJ6||Xi>m6-7YKB1LV6>$u}1touZ-SM>|#wfeSjnIor}YvGZx!!uG~T zSO@#H?=A;utMWI2jiDL}6 z;GFNaAEY)Tw6=fBAb+`$f^t?AGsfp- zUp_?QFpNQwt?(4Yh&Vb6S809mlz`$Am?WhBSir0R)r(Q2S`EsUj&W7-{n9&yF_Wp0 zYxtJ#v&;Eq1wXi=kLobq(A&)@uZ$(3a5P;);yExhE+_`$ux|BUx|*{&_n8IVmmN0T zfx3P%4qH@s|N9uf$Oehs^e(Ed)OY2lbZ11`U5%ihsrV=gxO8qm)W6!e?~nH&J_Ues!_i{` z8(sw#w?mCMUhOE;b^jc<{RF&PxhQsvhQjvko#30d_~n+7C&$mR27zidAmHZ=xq#B+ zQ2Vpi!PVNTy!F|->X?E)RgCk`GWGrb*PWf!Y1p$ZzAc$(n)O+IyW}3EO562fqK@%+ z;{nr6zGNFE4Y$q94LHp3Vlt-2u+sdTNjD-O1c4Gl-B9$TFU z)3B0bk2lj{)vxD}5g(=ov{hS!22>G|Kjjb#j+5o5)SQmJy(`xLRJqKvr=5UzRU7qg zB9i=N9y*$vm^eB9xySgU!t{T47-I>&vLHl=As6IbAnEP3S|~=+%esvc2)d2DKz*ll zRIs3B?cSc8=8ZDZ7zp?j%w9LI#wiuc+;<^VEBwfs>p(&?z3SMc9oCC#Uwa%BjFVu3 zkm1t|=5=&G=1!F>66{S6j!_5K2h}CHQ3x-PcqF@ri8sxL?y8iVU<)gIl!Bv1mb4-r zaf7sxkgs4uSr}FO+aT20zGdS*ty^{=cwEGPPe&srmjat*PEiEek5IMfA@;wSUAI`f zR+A!}kfM82bg2BC*wFYq^E}KegEFM?co=uG`R$;`hyf6YzmFZA^ej>TYL~Em7}DnY zw}wi1*N^LycUNNy1rQMCf6Urn^OdD(W4FQq`{vX03emrjI$$S(!-99r)epDix7EDj z?Sj|Dcr47TW%0w;cP`Q>zFEyvFGJLQEVO_!A;JCT3LwuxU-`w+XSHSBdW-8c?MlbT z>qpDamxtS}@6ML-_^uYcvDQ0Xbw6I+o(~Av&2)JdJqLq8kYfOYYcLAc2RB?R4MLie zZX+?5qa-!(<$g8uvmJCaDd%%Sp@!%5=%bHB?c zK(8m>x(m^bLTsX|L{&L*d!bPTl89Gv7_Dwff;tCrReXGS?#aBU3>Rwj(hZ``Pculq z!wnIOFA8J)tJ*X_vTpz{9`Fgab}pP_{26QIOFM&|H;XI97ZY}NN&Y_jr`lRSxrx0W zCZJQGngg93@Onq-PZd@h3j9Jldz^P8KH0y0BzT3*1B;tGMv_P{CH6JI?g9Yo{w#Id z`^nGMUHjuFJGPf%#Y2^Y%PAud+P0m~TjZXcC-4_xeKd^IcB~DMP^+; z$cGbLha+A6`CAq5oSW<=J8fT9m{)V>J>YeB{icT`sr2f^Y|CDRpO#5x#uijv@gH(U zB{#>b3n;a9s|KZjVJLO_B2noxsyeB1*c4PMSR~dsPU%>n+S;ZWuomPYqWfDP# znWUvDO(X4ON7&4yhM927U(IN|0&<0pF}9im#1Nh92igzXgN>miQl)J&;TZ_rpvOSN z1r{C7KfbUlJfUhN7*>K2^)>8?0@!$mf~eOC8D9fDP#}>|RF(ytoXEBOwx|jptCWeO z*DaYW?aL`jGTfl*W5g26G^Umb50mWPJnAnxE@Cnqc)7GH@tJy}b>-vqJPrqGA%jlB zNQf&I24Vwep4IIZ$4)UOTyP{^mPiv1B*;u;m4q4B31n(sw64UYdz$WKI&?(_xCIZo zwe?ngR5j!mw@Z{01GCj|lBxYVVkK^Od@f1a#4gbg4ouY9bQyn_lf>4U)6q&=(qYFx zyw36&8*!$cT>%~xL&H+G)>ZC{_DqCGc+p)+>x^IAdgro(xE=R>&-F>YJNb!p-DJZ- z+DXRO(kS@6g4y64GbV^Hx8T&!lM}dUwcX0yacSyy^o3%Eij_q|;}&WGy6p>P{5PjT zKp7oEb_G0MjV35i)pf|NpFuiAv{1M2&oWp+>^G zmBy_Qx9l*WCO6=#E^bZ^Lq!r1(MCUDUxRPiv7_&Q7217%;feq=ee+mxcZEhCKg@&q0sNf#@w;v;pW(ZIk95bmyQLPBpfMK+Y#j~z8l=h zlGCQ6*EP)iC>L1%CdDDo_TA8@qk+;XG#8y7Jowd&&^^SZ($C_R%ZQp~0Qy<85{2f8 zpSQE?7;M@qLKiXptQmu4vq$j?YJu&5)?V`@AU4HPl01ppE6(a^wLlB|fG77h;bsPkC+ zDg%0Sp`E$WMwkk9!BbH^xs=qxu3Zx_LdC*HSm|n}m6kF7kNbc_9xB1mRLC{17CEI{ zV1v%1dH!F>!TDztcZkjRNmf(sQ>bAsn0FQ_Z^7EJ-<)UU&6n`*R0R18%|YrZT5%~E3MBwl zf*_qbZQv{>?1bE55JI%vaLa?J6_5*tXct4+eUez55p_+{$~{h;PW;4&X2S;PZ3XSy(ODKiVo zDGu+P(BqA+xHB^4Ls?48MZ!tJJMcNOg%b^91yL^GEJ7FvZ)aF>?Q{2v+PQ*DGDBx@ z5Pc4~5eZTgiBkoYJTlB+A*b~avKI30jr!JA%q_hXj|RYl%;Y*%*?kVdgFq^Blq1E@ zj9*A`IIB0`9+|%2sBGIetqUqbd_8+X?x8=HUVJ1A(wkd@dL)BdynL&sfzgevLTPck zFt`eIhJ9{}<@=DbZq`TkQm@SJdPbTL0r5oVQz8PNTkku(yw_xX|EMa!7Z(SIITcm_ zw>1)j7vwTiuzAoT9y%XTu4I`tKw0})PS*JbI; zA*r^j;88A{ZNalOu6eFQ#BF{`fb2JTP2)RzI&p*LDj5gG!`v6W?RSSZ?J!PoY!s@e{mnWP_A@pSRAlZyS4lRK^3R~x=5J+&YP@{i?42f^)G3D^zAdean?<2`YAHu~8Tu(!% zDORk)BuRMC4KS4V^kyO=IfYA>^z|(7klSL67lezV+)(>cN0wi=)Bo|_2&_IiNvl^%Usm0WlaTtmnH>O@tk(mpz0O^71u znssT4345F@?H&_gt}c?AVc3VJ}i1_)SN61eju-E?pGS zjX8+{xW8D^LjFsiF97P6-3F;KIG(7=4H%`4t?0qca9~8b`GqELo}G|BhsX2n^5`bC zo5%O|@UdU7zO{*3$`*o5N$Gq(R`2KQE5^EB_uJ+175mW~#Z})x_sipGYWL48E|7meB=yq=7&D+k;=OtRW_mp+p{}6Swlc zi7{{uEu!-l*Am;74ZOx6SjVSstIun(58B8>V`_y{Z51)ZIAbO3p_f zR|nRF4~JP@G00K^pKq{=)V39uqiaQOz(vA^m9q4Y$~TsX24m*ehz9CU@UcFVFA00N zhAj#VSR@c})_$PE_(8<@V6sIFS>0^i;JR|caIEQ!K(H*ngdWBlz=WO?r>vWxV|)7*JZ8##F;MDA;0L?WkU}bz2cY6bXg7)${yJ< z@G_gMANB|aB4;TV!44S$+PfuAq9HZRQa#Ym-E{0neeD{C^!3(x&bmOcyezTAPwi|n zi3%1#-Fsk>P%*2WV2D$m9g&=?E$dp9Yz^&xtis7-Y5-)V6WG!DD8A=hZcuryf{kIJ zezN^`p>%95NySY8oh8%qQ(t3N=w=r@B+;11EL)BBKqdNTca0_6+1`!jen$#tjfu9M z9zjvc$S6jc^)+4W`QreN7B7n6QJ1!S2aVIBSP#y)8&F^yUa@CF1^VX4fEta!kvR;c@h_$v0me~6xjx(o}(JvM08S;eb zv8jFY98TrN5!LfI=WCLk=b^IBsm>DoBRL^KjOj83$Ak)c@BtXQVuCfN(&{XDAF(B0 zXH=xAX6eIzqSq%NorvdT-@u@WiD&2=CpQQ(0HyEKaLS;)Z@6`4b}bi?OE+xAali%)hF&&F5>q&U z$c#w~pj@=#>7F>%iGJnUI1XD^rt!hlG2+32-*;r}fAzQOECHxo+!!pQ(07 zD>kCU6LUh(KNV(7bcbG)a1s&YtFqSrWMTy#Se<#P{{?>Op4WN9OPOXAOM;n?xvH{2 zR>qa-)jqVNmzqzQR@LHo;bM+>(s|sZI*{Kv2icCpd1ITOp=>WvV-9r(z*6bp!e*(& z-5=0XD{K?ErfL>6i+tg69k@DDrUEfgUp?nvQ!_TMW#eLxksyX}j&Pt1eI|K$<^U?9 z=K+#l0=GnbZ>&nv$8I+vG1{#My&~`x*#)Lpbcr5xK2O;^=*%)7FJ#_;!4T+V%AQE0s{KR`#(A;{!qeMS~`w}Ey&*8 zD))RXFT8Ah4}r>ARoN0{*`ZoZP0oBdSc1rlOM{DXBV*kgs#0m_ibKa810y5(-a+sX z0{v%pdX=d&Rc<#K&OzXjgvC%bS<&;Ccmr@KwpgWPWjwuj4vY$JzGBetD)qYb(-mENAY4 zW0{ldb4wu7;TkXJqTI^S#c_#z1;EM2HPZ92yZxz(tXu9D{^tVA27s@N*R~XD&iuq% z=O|xJ^5H?jxBUM3>{ftQf95UL1aKkm;8QN)dvSunA>o@t5erayY`b`~Tz@NboutR( zFoX(RFzR-Han}2acIxzStkE{7Qelcfwt28yJA3Y*t@<`F`E||czTh>fcoZNC?5)_@ ztmF8Q`x?5oYMu52ZmVHKetdy?1monMJ?Qf`d!LzpfAe#3DIJ*3g_pajRZnSEJbA+& zmqy-k7r3}*DR`MM-lc_=b$$Bxgk*d-?eJt;;N0c`F1)+mcGi7UeEW!t(dv$N9(Fqv zjx7Dnv+_9zGY=4FfLF=m$M)0NQBpaIwbAOQx7KN9{H#l`mZ}+6zLSC#3-DH;UGz-? z*O9Xqh=(urN%ScRJESMk=RCW;LePO9QEq9XHkn|T?$;U6e*{D^dlq|e<4%f{VMDP- zIZasreKNLhL@>%@;NK6Q3tctWqbHQ^ACj$+w`tk)6K=N=={Is{~lSU5Tq74`GnVUzf+utj~v z;5$Ji3>$%pa=Q$8aj;_ASnXFm5|I^OQGP(75FyTubJzuNi-@3N-AmD>4b;p$Ro_Efj+-0AkuBXpGaAo zETef$0iz(}V|tGR&$MOP8%3XI5oS}2;iLAld%)&4deagdaV08&mpApL=R`ma+ECk;s|5q2P=&9|@$sEZ!B;lFPcbh^GH=<0IDahgEkLk(K>^)YzGB?`OY&gz zyd}|>`M?OUVC|~Hj-Pp|p7?p1N9DbLHCCPGG%=W@da$v@n|^XKn8fqvDq!6U7~hs| zgcoKruCAd#salJ}t0vL3GqLV6E?`Bj@AnY4DvIKv1FYG5Otd>lCGwauO=CkXZ+#Ab z$*S{NU_S)gF?`K}^O3dFSFJfESeUfe^KIdCZi255d^Y2J;1oot?{^008=5nC?I^mp zt&WY%S;)@ z48Mr<=>DymQSBG0Qf&7C?d0GO`pGZ)lRMcp^RLnRgoJc;T_JJM*9iNFy0pX$PtvgE zF8< zcNqCe(Oom+(37y24uczal0SP|e;+T>pD=byUh83ha!v^B@jYVHcZso>L&@1>=?=pl_3@#D*4(6!3vt;7Q3^9 z79eRIK;uOzC?9^$!Fx$qHc-;iH-rkB>=v5{flh2YM){m1uomg=*>o`E|Id+ zTu4w!I{Mk1lxguff(1}`hjQ`Rt4zB#*DbcR9#_nRBezzAe(Jq%t)pLk)=ksC4=5#y;wbW8<@O(H^EsxL$=m4x^8g*P=pvvr!WH`p%Lhs`)mLtY)mj`=cf9 z8!3~$)8tX3V(vLC_(d(YdFZSJsEa};73;~1w%g+*7eZK^L=&y6>USNr7b29!%~(dN zIqO_UloNCu4et}mNwq5Uf4&#(`vdUgN)j&P;XKkM38xu}7gE$3Ml)0W8MVupxnN#) ziX$0`Ars~aRYQ+&MG35waM{zOHyj&<%Jt;M??0R|KEf^4I8Fb)_l%IY&tPigio|@! zv6>&+Cs9vv<(s`W}l3YI64H5(N znjINLtzx{*^F^~cwH&i>;@oPNB2+;TW=Ol)0f2Ytu|w95M>`RB^4*F}pbGoxE5I_+ zbT96#ORvEFW*5O3DqV`h1&T!NP=V#bVnXyB&#?kycQxsg|2Ka)2Eb+KUX$V;t+DtZ z=yb&{HVSG4Ig2r++-B;y^%<)au5yK~l8T)M`*5O9ehUVj|?x0WT^Y^~<9Zs>rjbBtm844w$9XKP=-aU}g)->D2^; z+OX*Z<-luA5;=9V8pYlFIer%%BBoT0(!a{(>mk>A%atWq?+P#bd2NBZM7W_V6g}C7 z%Gvw$EL~W;LDzS3$)t&$5Acb`AUf_eZ18vc^ONruc1qDo*p?e*n`fq!WVVV6#i&hQ zZKf1?w~9aBUhYn)13lV+%Qq9`E&qgYyiha21aBp{55twJNJ)Hc~lI`$gXEB5(42uD}oR$_IVjr4jgLM{Sf zGdF0@_itsi8hqwR`aR^<=KT~M={-Ex*v?4a!Oq^1-pJ0u^b!<1bD)1f4T7L>E zsa87qy{r;UEnRdbE=&3=g-U4zKDC}WJPAVNGy(@0bgL37-* zut2?xEQA7_krKj}v5Wrkca#1C>6|8K@IPF+2D2N$2)>PUegznxw$UBW5QR+2T|U1D zFO^|0eAz&QP~Hzvvh{0SNSfp>a&xHJ+QVl=6!y7n*aR) zD6A*6&?q_aI%S=&GPO+FFd_KLMm)heYdUG__P7w?(y9xATX)zf>StI`(8zd(F$+=B z0P{AK{QkpfVA96lF2R9i1tXaEkOX}qAfWfCgkN#K_V4<^$;3g)#L4NGqWHbeNM~qa zd#dJXk2a6R2OPn|1(FPu5^hrq28)(}=;%)cvO2Ubw!ePr;7Z>ssKe$tDbu{%Zq|U1 zFjZcFC?rOVcO^g~>CZbb0&AJgP*r3B6I~@=mWF;5X2J!M5W-J|3Aa$Pu`FYVFWu-| z@_Mk&y?pbe`~2Ls;p(th2q=SepJHl+9Q53h8X{IcjQVE(od~D=ay&5IAGL@G#`|R? zgXQz}in&!zx9<(D1|`fZZKkwD9|u}hc3d!dk0Wtq0c2nbn4D4Cxi9xPwtUeJ=tDaS z`RgT?4Lu^2)M~J*$wk-M_HA`6IRVzBp5cH2ddB%u$pS@>>r5I28bOrN!)ZeYK7j3s z8PI?^cK9X+>6JApr>d14jYxnZS-#7noQ4PPvYQWDDr+p)@HQSoN{>3=u>9cGZw1Z13yaya)zOP;zXae-*7l&ZWv*2 zEu*0u9X^tq)?&E9R#7`vk~B?>jIwh62_(626K$~hu#$S(9NGX@glSbXnM&QU5O|&Q zK$VN=XGUmDKtRGEA~L;3GZCjp@NlW5gR^kOzB7oFMA87&h$?N^pq#SdrNu~X++wph zHtH_YN&!jl@mw4g(}?6LMTQ9?S~8L^7iWyMx}p;L^yN4xn6_EC$x<_{1RR!P=aJ~J zrRA%5WXvXoW<;o$Y#O^G*gk+-gpDt`ooMSy8y+h9AffQwbmJJF`rdhJM5ve5jLCHkd>hBiyK+^cv_Kw z2=Ghw2a2Pm&14x&5i9}yZCO>8Sy`igyG(h?7(cCQ#w=yheBGhpOhk3ibopLR!%_b7 zj|SU*36fxF{f!(@UrHk=nM4IY^-fXhhn6SSY3T}xe49fW`bbBO<%m7~(YlCKe!oKe zOnjWNPeF0iypc=nd+Ul-E#p3D-+EYdn5kj^7A&EK7F~WQ3Eu5fI*^ub2!~c;1!@(t zB)nwSS5h53&EVEvR+tVp{{_0TC~1}ia*0p1CEw0i@oeEw-BjGf;HDxCVN4AqRn;)8 z6;ndapv_s9kRQeQwNI(gy%|za*JwhrouRg01PaJQ1@nW@AiX292*5fOas-6v_vP`g zwis~=b?H9!70vc>1RgdrFNZ{dB#`A(>r^We*+d$bf9Q!mhK^PF$gR$WnCu`OHeCcQ z&UnT`*e`C{DW#pZ6HFHIkkRi0 zR&lSrf0@!6!-p^hgf;4DrVMG^EEGX|D(mC~=$elxT8`E|836ZJ9|5O9CIQ%Q%b;re z(bM(!NJ2)eJ<%EWb-p{y}UsDid=PgKsR@f7o zS5S}j0UyGlJBb5@S1&1l#~KY|y#q>s4#rjtw*YY8U-gjkdX2id*vI{Bl6&nBw@eRdP!4Y>pDqi2XL~R zuzH=e9;F(#ZNC@+z_9JfU3D>l41KD@$dpn&u(tbD{I=|R1R!9a>ufd2e5kb_gnJ-~II zW~br7RXJV9^Bab|5qFbio+o|nj;iB1%bo=_52+9o#c%E)l%mXxuotwyR|4s(MHFA~44Xv>KNgW|Z}Nl$hpPr45G8Obu-i8?S{ zzaC#q?XDMcU-(+(J9=gX@8nuUKL0q0K2THXSrGbsIVBe}=!*B_s_EJGlW+5`{t9P1 zZO3+?Dv8>o=Zd>S@#fLc2KU=*^89_)&Di?NU6+H)_TGW+aqja(@$H`S&f{^v-Po6O zpC{NOYve2L<%90qvx)YNNs3gLx4l(}#96U1lndL3)vB3vlW~~RVEpyPnfbc=>x_f@ zB?QtGc0KkO+tPd?W-WP)!?O9L2TuP_L#-L}yCm@rN|d9NvmM4&8Rw`fE@D&{2TZLu-NWwSr@ zm5Q-e^+M@-0(+YZfMMo1uq1!AW!_7}X-8eM;_v-EMz&4aA?HgrD?XcbCC&~~k{4AO zH$(qS^PtJxmr`eh6hpeH*UhFX;j1-5SdQdF85eRyBe;4-&LLhRlULe(F>7GRWaD1- zPoa)ypIqz!#~(L06*(}Y2v=K#SGlkK^fIePZt1s~`-i!=KSdGTP(>Z=HBD$x9A}fL zzlb>b-{FjryNm>(Fw0GwVc=i!p6yf-xvl>+Q8?2~%->Ixxle~aj#0~f)vvdZ;oyI* zd~9I!8m$2a%meDZl&(p}EG4@>b-%rzcKsu~8TbS3yQAdq{|exrzuG^`zxiVTd8vN~ z`1kzee=`0mE8j)qU$T{dXZ-h^&3|Q_2LHo8{~t0qf5-VfgYPdSnD=M?EuZgq=HFAQ z{$l?6o-X*#{8xh2@2tOvhX2J%{T|ZuXTSfiK=I!Je~+5}3sC>vvGwnOf5lM$j`DkG z(O)S3cz>Y$9%%GC!0%_He*plK{p~RRI4S)d<#)H-UnpaAf1vz_lkRtfe|H-E#Sa8z z&jbYYA1;L7*?-^g{RMEt^2Y%Ff3|