airship/spyglass
Code Issues Proposed changes

Remove flask YAML web editor from Spyglass

Browse Source 
Spyglass was originally built with a web-based YAML editor.
Unfortunately, the presence of the editor has caused some problems. It
can cause the Docker container to freeze before creating the
intermediary YAML file. The execution of the flask app is also the
root cause for the Bandit B104 errors and B605 error.

Since the target audience for Spyglass is developers, it can be
assumed that they will have access to an editor with support for YAML
files. Having a web-based version of the YAML editor is unnecessary and
will just result in more code to maintain in the future.

Removes the editor script from utils.

Removes the yaml-editor entry point from the package.

Removes references to the yaml-editor from the CLI and engine.

Resolves all known Bandit errors.

In the future, a pause in execution could be provided to allow users to
make quick edits. Log messages could also notify users when a
placeholder value is inserted in the intermediary so they can fix it.

Change-Id: Ibc37e61f93b33904ea839e12fe5a8d586985e0b1
This commit is contained in:
Ian Pittwood 2019-04-08 15:57:44 -05:00
parent 6149c4a1a5
commit 8dd891aced
15 changed files with 3 additions and 601 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
MANIFEST.in
View File

@ -1,4 +1,2 @@
recursive-include spyglass/utils/editor/static *
recursive-include spyglass/utils/editor/templates *
recursive-include spyglass/ **.yaml
recursive-include spyglass/ **.json

4
doc/source/getting_started.rst
View File

@ -72,9 +72,6 @@ Supported Features
2. Remote Data Source Plugin: Supports extracting site data from a REST
endpoint.
3. YAML Editor for Intermediary YAML: Support runtime editing of missing
site parameters, see :ref:`yaml-editor-info`
Future Work
-----------
1) Schema based manifest generation instead of Jinja2 templates. It shall
@ -160,7 +157,6 @@ Options:
excel spec
-idir, --intermediary_dir PATH The path where intermediary file needs to be
generated
-e, --edit_intermediary Flag to let user edit intermediary
-m, --generate_manifests Generate manifests from the generated
intermediary file
-mdir, --manifest_dir PATH The path where manifest files needs to be

1
doc/source/index.rst
View File

@ -33,4 +33,3 @@ fed to Shipyard for site deployment / updates.
getting_started
tugboat
yaml-editor

65
doc/source/yaml-editor.rst
View File

@ -1,65 +0,0 @@
..
Copyright 2018 AT&T Intellectual Property.
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
.. _yaml-editor-info:
===========
Yaml Editor
===========
What is Yaml Editor?
--------------------
Yaml Editor is a spyglass utility which lets user edit their generated
intermediary file in a browser window. It is a minimal flask app which is
invoked from the parser engine in order to let user edit fields which
could not be fetched via :ref:`tugboatinfo` plugin.
Yaml Editor Utility Inputs
--------------------------
a) Yaml File: Yaml file required to be edited (This is required field)
b) Port: Port on which app shall be running
c) Host: This is only used to form URL which can be followed to open file in browser
d) String: String which is required to be updated in the file (default is '#CHANGE_ME')
Yaml Editor Utility Usage
-------------------------
With Spyglass (edit option is True by default):
::
spyglass -mg --edit_intermediary -t tugboat -x SiteDesignSpec_v0.1.xlsx -e excel_spec_upstream.yaml -d site_config.yaml -s airship-seaworthy --template_dir=<relative path to '../examples/templates'
As a stand-alone editor:
::
yaml-editor -f <yaml-file>
Help:
::
> yaml-editor --help
Usage: yaml-editor [OPTIONS]
Options:
-f, --file FILENAME Path with file name to the intermediary yaml file.
[required]
-h, --host TEXT Optional host parameter to run Flask on.
-p, --port INTEGER Optional port parameter to run Flask on.
-s, --string TEXT Text which is required to be changed on yaml file.
--help Show this message and exit.

3
setup.py
View File

@ -30,13 +30,10 @@ setup(
'netaddr',
'pyyaml',
'jinja2',
'flask',
'flask-bootstrap',
],
entry_points={
'console_scripts': [
'spyglass=spyglass.spyglass:main',
'yaml-editor=spyglass.utils.editor.editor:main',
],
'data_extractor_plugins':
['formation=spyglass.data_extractor.plugins.formation:FormationPlugin',

23
spyglass/parser/engine.py
View File

@ -15,11 +15,9 @@
import copy
import json
import logging
import os
import pkg_resources
import pprint
import sys
import tempfile
import jsonschema
import netaddr
@ -394,28 +392,11 @@ class ProcessDataSource:
f.write(yaml_file)
f.close()
def generate_intermediary_yaml(self, edit_intermediary=False):
def generate_intermediary_yaml(self):
""" Generating intermediary yaml """
LOG.info("Start: Generate Intermediary")
self._apply_design_rules()
self._get_genesis_node_details()
# This will validate the extracted data from different sources.
self._validate_intermediary_data(self.data)
if edit_intermediary:
self.edit_intermediary_yaml()
# This will check if user edited changes are in order.
self._validate_intermediary_data(self.data)
self.intermediary_yaml = self.data
return self.intermediary_yaml
def edit_intermediary_yaml(self):
""" Edit generated data using on browser """
LOG.info(
"edit_intermediary_yaml: Invoking web server for yaml editing"
)
with tempfile.NamedTemporaryFile(mode="r+") as file_obj:
yaml.safe_dump(self.data, file_obj, default_flow_style=False)
host = self._get_genesis_node_ip()
os.system("yaml-editor -f {0} -h {1}".format(file_obj.name, host))
file_obj.seek(0)
self.data = yaml.safe_load(file_obj)
return self.data

11
spyglass/spyglass.py
View File

@ -64,12 +64,6 @@ LOG = logging.getLogger("spyglass")
type=click.Path(exists=True),
help="The path where intermediary file needs to be generated",
)
@click.option(
"--edit_intermediary/--no_edit_intermediary",
"-e/-nedit",
default=True,
help="Flag to let user edit intermediary",
)
@click.option(
"--generate_manifests",
"-m",
@ -117,7 +111,6 @@ def main(*args, **kwargs):
# Extract user provided inputs
generate_intermediary = kwargs["generate_intermediary"]
intermediary_dir = kwargs["intermediary_dir"]
edit_intermediary = kwargs["edit_intermediary"]
generate_manifests = kwargs["generate_manifests"]
manifest_dir = kwargs["manifest_dir"]
intermediary = kwargs["intermediary"]
@ -208,9 +201,7 @@ def main(*args, **kwargs):
)
LOG.info("Generate intermediary yaml")
intermediary_yaml = process_input_ob.generate_intermediary_yaml(
edit_intermediary
)
intermediary_yaml = process_input_ob.generate_intermediary_yaml()
else:
LOG.info("Loading intermediary from user provided input")
with open(intermediary, "r") as intermediary_file:

0
spyglass/utils/editor/__init__.py
View File

170
spyglass/utils/editor/editor.py
View File

@ -1,170 +0,0 @@
# Copyright 2018 AT&T Intellectual Property. All other rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the 'License');
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an 'AS IS' BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import json
import logging
import os
import sys
import click
import yaml
from flask import Flask, request, render_template, send_from_directory
from flask_bootstrap import Bootstrap
app_path = os.path.dirname(os.path.abspath(__file__))
app = Flask(
"Yaml Editor!",
template_folder=os.path.join(app_path, "templates"),
static_folder=os.path.join(app_path, "static"),
)
Bootstrap(app)
logging.getLogger("werkzeug").setLevel(logging.ERROR)
LOG = app.logger
@app.route("/favicon.ico")
def favicon():
return send_from_directory(app.static_folder, "favicon.ico")
@app.route("/", methods=["GET", "POST"])
def index():
"""Renders index page to edit provided yaml file."""
LOG.info("Rendering yaml file for editing")
with open(app.config["YAML_FILE"]) as file_obj:
data = yaml.safe_load(file_obj)
return render_template(
"yaml.html",
data=json.dumps(data),
change_str=app.config["STRING_TO_CHANGE"],
)
@app.route("/save", methods=["POST"])
def save():
"""Save current progress on file."""
LOG.info("Saving edited inputs from user to yaml file")
out = request.json.get("yaml_data")
with open(app.config["YAML_FILE"], "w") as file_obj:
yaml.safe_dump(out, file_obj, default_flow_style=False)
return "Data saved successfully!"
@app.route("/saveExit", methods=["POST"])
def save_exit():
"""Save current progress on file and shuts down the server."""
LOG.info(
"Saving edited inputs from user to yaml file and shutting"
" down server"
)
out = request.json.get("yaml_data")
with open(app.config["YAML_FILE"], "w") as file_obj:
yaml.safe_dump(out, file_obj, default_flow_style=False)
func = request.environ.get("werkzeug.server.shutdown")
if func:
func()
return "Saved successfully, Shutting down app! You may close the tab!"
@app.errorhandler(404)
def page_not_found(e):
"""Serves 404 error."""
LOG.info("User tried to access unavailable page.")
return "<h1>404: Page not Found!</h1>"
def run(*args, **kwargs):
"""Starts the server."""
LOG.info("Initiating web server for yaml editing")
port = kwargs.get("port", None)
if not port:
port = 8161
app.run(host="0.0.0.0", port=port, debug=False)
@click.command()
@click.option(
"--file",
"-f",
required=True,
type=click.File(),
multiple=False,
help="Path with file name to the intermediary yaml file.",
)
@click.option(
"--host",
"-h",
default="0.0.0.0",
type=click.STRING,
multiple=False,
help="Optional host parameter to run Flask on.",
)
@click.option(
"--port",
"-p",
default=8161,
type=click.INT,
multiple=False,
help="Optional port parameter to run Flask on.",
)
@click.option(
"--string",
"-s",
default="#CHANGE_ME",
type=click.STRING,
multiple=False,
help="Text which is required to be changed on yaml file.",
)
def main(*args, **kwargs):
LOG.setLevel(logging.INFO)
LOG.info("Initiating yaml-editor")
try:
yaml.safe_load(kwargs["file"])
except yaml.YAMLError as e:
LOG.error("EXITTING - Please provide a valid yaml file.")
if hasattr(e, "problem_mark"):
mark = e.problem_mark
LOG.error(
"Error position: ({0}:{1})".format(
mark.line + 1, mark.column + 1
)
)
sys.exit(2)
except Exception:
LOG.error("EXITTING - Please provide a valid yaml file.")
sys.exit(2)
LOG.info(
"""
##############################################################################
Please go to http://{0}:{1}/ to edit your yaml file.
##############################################################################
""".format(
kwargs["host"], kwargs["port"]
)
)
app.config["YAML_FILE"] = kwargs["file"].name
app.config["STRING_TO_CHANGE"] = kwargs["string"]
run(*args, **kwargs)
if __name__ == "__main__":
"""Invoked when used as a script."""
main()

92
spyglass/utils/editor/static/app.js
View File

@ -1,92 +0,0 @@
// Copyright 2018 AT&T Intellectual Property. All other rights reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// This file includes all the frond-end functionality being used for the
// yaml editor application.
/**
* Calls /save URL to save edit progress.
* @param {String} data Stringified JSON data.
*/
function save(data) {
$.ajax({
type: 'POST',
url: '/save',
data: data,
success: function(res) {
setTimeout(function() { alert(res); }, 3);
},
contentType: 'application/json;charset=UTF-8'
});
}
/**
* Calls /saveExit URL to save edit progress and shut down web server.
* @param {String} data Stringified JSON data.
*/
function saveAndExit(data) {
$.ajax({
type: 'POST',
url: '/saveExit',
data: data,
success: function(res) {
setTimeout(function() { alert(res); }, 3);
},
contentType: 'application/json;charset=UTF-8'
});
}
/**
* Collects and validates data from textarea.
* @returns {String} Stringified JSON data.
*/
function getSimpleData() {
var data = $("#yaml_data").val();
try {
var index = data.indexOf(changeStr)
if (index != -1) {
var lineNum = data.substring(0, index).split('\n').length;
alert('Please change value on line '+ lineNum + '!')
return null
}
data = jsyaml.load(data)
}
catch(err) {
alert(err)
return null
}
return JSON.stringify({yaml_data : data})
}
/**
* Function to save edit progress.
*/
function saveSimple() {
var data = getSimpleData()
if (data) {
save(data)
}
}
/**
* Function to save edit progress and shut down web server.
*/
function saveExitSimple() {
var data = getSimpleData()
if (data) {
saveAndExit(data)
}
}

BIN
spyglass/utils/editor/static/favicon.ico
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 1.1 KiB

68
spyglass/utils/editor/static/jquery-linedtextarea.css
View File

@ -1,68 +0,0 @@
/**
* jQuery Lined Textarea Plugin
* http://alan.blog-city.com/jquerylinedtextarea.htm
*
* Copyright (c) 2010 Alan Williamson
*
* Released under the MIT License:
* http://www.opensource.org/licenses/mit-license.php
*
* Usage:
* Displays a line number count column to the left of the textarea
*
* Class up your textarea with a given class, or target it directly
* with JQuery Selectors
*
* $(".lined").linedtextarea({
* selectedLine: 10,
* selectedClass: 'lineselect'
* });
*
*/
.linedwrap {
border: 1px solid #c0c0c0;
padding: 3px;
}
.linedtextarea {
padding: 0px;
margin: 0px;
}
.linedtextarea textarea, .linedwrap .codelines .lineno {
font-size: 10pt;
font-family: monospace;
line-height: normal !important;
}
.linedtextarea textarea {
padding-right:0.3em;
padding-top:0.3em;
border: 0;
}
.linedwrap .lines {
margin-top: 0px;
width: 50px;
float: left;
overflow: hidden;
border-right: 1px solid #c0c0c0;
margin-right: 10px;
}
.linedwrap .codelines {
padding-top: 5px;
}
.linedwrap .codelines .lineno {
color:#AAAAAA;
padding-right: 0.5em;
padding-top: 0.0em;
text-align: right;
white-space: nowrap;
}
.linedwrap .codelines .lineselect {
color: red;
}

126
spyglass/utils/editor/static/jquery-linedtextarea.js vendored
View File

@ -1,126 +0,0 @@
/**
* jQuery Lined Textarea Plugin
* http://alan.blog-city.com/jquerylinedtextarea.htm
*
* Copyright (c) 2010 Alan Williamson
*
* Version:
* $Id: jquery-linedtextarea.js 464 2010-01-08 10:36:33Z alan $
*
* Released under the MIT License:
* http://www.opensource.org/licenses/mit-license.php
*
* Usage:
* Displays a line number count column to the left of the textarea
*
* Class up your textarea with a given class, or target it directly
* with JQuery Selectors
*
* $(".lined").linedtextarea({
* selectedLine: 10,
* selectedClass: 'lineselect'
* });
*
* History:
* - 2010.01.08: Fixed a Google Chrome layout problem
* - 2010.01.07: Refactored code for speed/readability; Fixed horizontal sizing
* - 2010.01.06: Initial Release
*
*/
(function($) {
$.fn.linedtextarea = function(options) {
// Get the Options
var opts = $.extend({}, $.fn.linedtextarea.defaults, options);
/*
* Helper function to make sure the line numbers are always
* kept up to the current system
*/
var fillOutLines = function(codeLines, h, lineNo){
while ( (codeLines.height() - h ) <= 0 ){
if ( lineNo == opts.selectedLine )
codeLines.append("<div class='lineno lineselect'>" + lineNo + "</div>");
else
codeLines.append("<div class='lineno'>" + lineNo + "</div>");
lineNo++;
}
return lineNo;
};
/*
* Iterate through each of the elements are to be applied to
*/
return this.each(function() {
var lineNo = 1;
var textarea = $(this);
/* Turn off the wrapping of as we don't want to screw up the line numbers */
textarea.attr("wrap", "off");
textarea.css({resize:'none'});
var originalTextAreaWidth = textarea.outerWidth();
/* Wrap the text area in the elements we need */
textarea.wrap("<div class='linedtextarea'></div>");
var linedTextAreaDiv = textarea.parent().wrap("<div class='linedwrap' style='width:" + originalTextAreaWidth + "px'></div>");
var linedWrapDiv = linedTextAreaDiv.parent();
linedWrapDiv.prepend("<div class='lines' style='width:50px'></div>");
var linesDiv = linedWrapDiv.find(".lines");
linesDiv.height( textarea.height() + 6 );
/* Draw the number bar; filling it out where necessary */
linesDiv.append( "<div class='codelines'></div>" );
var codeLinesDiv = linesDiv.find(".codelines");
lineNo = fillOutLines( codeLinesDiv, linesDiv.height(), 1 );
/* Move the textarea to the selected line */
if ( opts.selectedLine != -1 && !isNaN(opts.selectedLine) ){
var fontSize = parseInt( textarea.height() / (lineNo-2) );
var position = parseInt( fontSize * opts.selectedLine ) - (textarea.height()/2);
textarea[0].scrollTop = position;
}
/* Set the width */
var sidebarWidth = linesDiv.outerWidth();
var paddingHorizontal = parseInt( linedWrapDiv.css("border-left-width") ) + parseInt( linedWrapDiv.css("border-right-width") ) + parseInt( linedWrapDiv.css("padding-left") ) + parseInt( linedWrapDiv.css("padding-right") );
var linedWrapDivNewWidth = originalTextAreaWidth - paddingHorizontal;
var textareaNewWidth = originalTextAreaWidth - sidebarWidth - paddingHorizontal - 20;
textarea.width( textareaNewWidth );
linedWrapDiv.width( linedWrapDivNewWidth );
/* React to the scroll event */
textarea.scroll( function(tn){
var domTextArea = $(this)[0];
var scrollTop = domTextArea.scrollTop;
var clientHeight = domTextArea.clientHeight;
codeLinesDiv.css( {'margin-top': (-1*scrollTop) + "px"} );
lineNo = fillOutLines( codeLinesDiv, scrollTop + clientHeight, lineNo );
});
/* Should the textarea get resized outside of our control */
textarea.resize( function(tn){
var domTextArea = $(this)[0];
linesDiv.height( domTextArea.clientHeight + 6 );
});
});
};
// default options
$.fn.linedtextarea.defaults = {
selectedLine: -1,
selectedClass: 'lineselect'
};
})(jQuery);

1
spyglass/utils/editor/static/js-yaml.min.js vendored
View File

File diff suppressed because one or more lines are too long

38
spyglass/utils/editor/templates/yaml.html
View File

@ -1,38 +0,0 @@
{% extends "bootstrap/base.html" %}
{% block title %}YAML Editor{% endblock %}
{% block styles %}
{{super()}}
<link href="{{ url_for('static', filename='jquery-linedtextarea.css') }}" rel="stylesheet">
{% endblock %}
{% block scripts %}
{{super()}}
<script src="{{ url_for('static', filename='js-yaml.min.js') }}"></script>
<script src="{{ url_for('static', filename='jquery-linedtextarea.js') }}"></script>
<script type="text/javascript">
var changeStr = '{{ change_str }}'
$(document).ready(function(){
$("#yaml_data").val(jsyaml.dump(JSON.parse('{{ data|safe }}')))
$("#yaml_data").linedtextarea();
});
</script>
<script src="{{ url_for('static', filename='app.js') }}"></script>
{% endblock %}
{% block content %}
<div class="container" style="margin-top:30px;">
<div class="form-group">
<pre>Edit your YAML (Update corresponding fields with {{ change_str }} text):</pre>
<div>
<textarea class="form-control linedtextarea" id='yaml_data' style="height: 500px;box-shadow: none;"></textarea>
</div>
</div>
<div class="form-group pull-right">
<button type="button" onclick="saveSimple()" class="btn btn-lg btn-success ">Save</button>
<button type="button" onclick="saveExitSimple()" class="btn btn-lg btn-primary ">Save and Exit</button>
</div>
</div>
{% endblock %}