Tweaks and bugfixes moving towards a stable-ish release

Makefile

@@ -5,9 +5,11 @@ INSTALL_PATH=/opt/stack/artifice
 BILLING_PROGRAM=bill.py
 BINARY_PATH=/usr/local/bin
 
+CONF_DIR=./work/${INSTALL_PATH}/etc/artifice
+
 clean:
 	@rm -rf ./work
-	@rm *.deb
+	@rm -f *.deb
 
 init:
 	@mkdir ./work/
@@ -18,6 +20,8 @@ deb: clean init
 
 	@cp -r ./bin ./artifice ./scripts ./README.md ./INVOICES.md \
 		requirements.txt setup.py ./work/${INSTALL_PATH}
+	@mkdir -p ${CONF_DIR}
+	@cp ./examples/conf.yaml ${CONF_DIR}
 	@ln -s ./work${INSTALL_PATH}/bin/${BILLING_PROGRAM} ./work${BINARY_PATH}/artifice-bill
 	@fpm -s dir -t deb -n ${NAME} -v ${VERSION} \
 	--pre-install=packaging/scripts/pre_install.sh   \

README.md

@@ -15,16 +15,16 @@ The Artifice data store will prevent overlapping bills for a given tenant and re
 ## Requirements:
 
 Artifice requires:
- * Postgresql 9.1 or greater.
- * Python >=2.7.5, <3.0
- * Python modules:
-   * pyaml
-   * mock
-   * no
-   * TODO
- * OpenStack Grizzly or greater
- * Openstack-Keystone
- * Openstack-Ceilometer
+  * Postgresql 9.1 or greater.
+  * Python >=2.7.5, <3.0
+  * Python modules:
+    * pyaml
+    * mock
+    * requests
+    *
+  * OpenStack Grizzly or greater
+  * Openstack-Keystone
+  * Openstack-Ceilometer
 
 ## Installation
 

README.mw

@@ -0,0 +1,142 @@
+= Openstack-Artifice =
+
+== What ==
+
+Artifice is a layer sitting on top of Ceilometer to provide easy interactions with ERP systems, by exposing a configurable interface for turning Ceilometer data into a single billable line item.
+
+Artifice provides hooks to integrate with arbitrary ERP systems, by not imposing logic beyond the concept of a dated invoice that covers a given range.
+
+What the ranges are, and how Ceilometer data is aggregated is intended to be configurable.
+
+Artifice enforces its own rigid postgresql-backed data store, used to store what data has been billed, and for what time range. This is used by Artifice to add prevention of repeated billing of a range of data.
+
+The Artifice data store will prevent overlapping bills for a given tenant and resource ever being stored, while still allowing for regeneration of a given invoice statement.
+
+== Requirements: ==
+
+Artifice requires:
+ * Postgresql 9.1 or greater.
+ * Python >=2.7.5, <3.0
+ * Python modules:
+   * pyaml
+   * mock
+   * requests
+   * pyaml
+ * OpenStack Grizzly or greater
+ * Openstack-Keystone
+ * Openstack-Ceilometer
+
+== Installation ==
+
+Installing Artifice is as simple as: dpkg -i openstack-artifice-<version>.deb
+
+The library will be installed to /opt/stack/artifice, and the command-line tool 'artifice' will be added to the path.
+
+== Configuration ==
+
+Configuring Artifice is handled through its primary configuration file, stored in <code>/etc/openstack/artifice.conf</code>.
+
+This is a yaml-format config file, in the format of:
+
+<pre># Defines the database connection logic. This will be converted to a standard
+# database connection string.
+database:
+  database: artifice
+  host: localhost
+  password: aurynn
+  port: '5433'
+  username: aurynn
+# Configuration passed to the invoice system. This is arbitrary and may be
+# anything that the invoice object may require.
+invoice:config:
+  delimiter: ','
+  output_file: '%(tenant)s-%(start)s-%(end)s.csv'
+  output_path: /opt/openstack/artifice/invoices
+  row_layout:
+  - location
+  - type
+  - start
+  - end
+  - amount
+  - cost
+main:
+  # What invoice object we should be using
+  invoice:object: billing.csv_invoice:Csv
+# Configuration for OpenStack
+openstack:
+  # Location of the Keystone host
+  authentication_url: http://foo
+  # Location of the Ceilometer host
+  ceilometer_url: http://localhost:8777
+  # Default tenant to connect to. As this
+  default_tenant: demo
+  # Username to use
+  username: foo
+  # Password
+  password: bar</pre>
+A sample configuration is included, but '''must''' be modified appropriately.
+
+== Setup of Openstack environment ==
+
+As mentioned, Artifice relies entirely on the Ceilometer project for its metering and measurement collection.
+
+All development has (so far) occurred using a DevStack installation, but a production Ceilometer installation should work as expected.
+
+=== DevStack ===
+
+Installation on DevStack is relatively easy. First, prep the VM with DevStack. Since we need Ceilometer installed, we recommend a DevStack localrc similar to:
+
+<pre>ADMIN_PASSWORD=openstack
+MYSQL_PASSWORD=openstack
+RABBIT_PASSWORD=openstack
+SERVICE_PASSWORD=openstack
+
+# Enable Quantum
+disable_service n-net
+enable_service q-svc
+enable_service q-agt
+enable_service q-dhcp
+enable_service q-l3
+enable_service q-meta
+enable_service quantum
+
+# Enable Swift
+enable_service swift
+
+# Enable ceilometer!
+enable_service ceilometer-acompute,ceilometer-acentral,ceilometer-collector,ceilometer-api</pre>
+A localrc file can be found at '''devstack/localrc'''
+
+Create your VM and install DevStack into it. A Vagrant-compatible bootstrap script that will install most of the necessary components is included in this distribution, at '''devstack/bootstrap.sh'''
+
+Install Artifice and the packages it depends on from the Debian repositories.
+
+Artifices' post-intallation hooks will have set up the Postgres database as expected, and Artifice will be ready to run.
+
+=== Production OpenStack ===
+
+TODO: Fill this out
+
+== Using Artifice ==
+
+As mentioned, Artifice comes with a command-line tool to provide some simple commands.
+
+Actions one can perform with Artifice are:
+
+* Bill; Given a date range, generates the current usage bill for a tenant. This will result in a CSV file.
+
+=== Future things ===
+
+Eventually we also want Artifice to:
+
+* List current usage numbers
+* List historic usage numbers
+* Re-generate billing information
+
+Things we may eventually want include:
+
+* Listing this months' total usage of a given resource
+* Listing total usage by datacentre
+* Listing all usage ever
+* Etc
+

artifice/billing/csv_invoice.py

@@ -51,3 +51,12 @@ class Csv(object):
 
         fh.close()
         self.closed = True
+
+    def total(self):
+        total = 0.0
+        for line in self.lines:
+            # Cheatery
+            # Creates a dict on the fly from the row layout and the line value
+            v = dict([(k, v) for k, v in zip(self.config["row_layout"], line)])
+            total += v["cost"] or 0
+        return total

artifice/interface.py

@@ -4,6 +4,7 @@
 # Brings in HTTP support
 import requests
 import json
+import urllib
 
 
 from copy import copy
@@ -113,8 +114,9 @@ class Artifice(object):
         self.artifice = None
 
         self.ceilometer = ceilometer(
-            self.config["openstack"]["ceilometer_url"],
-            token=self.auth.auth_token
+            self.config["ceilometer"]["host"],
+            # Uses a lambda as ceilometer apparently wants to use it as a callable?
+            token=lambda: self.auth.auth_token
         )
         self._tenancy = None
 
@@ -137,10 +139,9 @@ class Artifice(object):
         # print "tenant list is %s" % self.auth.tenants.list()
         if not self._tenancy:
             self._tenancy = {}
-            invoice_type = __import__(self.config["invoices"]["plugin"])
             for tenant in self.auth.tenants.list():
                 t = Tenant(tenant, self)
-                self._tenancy[t.name] = t
+                self._tenancy[t["name"]] = t
         return self._tenancy
 
 class Tenant(object):
@@ -156,6 +157,12 @@ class Tenant(object):
         # Invoice type needs to get set from the config, which is
         # part of the Artifice setup above.
 
+    def __getitem__(self, item):
+
+        try:
+            return getattr(self.tenant, item)
+        except:
+            raise KeyError("No such key %s" % item)
 
     def __getattr__(self, attr):
         if attr not in self.tenant:
@@ -205,7 +212,7 @@ class Tenant(object):
                     "value": self.tenant["id"]
                 },
             ]
-            self._resources = self.ceilometer.resources.list(date_fields)
+            self._resources = self.conn.ceilometer.resources.list(date_fields)
         return self._resources
 
     # def usage(self, start, end, section=None):

artifice/invoice.py

@@ -84,3 +84,6 @@ class Invoice(object):
 
     def add_section(self, title):
         raise NotImplementedError("Not implemented in base class")
+
+    def total(self):
+        raise NotImplementedError("Not implemented in the base class")

bin/bill

@@ -0,0 +1,9 @@
+#!/bin/bash
+
+INSTALLED="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+ORIGIN=`pwd`
+
+# Bring up our python environment
+# Pass through all our command line opts as expected
+
+$INSTALLED/../env/bin/python $INSTALLED/bill.py $@

bin/bill.py

@@ -61,40 +61,42 @@ if __name__ == '__main__':
         invoice.bill(usage.objects)
         invoice.close()
 
-        for datacenter, sections in usage.iteritems():
-            # DC is the name of the DC/region. Or the internal code. W/E.
-            print datacenter
+        print invoice.total()
 
-            for section_name in args.sections:
-                assert section in sections
+        # for datacenter, sections in usage.iteritems():
+        #     # DC is the name of the DC/region. Or the internal code. W/E.
+        #     print datacenter
 
-                # section = sections[ section ]
-                print sections[section_name]
-                for resources in sections[section_name]:
-                    for resource in resources:
-                        print resource
-                        for meter in resource.meters:
-                            usage = meter.usage(start, end)
-                            if usage.has_been_saved():
-                                continue
-                            print usage.volume()
-                            print usage.cost()
-                            usage.save()
-                            # Finally, bill it.
-                            # All of these things need to be converted to the
-                            # publicly-viewable version now.
-                            invoice.bill(datacenter, resource, meter, usage)
+        #     for section_name in args.sections:
+        #         assert section in sections
 
-                # Section is going to be in the set of vm, network, storage, image
-                # # or just all of them.
-                # # It's not going to be an individual meter name.
-                # artifacts = section.usage(args.start, args.end)
-                # for artifact in artifacts:
-                #     if artifact.has_been_saved:
-                #         # Does this artifact exist in the DB?
-                #         continue
-                #     artifact.save() # Save to the Artifact storage
-                #     # Saves to the invoice.
-                #     invoice.bill ( artifact )
-                #     # artifact.bill( invoice.id )
-                # print "%s: %s" % (section.name, artifact.volume)
+        #         # section = sections[ section ]
+        #         print sections[section_name]
+        #         for resources in sections[section_name]:
+        #             for resource in resources:
+        #                 print resource
+        #                 for meter in resource.meters:
+        #                     usage = meter.usage(start, end)
+        #                     if usage.has_been_saved():
+        #                         continue
+        #                     print usage.volume()
+        #                     print usage.cost()
+        #                     usage.save()
+        #                     # Finally, bill it.
+        #                     # All of these things need to be converted to the
+        #                     # publicly-viewable version now.
+        #                     invoice.bill(datacenter, resource, meter, usage)
+
+        #         # Section is going to be in the set of vm, network, storage, image
+        #         # # or just all of them.
+        #         # # It's not going to be an individual meter name.
+        #         # artifacts = section.usage(args.start, args.end)
+        #         # for artifact in artifacts:
+        #         #     if artifact.has_been_saved:
+        #         #         # Does this artifact exist in the DB?
+        #         #         continue
+        #         #     artifact.save() # Save to the Artifact storage
+        #         #     # Saves to the invoice.
+        #         #     invoice.bill ( artifact )
+        #         #     # artifact.bill( invoice.id )
+        #         # print "%s: %s" % (section.name, artifact.volume)

bin/usage

@@ -0,0 +1,9 @@
+#!/bin/bash
+
+INSTALLED="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+ORIGIN=`pwd`
+
+# Bring up our python environment
+# Pass through all our command line opts as expected
+
+$INSTALLED/../env/bin/python $INSTALLED/usage.py $@

bin/usage.py

@@ -0,0 +1,86 @@
+#!/usr/bin/env python
+
+import os, sys
+
+try:
+    from artifice import interface
+except ImportError:
+    loc, fn = os.path.split(__file__)
+    print loc
+    here =  os.path.abspath(os.path.join(loc +"/../"))
+    sys.path.insert(0, here)
+    # # Are we potentially in a virtualenv? Add that in.
+    # if os.path.exists( os.path.join(here, "lib/python2.7" ) ):
+    #     sys.path.insert(1, os.path.join(here, "lib/python2.7"))
+    from artifice import interface
+
+import datetime
+import yaml
+
+date_format = "%Y-%m-%dT%H:%M:%S"
+other_date_format = "%Y-%m-%dT%H:%M:%S.%f"
+
+date_fmt_fnc = lambda x: datetime.datetime.strptime(date_fmt)
+
+if __name__ == '__main__':
+    import argparse
+    parser = argparse.ArgumentParser()
+    # Takes names to display.
+    # none means display them all.
+    parser.add_argument("-t", "--tenant", dest="tenants", help='Tenant to display', action="append", default=[])
+
+    # Add some sections to show data from.
+    # Empty is display all
+    parser.add_argument("-s", "--section", dest="sections", help="Sections to display", action="append")
+
+
+    # Ranging
+    # We want to get stuff from, to.
+
+    parser.add_argument(
+        "--from",
+        dest="start",
+        help="When to start our range, date format %s",
+        type=date_fmt_fnc,
+        default=datetime.datetime.now() - datetime.timedelta(days=31)
+    )
+    parser.add_argument("--to", dest="end", help="When to end our date range. Defaults to yesterday.",
+        type=date_fmt_fnc, default=datetime.datetime.now() - datetime.timedelta(days=1) )
+
+    parser.add_argument("-c", "--config", dest="config", help="Config file", default="/opt/stack/artifice/etc/artifice/conf.yaml")
+
+    args = parser.parse_args()
+    print "Range: %s -> %s" % (args.start, args.end)
+    try:
+        conf = yaml.load(open(args.config).read())
+    except IOError:
+        # Whoops
+        print "couldn't load %s " % args.config
+        sys.exit(1)
+
+    # Make ourselves a nice interaction object
+    instance = interface.Artifice(conf)
+    tenants = args.tenants
+    if not args.tenants:
+        # only parse this list of tenants
+        tenants = instance.tenants
+
+    for tenant_name in tenants:
+        # artifact = n.tenant(tenant_name).section(section).usage(args.start, args.end)
+        # data should now be an artifact-like construct.
+        # Data also knows about Samples, where as an Artifact doesn't.
+
+        # An artifact knows its section
+        tenant = instance.tenant(tenant_name)
+        # Makes a new invoice up for this tenant.
+        invoice = tenant.invoice(args.start, args.end)
+        print "Tenant: %s" % tenant.name
+
+
+        # usage = tenant.usage(start, end)
+        usage = tenant.usage(args.start, args.end)
+        # A Usage set is the entirety of time for this Tenant.
+        # It's not time-limited at all.
+        # But the
+
+        print invoice.total()

examples/conf.yaml

@@ -0,0 +1,27 @@
+---
+ceilometer:
+  host: http://localhost:8777/v2
+database:
+  database: artifice
+  host: localhost
+  password: aurynn
+  port: '5433'
+  username: aurynn
+invoice_object:
+  delimiter: ','
+  output_file: '%(tenant)s-%(start)s-%(end)s.csv'
+  output_path: ./
+  row_layout:
+  - location
+  - type
+  - start
+  - end
+  - amount
+  - cost
+main:
+  invoice:object: billing.csv_invoice:Csv
+openstack:
+  authentication_url: http://localhost:35357/v2.0
+  default_tenant: demo
+  username: admin
+  password: openstack

packaging/scripts/post_install.sh

@@ -5,8 +5,14 @@ PASSWORD=`cat <%= install_path %>/etc/artifice/database`
 
 export DATABASE_URL="postgresql://<%= pg_user %>:$PASSWORD@localhost:<%=pg_port%>/<%=pg_database%>"
 
-pip install -r <%= install_path %>/requirements.txt
+pip install virtualenv
 
-python <%= install_path %>/scripts/initdb.py
+# Set up a virtualenv for ourselves in this directory
+virtualenv <%= install_path %>/env
 
-python <%= install_path%>/setup.py install # register with python!
+# this should now be limited to only this space
+<%=install_path%>/env/bin/pip install -r <%= install_path %>/requirements.txt
+<%=install_path%>/env/bin/python <%= install_path %>/scripts/initdb.py
+
+# And this. Woo.
+<%=install_path%>/env/bin/python <%= install_path%>/setup.py install # register with python!

packaging/scripts/pre_install.sh

@@ -3,13 +3,19 @@
 # Loads a SQL script into postgres that creates the artifice DB.
 # Post-install script is going to load all the DB stuff via pythons
 
-PASSWORD=`pwgen -s 16`
 
-mkdir -p <%=install_path%>/etc/artifice
-touch <%=install_path%>/etc/artifice/database
-chmod 0600 <%=install_path%>/etc/artifice/database
 
-echo $PASSWORD >> <%= install_path %>/etc/artifice/database
+if [ -e <%=install_path%>/etc/artifice/database ]; then
+    PASSWORD=`cat <%= install_path %>/etc/artifice/database`
+else
+    PASSWORD=`pwgen -s 16`
+    mkdir -p <%=install_path%>/etc/artifice
+    touch <%=install_path%>/etc/artifice/database
+    chmod 0600 <%=install_path%>/etc/artifice/database
+    echo $PASSWORD > <%= install_path %>/etc/artifice/database
+fi
+
+
 
 sudo -u postgres psql -d template1 <<EOF
 CREATE DATABASE <%=pg_database%>;

requirements.txt

@@ -1,3 +1,7 @@
 sqlalchemy>=0.8
 psycopg2>=2.5.1
 requests==1.1.0
+pyaml==13.07
+python-ceilometerclient==1.0.3
+python-keystoneclient==0.3.2
+urllib3==1.5