x/solar
Code Issues Proposed changes

Merge branch 'master' into new_db

Browse Source
This commit is contained in:
Jedrzej Nowak 2015-11-18 09:32:34 +01:00
commit 83416678a3
35 changed files with 821 additions and 168 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
README.md
View File

@ -300,10 +300,5 @@ Solar is shipped with sane defaults in `vagrant-setting.yaml_defaults`. If you n
# Image based provisioning with Solar
* In `vagrant-setting.yaml_defaults` or `vagrant-settings.yaml` file uncomment `preprovisioned: false` line.
* Run `vagrant up`, it will take some time because it builds image for bootstrap.
* Currently in order to perform provisioning, pre-built images from Fuel can be used
* Download images [using this link](https://drive.google.com/file/d/0B7I3b5vI7ZYXM0FPTDJEdjg0Qnc/view).
* Login into vm `vagrant ssh solar-dev`
* Go to `cd /vagrant/tmp/` directory
* Untar the images `tar vxf targetimages.tar`
* Run `vagrant up`, it will take some time because it builds image for bootstrap and IBP images.
* Now you can run provisioning `/vagrant/examples/provisioning/provision.sh`

31
bootstrap/README.md
View File

@ -1,16 +1,26 @@
# Solar image building
In `bootstrap/trusty64` directory there are `box.ovf` and `box-disk1.vmdk`
files from the `trusty64` Vagrant box (normally found in
`~/.vagrant.d/boxes/trusty64/0/virtualbox`).
To build, install Packer (https://www.packer.io/):
To build for a Virtualbox, install Packer (https://www.packer.io/):
```
cp vagrant-settings.yaml_defaults vagrant-settings.yaml
sed -i 's/master_image:.*$/master_image: solar-master/g' ./vagrant-settings.yaml
sed -i 's/slaves_image:.*$/slaves_image: solar-master/g' ./vagrant-settings.yaml
cd bootstrap
packer build solar-master.json
cp solar-master.box ../
packer build -only=virtualbox-iso solar-master.json
mv solar-master-virtualbox.box ../solar-master.box
cd ..
vagrant up
vagrant box add solar-master solar-master.box --provider virtualbox
vagrant up --provider virtualbox
```
To build for a libvirt, replace the following commands:
```
packer build -only=qemu solar-master.json
mv solar-master-libvirt.box ../solar-master.box
cd ..
vagrant box add solar-master solar-master.box --provider libvirt
vagrant up --provider libvirt
```
If Vagrant throws error about `vboxsf` try this:
@ -22,4 +32,7 @@ vagrant plugin install vagrant-vbguest
If you're rebuilding the same box, make sure Vagrant reimports it:
```
vagrant box remove solar-master
```
```
Note that you can also set `PACKER_LOG=debug` and/or `VAGRANT_LOG=debug`
the shell environment variables to get more information.

4
bootstrap/build-image.yaml
View File

@ -4,5 +4,5 @@
sudo: yes
tasks:
#- shell: vagrant init ubuntu/trusty64
- shell: /usr/local/bin/packer build solar.json -var 'is_master=true'
- shell: /usr/local/bin/packer build -only=qemu solar-master.json -var 'is_master=true'
- shell: /usr/local/bin/packer build -only=virtualbox-iso solar-master.json -var 'is_master=true'

67
bootstrap/http/preseed.cfg Normal file
View File

@ -0,0 +1,67 @@
# Ubuntu preseed file - preseed.cfg
# Works for Ubuntu 10.x, 11.x & 12.x
#
# For more information on preseed syntax and commands, refer to:
# https://help.ubuntu.com/12.04/installation-guide/i386/appendix-preseed.html
#
# For testing, you can fire up a local http server temporary.
# Download the preseed.cfg file locally, cd to the directory where the
# preseed.cfg resides and run hte following command:
# $ python -m SimpleHTTPServer
# You don't have to restart the server every time you make changes. Python
# will reload the file from disk every time there is a request. As long as you
# save your changes they will be reflected in the next HTTP download. Then to
# test with a PXE boot server, use the following kernel boot parameters:
# > linux auto url=http://<your_ip>:8000/preseed.cfg hostname=<hostname> locale=en_US keyboard-configuration/modelcode=SKIP
#
# NOTE: If you netboot/PXE boot Ubuntu, it will ignore the value in hostname,
# but you must provide a hostname as a boot parameter to prevent the Ubuntu
# install from prompting for a hostname
choose-mirror-bin mirror/http/proxy string
d-i base-installer/kernel/override-image string linux-server
d-i finish-install/reboot_in_progress note
d-i grub-installer/only_debian boolean true
d-i grub-installer/with_other_os boolean true
d-i partman-auto-lvm/guided_size string max
d-i partman-auto/choose_recipe select atomic
d-i partman-auto/method string lvm
d-i partman-lvm/confirm boolean true
d-i partman-lvm/confirm boolean true
d-i partman-lvm/confirm_nooverwrite boolean true
d-i partman-lvm/device_remove_lvm boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true
d-i partman/confirm_write_new_label boolean true
# Default user
d-i passwd/user-fullname string vagrant
d-i passwd/user-password password vagrant
d-i passwd/user-password-again password vagrant
d-i passwd/username string vagrant
d-i user-setup/allow-password-weak boolean true
d-i user-setup/encrypt-home boolean false
# german mirror
d-i mirror/country string manual
d-i mirror/http/hostname string archive.ubuntu.com
d-i mirror/http/directory string /ubuntu
d-i mirror/http/proxy string
d-i mirror/http/mirror select DE.archive.ubuntu.com
# german timezone and ntp server
d-i clock-setup/utc boolean true
d-i clock-setup/utc-auto boolean true
d-i time/zone string Europe/Berlin
d-i clock-setup/ntp boolean true
d-i clock-setup/ntp-server de.pool.ntp.org
#d-i pkgsel/include string openssh-server cryptsetup build-essential libssl-dev libreadline-dev zlib1g-dev linux-source dkms
d-i pkgsel/include string openssh-server ntp curl nfs-common linux-headers-$(uname -r) build-essential perl dkms
d-i pkgsel/install-language-support boolean false
# Policy for applying updates. May be "none" (no automatic updates),
# "unattended-upgrades" (install security updates automatically), or
# "landscape" (manage system with Landscape).
d-i pkgsel/update-policy select none
d-i pkgsel/upgrade select full-upgrade
tasksel tasksel/first multiselect standard, ubuntu-server

69
bootstrap/playbooks/files/cleanup.sh Normal file
View File

@ -0,0 +1,69 @@
#!/bin/bash -eux
CLEANUP_PAUSE=${CLEANUP_PAUSE:-0}
echo "==> Pausing for ${CLEANUP_PAUSE} seconds..."
sleep ${CLEANUP_PAUSE}
# Make sure udev does not block our network - http://6.ptmc.org/?p=164
echo "==> Cleaning up udev rules"
rm -rf /dev/.udev/
rm /lib/udev/rules.d/75-persistent-net-generator.rules
rm /etc/udev/rules.d/70-persistent-net.rules
mkdir /etc/udev/rules.d/70-persistent-net.rules
echo "==> Cleaning up leftover dhcp leases"
# Ubuntu 10.04
if [ -d "/var/lib/dhcp3" ]; then
rm /var/lib/dhcp3/*
fi
# Ubuntu 12.04 & 14.04
if [ -d "/var/lib/dhcp" ]; then
rm /var/lib/dhcp/*
fi
# Add delay to prevent "vagrant reload" from failing
echo "pre-up sleep 2" >> /etc/network/interfaces
echo "==> Cleaning up tmp"
rm -rf /tmp/*
# Cleanup apt cache
apt-get -y autoremove --purge
apt-get -y clean
apt-get -y autoclean
echo "==> Installed packages"
dpkg --get-selections | grep -v deinstall
# Remove Bash history
unset HISTFILE
rm -f /root/.bash_history
rm -f /home/vagrant/.bash_history
# Clean up log files
find /var/log -type f | while read f; do echo -ne '' > $f; done;
echo "==> Clearing last login information"
>/var/log/lastlog
>/var/log/wtmp
>/var/log/btmp
# Whiteout root
count=$(df --sync -kP / | tail -n1 | awk -F ' ' '{print $4}')
let count--
dd if=/dev/zero of=/tmp/whitespace bs=1024 count=$count
rm /tmp/whitespace
# Whiteout /boot
count=$(df --sync -kP /boot | tail -n1 | awk -F ' ' '{print $4}')
let count--
dd if=/dev/zero of=/boot/whitespace bs=1024 count=$count
rm /boot/whitespace
# Zero out the free space to save space in the final image
dd if=/dev/zero of=/EMPTY bs=1M
rm -f /EMPTY
# Make sure we wait until all the data is written to disk, otherwise
# Packer might quite too early before the large files are deleted
sync

28
bootstrap/playbooks/files/minimize.sh Normal file
View File

@ -0,0 +1,28 @@
#!/bin/bash -eux
echo "==> Installed packages before cleanup"
dpkg --get-selections | grep -v deinstall
# Remove some packages to get a minimal install
echo "==> Removing all linux kernels except the currrent one"
dpkg --list | awk '{ print $2 }' | grep 'linux-image-3.*-generic' | grep -v $(uname -r) | xargs apt-get -y purge
echo "==> Removing linux source"
dpkg --list | awk '{ print $2 }' | grep linux-source | xargs apt-get -y purge
echo "==> Removing documentation"
dpkg --list | awk '{ print $2 }' | grep -- '-doc$' | xargs apt-get -y purge
echo "==> Removing obsolete networking components"
apt-get -y purge ppp pppconfig pppoeconf
echo "==> Removing other oddities"
apt-get -y purge popularity-contest installation-report landscape-common wireless-tools wpasupplicant ubuntu-serverguide
# Clean up the apt cache
apt-get -y autoremove --purge
apt-get -y autoclean
apt-get -y clean
echo "==> Removing man pages"
rm -rf /usr/share/man/*
echo "==> Removing anything in /usr/src"
rm -rf /usr/src/*
echo "==> Removing any docs"
rm -rf /usr/share/doc/*

91
bootstrap/playbooks/files/prov.json Normal file
View File

@ -0,0 +1,91 @@
{
"image_data":{
"/boot":{
"container":"gzip",
"uri":"http://10.20.0.2:8080/targetimages/env_1_ubuntu_1404_amd64-boot.img.gz",
"format":"ext2"
},
"/":{
"container":"gzip",
"uri":"http://10.20.0.2:8080/targetimages/env_1_ubuntu_1404_amd64.img.gz",
"format":"ext4"
}
},
"output":"/vagrant/tmp/targetimages",
"repos":[
{
"name":"ubuntu",
"section":"main universe multiverse",
"uri":"http://archive.ubuntu.com/ubuntu/",
"priority":null,
"suite":"trusty",
"type":"deb"
},
{
"name":"ubuntu-updates",
"section":"main universe multiverse",
"uri":"http://archive.ubuntu.com/ubuntu/",
"priority":null,
"suite":"trusty-updates",
"type":"deb"
},
{
"name":"ubuntu-security",
"section":"main universe multiverse",
"uri":"http://archive.ubuntu.com/ubuntu/",
"priority":null,
"suite":"trusty-security",
"type":"deb"
},
{
"name":"mos",
"section":"main restricted",
"uri":"http://mirror.fuel-infra.org/fwm/7.0/ubuntu/",
"priority": 1050,
"suite":"mos7.0",
"type":"deb"
}
],
"codename":"trusty",
"packages": [
"acl",
"anacron",
"bash-completion",
"bridge-utils",
"bsdmainutils",
"build-essential",
"cloud-init",
"curl",
"daemonize",
"debconf-utils",
"gdisk",
"grub-pc",
"i40e-dkms",
"linux-firmware",
"linux-firmware-nonfree",
"linux-headers-generic-lts-trusty",
"linux-image-generic-lts-trusty",
"lvm2",
"mcollective",
"mdadm",
"ntp",
"openssh-client",
"openssh-server",
"puppet",
"python-amqp",
"ruby-augeas",
"ruby-ipaddress",
"ruby-json",
"ruby-netaddr",
"ruby-openstack",
"ruby-shadow",
"ruby-stomp",
"telnet",
"ubuntu-minimal",
"ubuntu-standard",
"uuid-runtime",
"vim",
"virt-what",
"vlan"
]
}

3
bootstrap/playbooks/files/sshd.sh Normal file
View File

@ -0,0 +1,3 @@
#!/bin/bash -eux
echo "UseDNS no" >> /etc/ssh/sshd_config

4
bootstrap/playbooks/files/supervisor.conf Normal file
View File

@ -0,0 +1,4 @@
[program:{{name}}]
command={{cmd}}
redirect_stderr=true
stdout_logfile=/var/log/{{name}}.log

5
bootstrap/playbooks/files/ubuntu-ansible.sh
View File

@ -3,8 +3,9 @@
# TODO: maybe this is better:
# http://docs.ansible.com/ansible/intro_installation.html#latest-releases-via-apt-ubuntu
apt-get remove -f python-pip
sudo apt-get install -y python-setuptools
sudo apt-get remove -f python-pip
sudo apt-get update
sudo apt-get install -y python-setuptools python-dev autoconf g++
sudo easy_install pip
sudo pip install -U pip
sudo pip install ansible

12
bootstrap/playbooks/files/update.sh Normal file
View File

@ -0,0 +1,12 @@
#!/bin/bash -eux
if [[ $UPDATE =~ true || $UPDATE =~ 1 || $UPDATE =~ yes ]]; then
echo "==> Updating list of repositories"
# apt-get update does not actually perform updates, it just downloads and indexes the list of packages
apt-get -y update
apt-get -y upgrade
echo "==> Performing dist-upgrade (all packages and kernel)"
apt-get -y dist-upgrade --force-yes
reboot
sleep 160
fi

33
bootstrap/playbooks/files/vagrant.sh Normal file
View File

@ -0,0 +1,33 @@
#!/bin/bash
date > /etc/vagrant_box_build_time
SSH_USER=${SSH_USER:-vagrant}
SSH_USER_HOME=${SSH_USER_HOME:-/home/${SSH_USER}}
VAGRANT_INSECURE_KEY="ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA6NF8iallvQVp22WDkTkyrtvp9eWW6A8YVr+kz4TjGYe7gHzIw+niNltGEFHzD8+v1I2YJ6oXevct1YeS0o9HZyN1Q9qgCgzUFtdOKLv6IedplqoPkcmF0aYet2PkEDo3MlTBckFXPITAMzF8dJSIFo9D8HfdOV0IAdx4O7PtixWKn5y2hMNG0zQPyUecp4pzC6kivAIhyfHilFR61RGL+GPXQ2MWZWFYbAGjyiYJnAmCP3NOTd0jMZEnDkbUvxhMmBYSdETk1rRgm+R4LOzFUGaHqHDLKLX+FIPKcF96hrucXzcWyLbIbEgE98OHlnVYCzRdK8jlqm8tehUc9c9WhQ== vagrant insecure public key"
# Packer passes boolean user variables through as '1', but this might change in
# the future, so also check for 'true'.
if [ "$INSTALL_VAGRANT_KEY" = "true" ] || [ "$INSTALL_VAGRANT_KEY" = "1" ]; then
# Create Vagrant user (if not already present)
if ! id -u $SSH_USER >/dev/null 2>&1; then
echo "==> Creating $SSH_USER user"
/usr/sbin/groupadd $SSH_USER
/usr/sbin/useradd $SSH_USER -g $SSH_USER -G sudo -d $SSH_USER_HOME --create-home
echo "${SSH_USER}:${SSH_USER}" | chpasswd
fi
# Set up sudo
echo "==> Giving ${SSH_USER} sudo powers"
echo "${SSH_USER} ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
echo "==> Installing vagrant key"
mkdir $SSH_USER_HOME/.ssh
chmod 700 $SSH_USER_HOME/.ssh
cd $SSH_USER_HOME/.ssh
# https://raw.githubusercontent.com/mitchellh/vagrant/master/keys/vagrant.pub
echo "${VAGRANT_INSECURE_KEY}" > $SSH_USER_HOME/.ssh/authorized_keys
chmod 600 $SSH_USER_HOME/.ssh/authorized_keys
chown -R $SSH_USER:$SSH_USER $SSH_USER_HOME/.ssh
fi

52
bootstrap/playbooks/files/vmtool.sh Normal file
View File

@ -0,0 +1,52 @@
#!/bin/bash -eux
if [[ $PACKER_BUILDER_TYPE =~ vmware ]]; then
echo "==> Installing VMware Tools"
# Assuming the following packages are installed
# apt-get install -y linux-headers-$(uname -r) build-essential perl
cd /tmp
mkdir -p /mnt/cdrom
mount -o loop /home/vagrant/linux.iso /mnt/cdrom
tar zxf /mnt/cdrom/VMwareTools-*.tar.gz -C /tmp/
/tmp/vmware-tools-distrib/vmware-install.pl -d
rm /home/vagrant/linux.iso
umount /mnt/cdrom
rmdir /mnt/cdrom
rm -rf /tmp/VMwareTools-*
fi
if [[ $PACKER_BUILDER_TYPE =~ virtualbox ]]; then
echo "==> Installing VirtualBox guest additions"
# Assuming the following packages are installed
# apt-get install -y linux-headers-$(uname -r) build-essential perl
# apt-get install -y dkms
VBOX_VERSION=$(cat /home/vagrant/.vbox_version)
mount -o loop /home/vagrant/VBoxGuestAdditions_$VBOX_VERSION.iso /mnt
sh /mnt/VBoxLinuxAdditions.run
umount /mnt
rm /home/vagrant/VBoxGuestAdditions_$VBOX_VERSION.iso
rm /home/vagrant/.vbox_version
if [[ $VBOX_VERSION = "4.3.10" ]]; then
ln -s /opt/VBoxGuestAdditions-4.3.10/lib/VBoxGuestAdditions /usr/lib/VBoxGuestAdditions
fi
/etc/init.d/vboxadd setup
fi
if [[ $PACKER_BUILDER_TYPE =~ parallels ]]; then
echo "==> Installing Parallels tools"
mount -o loop /home/vagrant/prl-tools-lin.iso /mnt
/mnt/install --install-unattended-with-deps
umount /mnt
rm -rf /home/vagrant/prl-tools-lin.iso
rm -f /home/vagrant/.prlctl_version
fi
if [[ $PACKER_BUILDER_TYPE =~ libvirt ]]; then
echo "==> Doing nothing special"
fi

30
bootstrap/playbooks/pxe.yaml
View File

@ -15,9 +15,11 @@
image_builder_path: /tmp/image_builder
http_ip: 10.0.0.2
http_port: 8000
supervisor_dir: /etc/supervisor/conf.d/
tasks:
# Istall and configure dnsmasq
- apt: update_cache=yes
- apt: name=dnsmasq state=present
- file: path={{tftp_root}} state=directory
- template: src=files/dnsmasq_pxe.conf dest=/etc/dnsmasq.d/pxe.conf
@ -33,7 +35,7 @@
- get_url: url={{insecure_pub_key}} dest={{insecure_pub_key_path}}
- apt: name=debootstrap state=present
- file: path={{tftp_root}}/ubuntu state=directory
- git: repo=https://github.com/rustyrobot/fuel-bootstrap-image-builder dest={{image_builder_path}}
- git: repo=https://github.com/Mirantis/fuel-bootstrap-image-builder dest={{image_builder_path}}
- shell: "{{image_builder_path}}/bin/fuel-bootstrap-image 2>&1 | tee /tmp/image_build.log"
environment:
BOOTSTRAP_SSH_KEYS: "{{insecure_pub_key_path}}"
@ -52,9 +54,29 @@
- service: name=nginx state=restarted
# Install discovery service
- shell: pip install git+https://github.com/rustyrobot/discovery.git
- shell: 'discovery &'
- shell: pip install git+https://github.com/Mirantis/discovery.git
# Install bareon-api
# Workaround is required because pbr does not handle git-eggs correctly and fails to install fuel-agent
- shell: 'pip install git+git://github.com/gitfred/fuel-agent.git@iso-build#egg=fuel_agent'
- shell: pip install git+https://github.com/Mirantis/bareon-api.git
- shell: 'bareon-api &'
# Install and configure supervisor
- apt: name=supervisor state=present
- set_fact: {'name': 'discovery', 'cmd': 'discovery'}
- template: src=files/supervisor.conf dest={{supervisor_dir}}/discovery.conf
- set_fact: {'name': 'discovery-scan', 'cmd': 'discovery-scan --ssh_key /vagrant/tmp/keys/ssh_private'}
- template: src=files/supervisor.conf dest={{supervisor_dir}}/discovery-scan.conf
- set_fact: {'name': 'bareon-api', 'cmd': 'bareon-api'}
- template: src=files/supervisor.conf dest={{supervisor_dir}}/bareon-api.conf
- service: name=supervisor state=restarted
# Add nat rules so slaves have internet access via solar-dev
- shell: iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o eth0 -j MASQUERADE
# build IBP images
- shell: fa_build_image --image_build_dir /tmp/ibp --log-file /tmp/ibp-image-build.log -d --data_driver nailgun_build_image --input_data_file files/prov.json

7
bootstrap/playbooks/solar.yaml
View File

@ -6,3 +6,10 @@
# Setup development env for solar
- shell: pip install -e . chdir=/vagrant/solar
- shell: pip install -e . chdir=/vagrant/solard
- hosts: all
tasks:
- lineinfile:
dest: /home/vagrant/.bashrc
line: eval "$(_SOLAR_COMPLETE=source solar)"
state: present

8
bootstrap/playbooks/tasks/base.yaml
View File

@ -1,6 +1,6 @@
---
- shell: apt-get update
- shell: sudo apt-get update
- name: Base packages
apt: name={{ item }} state=present
@ -38,14 +38,14 @@
#- shell: easy_install pip
#- shell: pip install -U pip
#- shell: pip install -U setuptools
- shell: pip install httpie
- shell: pip install docker-py==1.1.0
- shell: sudo pip install httpie
- shell: sudo pip install docker-py==1.1.0
# faster json
- shell: pip install ujson
# Redis
- shell: pip install redis
- shell: sudo pip install redis
- lineinfile: dest=/etc/redis/redis.conf regexp='^bind ' line='bind 0.0.0.0'
- service: name=redis-server state=restarted

11
bootstrap/playbooks/tasks/cloud_archive.yaml
View File

@ -1,7 +1,8 @@
---
- shell: apt-get update
- shell: apt-get -y upgrade
- shell: add-apt-repository -y cloud-archive:juno
- shell: apt-get update
- shell: apt-get update --fix-missing
- shell: sudo apt-get update
- shell: sudo apt-get -y upgrade
- shell: sudo apt-get -y install python-software-properties software-properties-common
- shell: sudo add-apt-repository -y cloud-archive:juno
- shell: sudo apt-get update
- shell: sudo apt-get update --fix-missing

6
bootstrap/playbooks/tasks/puppet.yaml
View File

@ -1,9 +1,9 @@
---
# Puppet
- shell: wget https://apt.puppetlabs.com/puppetlabs-release-trusty.deb -O /root/puppetlabs-release-trusty.deb
- shell: dpkg -i /root/puppetlabs-release-trusty.deb
- shell: apt-get update
- shell: sudo wget https://apt.puppetlabs.com/puppetlabs-release-trusty.deb -O /root/puppetlabs-release-trusty.deb
- shell: sudo dpkg -i /root/puppetlabs-release-trusty.deb
- shell: sudo apt-get update
- apt: name=puppet state=present
- template: src=files/hiera.yaml dest=/etc/puppet/hiera.yaml

179
bootstrap/solar-master.json
View File

@ -1,49 +1,182 @@
{
"variables": {
"cleanup_pause": "",
"headless": "",
"update": "true",
"ssh_username": "vagrant",
"ssh_password": "vagrant",
"install_vagrant_key": "true",
"http_proxy": "{{env `http_proxy`}}",
"https_proxy": "{{env `https_proxy`}}",
"ftp_proxy": "{{env `ftp_proxy`}}",
"rsync_proxy": "{{env `rsync_proxy`}}",
"no_proxy": "{{env `no_proxy`}}",
"iso_url": "http://releases.ubuntu.com/14.04/ubuntu-14.04.3-server-amd64.iso",
"iso_checksum": "9e5fecc94b3925bededed0fdca1bd417",
"custom_script": ".",
"disk_size": "5000",
"mem_size": "512",
"vcpu_num": "1",
"ansible_config_path": "/etc/ansible",
"ansible_config_file": "ansible.cfg",
"ansible_log_file": "/var/tmp/ansible.log"
},
"builders": [{
"type": "virtualbox-ovf",
"source_path": "trusty64/box.ovf",
"ssh_username": "vagrant",
"ssh_password": "vagrant",
"shutdown_command": "echo 'vagrant' | sudo -S shutdown -P now"
"vm_name": "ubuntu1404",
"type": "qemu",
"format": "qcow2",
"accelerator": "kvm",
"http_directory": "http",
"iso_url": "{{ user `iso_url` }}",
"iso_checksum": "{{ user `iso_checksum` }}",
"iso_checksum_type": "md5",
"ssh_username": "{{ user `ssh_username` }}",
"ssh_password": "{{ user `ssh_password` }}",
"ssh_wait_timeout": "10000s",
"headless": "{{ user `headless` }}",
"boot_wait": "30s",
"boot_command": [
"<esc><esc><enter><wait>",
"/install/vmlinuz auto",
" console-setup/ask_detect=false",
" console-setup/layoutcode=us",
" console-setup/modelcode=SKIP",
" debconf/frontend=noninteractive",
" debian-installer=en_US",
" fb=false",
" initrd=/install/initrd.gz",
" keymap=us debian-installer/keymap=us",
" kbd-chooser/method=us",
" keyboard-configuration/layout=USA",
" keyboard-configuration/variant=USA",
" locale=en_US",
" netcfg/get_domain=vm",
" netcfg/get_hostname=vagrant",
" noapic" ,
" preseed/url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/preseed.cfg",
" -- ",
"<enter>"
],
"shutdown_command": "echo 'vagrant'|sudo -S shutdown -P now",
"disk_size": "{{ user `disk_size`}}",
"qemuargs": [
[
"-m",
"{{user `mem_size`}}M"
],
[
"-smp",
"{{ user `vcpu_num`}}"
]
]
},
{
"vm_name": "ubuntu1404",
"type": "virtualbox-iso",
"http_directory": "http",
"iso_url": "{{ user `iso_url` }}",
"iso_checksum": "{{ user `iso_checksum` }}",
"iso_checksum_type": "md5",
"guest_os_type": "Ubuntu_64",
"guest_additions_path": "VBoxGuestAdditions_{{.Version}}.iso",
"virtualbox_version_file": ".vbox_version",
"headless": "{{ user `headless` }}",
"ssh_username": "{{ user `ssh_username` }}",
"ssh_password": "{{ user `ssh_password` }}",
"ssh_wait_timeout": "10000s",
"boot_wait": "30s",
"boot_command": [
"<esc><esc><enter><wait>",
"/install/vmlinuz noapic ",
"preseed/url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/preseed.cfg ",
"debian-installer=en_US auto locale=en_US kbd-chooser/method=us ",
"hostname={{ .Name }} ",
"fb=false debconf/frontend=noninteractive ",
"keyboard-configuration/modelcode=SKIP keyboard-configuration/layout=USA ",
"keyboard-configuration/variant=USA console-setup/ask_detect=false ",
"keymap=us debian-installer/keymap=us ",
"initrd=/install/initrd.gz -- <enter>"
],
"shutdown_command": "echo 'vagrant'|sudo -S shutdown -P now",
"disk_size": "{{ user `disk_size`}}",
"vboxmanage": [
["modifyvm", "{{.Name}}", "--memory", "{{ user `mem_size`}}"],
["modifyvm", "{{.Name}}", "--cpus", "{{ user `vcpu_num`}}"]
]
}],
"provisioners": [
{
"provisioners": [{
"type": "shell",
"execute_command": "echo '{{user `ssh_password`}}' | {{ .Vars }} sudo -E -S sh '{{ .Path }}'",
"inline": [
"echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers"
]}, {
"type": "shell",
"inline": ["sudo mkdir -p {{ user `ansible_config_path` }}"]
},
{
}, {
"type": "file",
"source": "ansible.cfg",
"destination": "/tmp/ansible.cfg"
},
{
}, {
"type": "shell",
"inline": ["sudo mv /tmp/ansible.cfg {{ user `ansible_config_path` }}/{{ user `ansible_config_file` }}"]
},
{
}, {
"type": "shell",
"inline": [
"sudo echo 'log_path = {{ user `ansible_log_file` }}' >> {{ user `ansible_config_path` }}/{{ user `ansible_config_file` }}",
"touch {{ user `ansible_log_file` }}",
"chmod 666 {{ user `ansible_log_file` }}"
]
},
{
"sudo touch {{ user `ansible_log_file` }}",
"sudo chmod 666 {{ user `ansible_log_file` }}"
]}, {
"type": "shell",
"script": "playbooks/files/ubuntu-ansible.sh"
"environment_vars": [
"CLEANUP_PAUSE={{user `cleanup_pause`}}",
"UPDATE={{user `update`}}",
"INSTALL_VAGRANT_KEY={{user `install_vagrant_key`}}",
"SSH_USERNAME={{user `ssh_username`}}",
"SSH_PASSWORD={{user `ssh_password`}}",
"http_proxy={{user `http_proxy`}}",
"https_proxy={{user `https_proxy`}}",
"ftp_proxy={{user `ftp_proxy`}}",
"rsync_proxy={{user `rsync_proxy`}}",
"no_proxy={{user `no_proxy`}}"
],
"execute_command": "echo 'vagrant' | {{.Vars}} sudo -E -S bash '{{.Path}}'",
"scripts": [
"playbooks/files/update.sh",
"playbooks/files/vagrant.sh",
"playbooks/files/sshd.sh"
]
}, {
"type": "shell",
"script": "playbooks/files/ubuntu-ansible.sh",
"execute_command": "echo 'vagrant' | {{.Vars}} sudo -E -S bash '{{.Path}}'"
}, {
"type": "ansible-local",
"playbook_dir": "playbooks",
"playbook_file": "playbooks/build-main.yaml",
"extra_arguments": ["--verbose"]
}, {
"type": "shell",
"execute_command": "echo 'vagrant' | {{.Vars}} sudo -E -S bash '{{.Path}}'",
"scripts": [
"playbooks/files/vmtool.sh",
"playbooks/files/minimize.sh",
"playbooks/files/cleanup.sh"
]
}],
"post-processors": [{
"type": "vagrant",
"output": "solar-master.box"
}]
"post-processors": [
[
{
"type": "vagrant",
"keep_input_artifact": false,
"only": ["qemu"],
"output": "solar-master-libvirt.box"
},
{
"type": "vagrant",
"keep_input_artifact": false,
"only": ["virtualbox-iso"],
"output": "solar-master-virtualbox.box"
}
]
]
}

71
examples/provisioning/provision.py
View File

@ -1,36 +1,79 @@
#!/usr/bin/env python
import requests
from solar.core import resource
from solar.core import signals
from solar.core import validation
from solar.core.resource import virtual_resource as vr
from solar.events.controls import React
from solar.events.api import add_event
from solar.events.controls import React
discovery_service = 'http://0.0.0.0:8881'
bareon_partitioning = 'http://0.0.0.0:9322/v1/nodes/{0}/partitioning'
bareon_repos = 'http://0.0.0.0:9322/v1/nodes/{0}/repos'
bareon_sync = 'http://0.0.0.0:9322/v1/actions/sync_all'
class NodeAdapter(dict):
def __getattr__(self, name):
try:
return self[name]
except KeyError:
raise AttributeError(name)
@property
def node_id(self):
return self['id']
@property
def partitioning(self):
return requests.get(bareon_partitioning.format(self['id'])).json()
@property
def repos(self):
return requests.get(bareon_repos.format(self['id'])).json()
# Sync hw info about nodes from discovery service into bareon-api
requests.post(bareon_sync)
# Get list of nodes from discovery service
nodes_list = requests.get(discovery_service).json()
# Create slave node resources
node_resources = vr.create('nodes', 'templates/not_provisioned_nodes.yaml', {'nodes': nodes_list})
node_resources = vr.create('nodes', 'templates/not_provisioned_nodes.yaml',
{'nodes': nodes_list})
# Get master node
master_node = filter(lambda n: n.name == 'node_master', node_resources)[0]
with open('/vagrant/tmp/keys/ssh_public') as fp:
master_key = fp.read().strip()
# Dnsmasq resources
for node in nodes_list:
dnsmasq = vr.create('dnsmasq_{0}'.format(node['mac'].replace(':', '_')), 'resources/dnsmasq', {})[0]
node = filter(lambda n: n.name.endswith('node_{0}'.format(node['mac']).replace(':', '_')), node_resources)[0]
master_node.connect(dnsmasq)
node.connect(dnsmasq, {'admin_mac': 'exclude_mac_pxe'})
node = NodeAdapter(node)
node_resource = next(n for n in node_resources
if n.name.endswith('node_{0}'.format(node.node_id)))
event = React(node.name, 'run', 'success', node.name, 'provision')
node_resource.update(
{
'partitioning': node.partitioning,
'master_key': master_key,
'repos': node.repos,
}
)
dnsmasq = vr.create('dnsmasq_{0}'.format(node.node_id),
'resources/dnsmasq', {})[0]
master_node.connect(dnsmasq)
node_resource.connect(dnsmasq, {'admin_mac': 'exclude_mac_pxe'})
event = React(node_resource.name, 'run', 'success', node_resource.name,
'provision')
add_event(event)
event = React(node.name, 'provision', 'success', dnsmasq.name, 'exclude_mac_pxe')
event = React(node_resource.name, 'provision', 'success', dnsmasq.name,
'exclude_mac_pxe')
add_event(event)
event = React(dnsmasq.name, 'exclude_mac_pxe', 'success', node.name, 'reboot')
event = React(dnsmasq.name, 'exclude_mac_pxe', 'success',
node_resource.name, 'reboot')
add_event(event)

1
examples/riak/riaks.py
View File

@ -33,6 +33,7 @@ def setup_riak():
ModelMeta.remove_all()
resources = vr.create('nodes', 'templates/nodes.yaml', {'count': 3})
nodes = [x for x in resources if x.name.startswith('node')]
hosts_services = [x for x in resources if x.name.startswith('hosts_file')]
node1, node2, node3 = nodes
hosts_services = [x for x in resources if x.name.startswith('hosts_file')]

103
examples/riak/riaks_big.py Executable file
View File

@ -0,0 +1,103 @@
#!/usr/bin/env python
# this allows you to create riak cluster as big as you want
import click
import sys
from solar.core import resource
from solar.core import signals
from solar.core import validation
from solar.core.resource import virtual_resource as vr
from solar import errors
from solar.interfaces.db import get_db
from solar.events.controls import React, Dep
from solar.events.api import add_event
db = get_db()
NODES = 3
def setup_riak(nodes_num=None, hosts_mapping=False):
if nodes_num is None:
nodes_num = NODES
db.clear()
resources = vr.create('nodes', 'templates/nodes.yaml', {'count': nodes_num})
nodes = [x for x in resources if x.name.startswith('node')]
hosts_services = [x for x in resources if x.name.startswith('hosts_file')]
riak_services = []
ips = '10.0.0.%d'
for i in xrange(nodes_num):
num = i + 1
r = vr.create('riak_service%d' % num,
'resources/riak_node',
{'riak_self_name': 'riak%d' % num,
'riak_hostname': 'riak_server%d.solar' % num,
'riak_name': 'riak%d@riak_server%d.solar' % (num, num)})[0]
riak_services.append(r)
for i, riak in enumerate(riak_services):
nodes[i].connect(riak)
for i, riak in enumerate(riak_services[1:]):
riak_services[0].connect(riak, {'riak_name': 'join_to'})
if hosts_mapping:
for riak in riak_services:
for hosts_file in hosts_services:
riak.connect_with_events(hosts_file,
{'riak_hostname': 'hosts:name',
'ip': 'hosts:ip'})
res_errors = resource.validate_resources()
for r, error in res_errors:
click.echo('ERROR: %s: %s' % (r.name, error))
has_errors = False
if has_errors:
click.echo("ERRORS")
sys.exit(1)
events = []
for x in xrange(nodes_num):
i = x + 1
if hosts_mapping:
events.append(Dep('hosts_file%d' % i, 'run', 'success', 'riak_service%d' % i, 'run'))
if i >= 2:
events.append(React('riak_service%d' % i, 'run', 'success', 'riak_service%d' % i, 'join'))
events.append(React('riak_service%d' % i, 'join', 'success', 'riak_service1', 'commit'))
for event in events:
add_event(event)
click.echo('Use solar changes process & orch')
sys.exit(0)
@click.group()
def main():
pass
@click.command()
@click.argument('nodes_count', type=int)
@click.argument('hosts_mapping', type=bool)
def deploy(nodes_count, hosts_mapping):
click.secho("With big nodes_count, this example is DB heavy, it creates NxN connections, continue ? [y/N] ", fg='red', nl=False)
c= click.getchar()
if c in ('y', 'Y'):
setup_riak(nodes_count, hosts_mapping)
else:
click.echo("Aborted")
if __name__ == '__main__':
main.add_command(deploy)
main()

3
resources/not_provisioned_node/actions/provision.sh
View File

@ -6,4 +6,5 @@ DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
# TODO should be a way to render configs, in order to do this
# we should have scripts dir variable passed from above
sed -i "s|<ROOT>|${DIR}|" "${DIR}"/templates/agent.config
provision --input_data_file "${DIR}"/templates/provisioning.json --config-file "${DIR}"/templates/agent.config
provision --log-file /tmp/fa_provision.log -d --data_driver nailgun_simple --input_data_file "${DIR}"/templates/provisioning.json --config-file "${DIR}"/templates/agent.config

9
resources/not_provisioned_node/meta.yaml
View File

@ -11,9 +11,15 @@ input:
ip:
schema: str!
value:
master_key:
schema: str!
value:
admin_mac:
schema: str!
value:
repos:
schema: list!
value: []
name:
schema: str
value: a node
@ -21,5 +27,8 @@ input:
schema: str!
value: $uuid
reverse: True
partitioning:
schema: dict!
value:
tags: [resources=node]

2
resources/not_provisioned_node/templates/agent.config
View File

@ -1,2 +1,4 @@
[DEFAULT]
debug=true
nc_template_path=<ROOT>/templates/cloud-init-templates/
log_file=/var/log/fuel-agent.log

84
resources/not_provisioned_node/templates/provisioning.json → resources/not_provisioned_node/templates/provisioning.json.jinja
View File

@ -1,4 +1,5 @@
{
"partitioning": {{ partitioning | to_pretty_json }},
"profile": "ubuntu_1404_x86_64",
"name_servers_search": "\"example.com\"",
"uid": "2",
@ -8,8 +9,8 @@
"mac_address": "08:00:27:6e:6d:b4"
},
"eth0": {
"ip_address": "10.0.2.15",
"dns_name": "node-8.test.domain.local",
"ip_address": "{{ ip }}",
"dns_name": "{{ name }}.test.domain.local",
"netmask": "255.255.255.0",
"static": "0",
"mac_address": "08:00:27:ea:35:e7"
@ -42,72 +43,7 @@
"local": "/var/www/nailgun/ubuntu/x86_64/images/linux",
"remote_relative": "dists/trusty/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64/linux"
},
"repos": [
{
"name": "ubuntu",
"section": "main universe multiverse",
"uri": "http://archive.ubuntu.com/ubuntu/",
"priority": null,
"suite": "trusty",
"type": "deb"
},
{
"name": "ubuntu-updates",
"section": "main universe multiverse",
"uri": "http://archive.ubuntu.com/ubuntu/",
"priority": null,
"suite": "trusty-updates",
"type": "deb"
},
{
"name": "ubuntu-security",
"section": "main universe multiverse",
"uri": "http://archive.ubuntu.com/ubuntu/",
"priority": null,
"suite": "trusty-security",
"type": "deb"
},
{
"name": "mos",
"section": "main restricted",
"uri": "http://127.0.0.1:8080/2015.1.0-7.0/ubuntu/x86_64",
"priority": 1050,
"suite": "mos7.0",
"type": "deb"
},
{
"name": "mos-updates",
"section": "main restricted",
"uri": "http://mirror.fuel-infra.org/mos/ubuntu/",
"priority": 1050,
"suite": "mos7.0-updates",
"type": "deb"
},
{
"name": "mos-security",
"section": "main restricted",
"uri": "http://mirror.fuel-infra.org/mos/ubuntu/",
"priority": 1050,
"suite": "mos7.0-security",
"type": "deb"
},
{
"name": "mos-holdback",
"section": "main restricted",
"uri": "http://mirror.fuel-infra.org/mos/ubuntu/",
"priority": 1100,
"suite": "mos7.0-holdback",
"type": "deb"
},
{
"name": "Auxiliary",
"section": "main restricted",
"uri": "http://127.0.0.1:8080/2015.1.0-7.0/ubuntu/auxiliary",
"priority": 1150,
"suite": "auxiliary",
"type": "deb"
}
],
"repos": {{ repos | to_pretty_json }},
"metadata": {
"always_editable": true,
"weight": 50,
@ -118,7 +54,7 @@
"remote_relative": "dists/trusty/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64/initrd.gz"
}
},
"authorized_keys": [],
"authorized_keys": ["{{ master_key }}"],
"mlnx_iser_enabled": false,
"mco_pskey": "Gie6iega9ohngaenahthohngu8aebohxah9seidi",
"mco_user": "guest",
@ -128,12 +64,12 @@
"image_data": {
"/boot": {
"container": "gzip",
"uri": "http://10.0.0.2:8001/tmp/targetimages/env_3_ubuntu_1404_amd64-boot.img.gz",
"uri": "http://10.0.0.2:8001/tmp/targetimages/env_1_ubuntu_1404_amd64-boot.img.gz",
"format": "ext2"
},
"/": {
"container": "gzip",
"uri": "http://10.0.0.2:8001/tmp/targetimages/env_3_ubuntu_1404_amd64.img.gz",
"uri": "http://10.0.0.2:8001/tmp/targetimages/env_1_ubuntu_1404_amd64.img.gz",
"format": "ext4"
}
},
@ -212,9 +148,9 @@
"admin_net": "10.20.0.0/24",
"mco_host": "localhost"
},
"name": "node-2",
"hostname": "node-2.example.com",
"slave_name": "node-2",
"name": "{{ name }}",
"hostname": "{{ name }}.example.com",
"slave_name": "{{ name }}",
"power_pass": "/root/.ssh/bootstrap.rsa",
"netboot_enabled": "1"
}

3
resources/riak_node/meta.yaml
View File

@ -37,3 +37,6 @@ input:
join_to:
schema: str
value:
storage_backend:
schema: str!
value: bitcask

2
resources/riak_node/templates/riak.conf → resources/riak_node/templates/riak.conf.jinja
View File

@ -331,7 +331,7 @@ anti_entropy = active
##
## Acceptable values:
## - one of: bitcask, leveldb, memory, multi
storage_backend = bitcask
storage_backend = {{storage_backend}}
## Controls which binary representation of a riak value is stored
## on disk.

2
solar/requirements.txt
View File

@ -3,7 +3,7 @@ ply
click==4.0
jinja2==2.7.3
networkx>=1.10
PyYAML==3.11
PyYAML>=3.1.0
jsonschema==2.4.0
requests==2.7.0
dictdiffer==0.4.0

18
solar/solar/cli/resource.py
View File

@ -62,6 +62,24 @@ def action(dry_run_mapping, dry_run, action, resource):
str(key)
))
def backtrack_single(i):
def format_input(i):
return '{}::{}'.format(i.resource.name, i.name)
if isinstance(i, list):
return [backtrack_single(bi) for bi in i]
if isinstance(i, dict):
return {
k: backtrack_single(bi) for k, bi in i.items()
}
bi = i.backtrack_value_emitter(level=1)
if isinstance(i, orm.DBResourceInput) and isinstance(bi, orm.DBResourceInput) and i == bi:
return (format_input(i), )
return (format_input(i), backtrack_single(bi))
@resource.command()
@click.option('-v', '--values', default=False, is_flag=True)
@click.option('-i', '--input', default=None)

18
solar/solar/core/resource/virtual_resource.py
View File

@ -13,6 +13,7 @@
# License for the specific language governing permissions and limitations
# under the License.
from collections import defaultdict
import os
from StringIO import StringIO
import yaml
@ -168,14 +169,19 @@ def update_resources(template_resources):
def update_inputs(child, args):
child = load_resource(child)
connections, assignments = parse_inputs(args)
parents = defaultdict(lambda: defaultdict(dict))
for c in connections:
mapping = {}
parent = load_resource(c['parent'])
events = c['events']
use_defaults = not c['events'] is False
mapping[c['parent_input']] = c['child_input']
mapping = {c['parent_input']: c['child_input']}
parents[c['parent']]['mapping'].update(mapping)
if parents[c['parent']].get('events', None) is None:
parents[c['parent']]['events'] = c['events']
for parent, data in parents.iteritems():
parent = load_resource(parent)
use_defaults = not data['events'] is False
mapping = data['mapping']
parent.connect_with_events(
child, mapping, events, use_defaults=use_defaults)
child, mapping, {}, use_defaults=use_defaults)
child.update(assignments)

2
solar/solar/core/transports/bat.py
View File

@ -54,6 +54,7 @@ class BatTransport(SolarTransport):
super(BatTransport, self).__init__(*args, **kwargs)
self._cache = {}
self._used_transports = []
self._other_remember = None
def select_valid_transport(self, resource, *args, **kwargs):
key_name = '_bat_transport_%s' % self._mode
@ -114,4 +115,3 @@ class BatRunTransport(RunTransport, BatTransport):
def run(self, resource, *args, **kwargs):
transport = self.select_valid_transport(resource)
return transport.run(resource, *args, **kwargs)

2
solar/solar/test/test_virtual_resource.py
View File

@ -134,7 +134,7 @@ def test_add_connections(mocker, resources):
'alias': 'ser1'
}
vr.update_inputs('service1', args)
assert mocked_signals.call_count == 3
assert mocked_signals.call_count == 2
def test_add_list_values(mocker, resources):

20
templates/not_provisioned_nodes.yaml
View File

@ -1,25 +1,25 @@
id: not_provisioned_nodes
resources:
{% for node in nodes %}
{% set mac = node.mac | replace(':', '_') %}
- id: ssh_transport{{ mac }}
{% set id = node.id | replace(':', '_') %}
- id: ssh_transport{{ id }}
from: resources/transport_ssh
values:
ssh_user: 'root'
ssh_key: '/vagrant/tmp/keys/ssh_private'
- id: transports{{mac}}
- id: transports{{id}}
from: resources/transports
values:
transports:key: ssh_transport{{mac}}::ssh_key
transports:user: ssh_transport{{mac}}::ssh_user
transports:port: ssh_transport{{mac}}::ssh_port
transports:name: ssh_transport{{mac}}::name
- id: node_{{mac}}
transports:key: ssh_transport{{id}}::ssh_key
transports:user: ssh_transport{{id}}::ssh_user
transports:port: ssh_transport{{id}}::ssh_port
transports:name: ssh_transport{{id}}::name
- id: node_{{id}}
from: resources/not_provisioned_node
values:
ip: {{node.ip}}
transports_id: transports{{mac}}::transports_id
name: node_{{mac}}
transports_id: transports{{id}}::transports_id
name: node_{{id}}
admin_mac: {{node.mac}}
{% endfor %}

2
vagrant-settings.yaml_defaults
View File

@ -1,4 +1,4 @@
# rename it to vagrant-settings.yaml then Vagrantfile
# copy it to vagrant-settings.yaml then Vagrantfile
# will use values from this file
slaves_count: 2