compose: Add "image" sub-command

The purpose of the "image" sub-command is to allow a user
to create a raw disk-image from an ostree branch to be used
by libvirt. This allows the user to test an ostree branch
by making it bootable.

The way that it works is that the user provides a debos
configuration file in order to create a raw disk. A sample
configuration file is included in this commit. In order
to create a disk image apt-ostree will create a scratch
ostree repository and copy the desired branch into
the scratch repo. Once that happens debos will run
with a specific branch chosen by the user.

Debos is a golang based tool that allows the creation of
Debian based OS images simpler.

To build an image from a provided configuration file, one
simply runs the following command:

   sudo apt-ostree compose image --repo <path to ostree repo> \
     --base <path to configuration directory> \
     <ostree branch>

More details can be found in the man page.

Testing:
PASSED Installed apt-ostree from git repo.
PASSED Run "apt-ostree compose image --base config/debian/image \
        --repo=/repo test"
PASSED Checked for raw disk image in
       /var/tmp/apt-ostree/build/test/image

Story: 2010867
Task: 48556

Depends-On: https://review.opendev.org/c/starlingx/apt-ostree/+/890704

Change-Id: I4ce64214dc3bb59ef03b35c2c27def017ea35487
Signed-off-by: Charles Short <charles.short@windriver.com>

.yamllint

@@ -8,3 +8,4 @@ rules:
 
 ignore: |
   .zuul.yaml
+  config/debian/bookworm/image/image.yaml

apt_ostree/cmd/compose/__init__.py

@@ -8,6 +8,7 @@ SPDX-License-Identifier: Apache-2.0
 import click
 
 from apt_ostree.cmd.compose.create import create
+from apt_ostree.cmd.compose.image import image
 
 
 @click.group(help="Commands to build ostree repo/image")
@@ -17,3 +18,4 @@ def compose(ctxt):
 
 
 compose.add_command(create)
+compose.add_command(image)

apt_ostree/cmd/compose/image.py

@@ -0,0 +1,56 @@
+"""
+Copyright (c) 2023 Wind River Systems, Inc.
+
+SPDX-License-Identifier: Apache-2.0
+
+"""
+
+import shutil
+
+import click
+
+from apt_ostree.cmd.options import compose_options
+from apt_ostree.cmd import pass_state_context
+from apt_ostree.log import complete_step
+from apt_ostree.log import log_step
+from apt_ostree.utils import run_command
+
+
+@click.command(help="Create an raw image from ostree branch")
+@pass_state_context
+@compose_options
+def image(state, repo, base, branch):
+    click.secho(f"Found ostree repository: {state.repo}")
+    click.secho(f"Found ostree branch: {state.branch}")
+    with complete_step(f"Setting up workspace for {state.branch}"):
+        workdir = state.workspace.joinpath(f"build/{state.branch}")
+        img_dir = workdir.joinpath("image")
+        ostree_repo = img_dir.joinpath("ostree_repo")
+        if img_dir.exists():
+            shutil.rmtree(img_dir)
+        shutil.copytree(
+            state.base.joinpath("image"),
+            img_dir, dirs_exist_ok=True)
+
+    with complete_step("Creating local ostree repository"):
+        log_step("Creating image build repository")
+        run_command(
+            ["ostree", "init", f"--repo={str(ostree_repo)}"],
+            cwd=img_dir)
+        log_step(f"Pulling {state.branch} in image build repository")
+        run_command(
+            ["ostree", "pull-local", f"--repo={str(ostree_repo)}",
+             str(state.repo), str(state.branch)],
+            cwd=img_dir)
+        log_step("Running debos...")
+
+        cmd = ["debos",
+               "-t", f"branch:{state.branch}",
+               ]
+        if state.debug:
+            cmd += ["-v"]
+        cmd += ["image.yaml"]
+
+        run_command(cmd, cwd=img_dir)
+
+    click.secho(f"Image can be found in {img_dir}")

apt_ostree/tests/compose/test_create.py

@@ -16,3 +16,8 @@ class TestComposeCLI(base.TestCase):
         runner = CliRunner()
         result = runner.invoke(cli, ["compose", "--help"])
         assert result.exit_code == 0
+ 
+    def test_compose_image(self):
+        runner = CliRunner()
+        result = runner.invoke(cli, ["compose", "image", "--help"])
+        assert result.exit_code == 0

config/debian/bookworm/image/image.yaml

@@ -0,0 +1,62 @@
+# Copyright (c) 2023 Wind River Systems, Inc.
+#
+# SPDX-License-Identifier: Apache-2.0
+#
+{{- $architecture := or .architecture "amd64" -}}
+{{- $image := or .image (printf "debian-ostree-qemu-uefi-%s.img" $architecture) -}}
+{{- $cmdline := or .cmdline "console=tty0 console=ttyS0,115200n8 rootwait rw fsck.mode=auto fsck.repair=yes systemd.gpt_auto=false" -}}
+{{- $branch := or .branch "debian/bookworm" -}}
+{{- $repo := or .repo "ostree_repo" -}}
+{{- $size := or .size "3G" -}}
+
+architecture: {{ $architecture }}
+
+actions:
+  - action: image-partition
+    imagename: {{ $image }}
+    imagesize: {{ $size }}
+    partitiontype: gpt
+
+    mountpoints:
+      - mountpoint: /
+        partition: system
+      - mountpoint: /boot/efi
+        partition: EFI
+
+    partitions:
+      - name: EFI
+        fs: vfat
+        start: 0%
+        end: 256M
+        flags: [boot]
+      - name: system
+        fs: ext4
+        start: 266M
+        end: 100%
+
+  # Reset the rootfs to allow to deploy OSTree from a clean rootfs
+  - action: run
+    description: Reset rootfs before deploying OSTree
+    chroot: false
+    command: find ${ROOTDIR} -maxdepth 1 -mindepth 1 -exec rm -rf {} \;
+
+  - action: ostree-deploy
+    repository: ostree_repo
+    branch: {{ $branch }}
+    os: debian
+    append-kernel-cmdline: {{ $cmdline }}
+
+  - action: run
+    description: enable signature verification
+    chroot: false
+    command: ostree --repo="${ROOTDIR}/ostree/repo" config set 'remote "origin"'.sign-verify "true"
+
+  - action: run
+    description: enable update bundle verification
+    chroot: false
+    command: ostree --repo="${ROOTDIR}/ostree/repo" config set core.sign-verify-deltas "true"
+
+  - action: run
+    description: install bootloader
+    chroot: false
+    script: scripts/setup-uefi-bootloader.sh debian

config/debian/bookworm/image/scripts/setup-uefi-bootloader.sh

@@ -0,0 +1,52 @@
+#!/bin/sh
+
+# Based on original script from OStree upstream:
+# https://github.com/ostreedev/ostree/blob/master/src/switchroot/switchroot.sh
+# Copyright (C) 2013, 2016 Collabora Ltd
+# Copyright (C) 2023 Wind River Systems, Inc.
+
+set -eu
+
+sysroot=${ROOTDIR}
+
+osname=$1
+
+bootconf=$sysroot/boot/loader/entries/ostree-1-${osname}.conf
+
+if [ ! -f "$bootconf" ]; then
+    echo "OStree setup is not available!"
+    exit 1
+fi
+
+ostree=$(grep -o 'ostree=[/.[:alnum:]]\+' $bootconf)
+ostree=${ostree#*=}
+# Due symlink
+ostree=$(realpath $sysroot$ostree)
+
+mkdir -p $ostree/boot/efi
+
+## /etc/machine-id must be writable to allow to work systemd-nspawn
+# but original `machine-id` file must stay empty in build time.
+touch /tmp/machine-id
+mount --bind /tmp/machine-id $ostree/etc/machine-id
+
+# NB: The native resolv.conf management is supported only from systemd v.232.
+systemd-nspawn --resolv-conf=off -D $ostree systemd-machine-id-setup
+
+# install EFI
+systemd-nspawn \
+  --resolv-conf=off \
+  --bind $sysroot/boot:/boot \
+  --bind $sysroot/boot/efi:/boot/efi \
+  --bind $sysroot/ostree/deploy/${osname}/var:/var \
+  -D $ostree bootctl --path=/boot/efi install
+
+umount $ostree/etc/machine-id
+rmdir $ostree/boot/efi
+
+# Copy config, kernel and initrd
+# Change the name of config to unify with patch added in T4469
+rsync -Pav $sysroot/boot/ostree $sysroot/boot/efi/
+mkdir $sysroot/boot/efi/loader/entries/
+cp $bootconf $sysroot/boot/efi/loader/entries/ostree-0-1.conf
+rm -f $sysroot/boot/efi/loader/loader.conf

doc/source/cli/index.rst

@@ -1,5 +1,111 @@
-================================
-Command line interface reference
-================================
+.. _manpage:
 
-CLI reference of apt-ostree.
+=====================
+:program:`apt-ostree`
+=====================
+
+Hybrid package/image manager for ostree
+
+SYNOPSIS
+========
+
+:program:`apt-ostree` [<global-options>] <command> [<command-arguments>]
+
+:program:`apt-ostree help` <command>
+
+DESCRIPTION
+===========
+
+:program:`apt-ostree` provides a common commandline-interface to apt and
+ostree. It is generally equivalent to the CLI provided by ostree and apt,
+but with a distinct and consistent command structure.
+
+OPTIONS
+=======
+
+:program:`apt-ostree` takes global options that control overall behaviour and command-specific
+ options that control the command operation. Most global options have a
+ corresponding environment variable that may also be used to set the value.
+ If both are present, the command-line option takes priority. The environment
+ variable names are derived from the option name by dropping the leading dashes
+ ('--'), converting each embedded dash ('-') to an underscore ('_'),
+ and converting to upper case.
+
+:program:`apt-ostree` recognizes the following global options:
+
+.. option:: --debug
+
+   Show tracbacks on errors and set verbosity to debug.
+
+.. option:: --workspace
+
+   :program:`openstack` will create a default workspace to build images from.
+   The default is '/var/tmp/apt-ostree'.
+
+
+COMMANDS
+========
+
+To get a list of the available commands::
+
+    apt-ostree --help
+
+To get a description of a specific command::
+
+    apt-ostree help <command>
+
+Command Objects
+---------------
+
+The list of command objects is growing longer with the addition of apt-ostree
+project support.  The object names may consist of multiple words to compose a
+unique name.
+
+Command Actions
+---------------
+
+The actions used by apt-ostree are defined with specific meaning to
+provide a consistent behavior for each object. Some actions have
+logical opposite actions,and those pairs wil
+always match for any object that uses them.
+
+EXAMPLES
+========
+
+Create an ostree repository and branch::
+
+      apt-ostree \
+         --repo /home/user/ostree_repo \
+         --base config/debian/bookworm \
+         debian/bookworm
+
+Create an ostree image based off a ostree branch::
+
+      apt-ostree \
+         --repo /home/user/ostree-repo \
+         --base config/debian/bookworm/image \
+         debian/bookworm
+
+BUGS
+====
+
+Bug reports are accepted at the python-openstackclient StoryBoard project
+"https://storyboard.openstack.org/#!/project/975".
+
+
+AUTHORS
+=======
+
+Please refer to the AUTHORS file distributed with apt-ostree.
+
+
+COPYRIGHT
+=========
+
+Copyright 2023 Wind River Inc and the authors listed in the AUTHORS file.
+
+
+LICENSE
+=======
+
+http://www.apache.org/licenses/LICENSE-2.0

doc/source/contributor/contributing.rst

@@ -4,9 +4,9 @@ So You Want to Contribute...
 
 For general information on contributing to OpenStack, please check out the
 `contributor guide <https://docs.openstack.org/contributors/>`_ to get started.
-It covers all the basics that are common to all OpenStack projects: the accounts
-you need, the basics of interacting with our Gerrit review system, how we
-communicate as a community, etc.
+It covers all the basics that are common to all OpenStack projects:
+the accounts you need, the basics of interacting with
+our Gerrit review system, how we communicate as a community, etc.
 
 Below will cover the more project specific information you need to get started
 with replace with the service it implements.

docker/Dockerfile

@@ -1,8 +1,13 @@
+# Copyright (c) 2023 Wind River Systems, Inc.
+#
+# SPDX-License-Identifier: Apache-2.0
+#
 FROM debian:bullseye
 
 RUN apt-get update && \
     apt-get install -y \
 	bdebstrap \
+	debos \
 	ostree \
 	python3 \
 	python3-click \

test-requirements.txt

@@ -10,3 +10,4 @@ oslotest>=1.10.0 # Apache-2.0
 stestr>=1.0.0 # Apache-2.0
 testtools>=1.4.0 # MIT
 yamllint<1.26.4  # GPLv3
+doc8

tox.ini

@@ -50,8 +50,10 @@ commands =
 [testenv:debug]
 commands = oslo_debug_helper {posargs}
 
-[testenv:yamllint]
-commands = yamllint config/
+[testenv:linters]
+commands =
+  doc8 doc/
+  yamllint config/
 
 [flake8]
 # E123, E125 skipped as they are invalid PEP-8.
@@ -60,3 +62,12 @@ show-source = True
 ignore = E123,E125
 builtins = _
 exclude=.venv,.git,.tox,dist,doc,*lib/python*,*egg,build
+
+[testenv:bindep]
+# Do not install any requirements. We want this to be fast and work even if
+# system dependencies are missing, since it's used to tell you what system
+# dependencies are missing! This also means that bindep must be installed
+# separately, outside of the requirements files.
+skip_install = True
+deps = bindep
+commands = bindep test