OpenWrt ImageBuilder Automation
===============================

This project aims to provide a convenient, declarative means to assemble OpenWrt
installation images in parallel using the ImageBuilder[0].


Advanced users of OpenWrt may use the ImageBuilder to assemble custom
installation images from pre-built binary packages published by the OpenWrt
project. This has a number of advantages if you have requirements that surpass
what pre-built installation images provided by OpenWrt offer, such as extra
packages or configuration to be included in your images/installed on your
devices, and makes upgrading OpenWrt devices a much smoother experience.

Creating these images can become tedious once you need to repeat the exercise
for multiple devices that differ in architecture/hardware, desired
packages/release, etc. - `openwrt_autobuild.py` strives to save you time when
doing this.


License, Disclaimer, Compatibility
----------------------------------

    openwrt_autobuild.py - OpenWrt ImageBuilder Automation
    Copyright (C) 2021-2023 Johannes Truschnigg

    This program is free software: you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation, either version 3 of the License, or
    (at your option) any later version.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

    You should have received a copy of the GNU General Public License
    along with this program.  If not, see <https://www.gnu.org/licenses/>.

This program is not officially endorsed by the OpenWrt project, and it may not
be able to help you assemble images for all past, current, and future OpenWrt
releases. It is good enough for its author though, and helps him keep a handful
of devices up to date with (at the time of writing) fresh OpenWrt 19.07 builds.

The software is developed, tested and used primarily on Debian GNU/Linux
(stable) for amd64 architectures.


Requirements
------------

`openwrt_autobuild.ini` requires a host system that is capable of extracting
and executing the OpenWrt ImageBuilder, and requires a Python 3.7 (or later)
installation in addition to that.


Getting Started
---------------

To make use of this tool, you will need to know how to operate the ImageBuilder
as provided by OpenWrt.

To get you off the ground, simply copy or rename the provided example config
`openwrt_autobuild.ini.sample` to `openwrt_autobuild.ini`, and edit all
required parameters to your own liking. Then, execute `./openwrt_autobuild.py`
located in the same directory as the configuration file.

`openwrt_autobuild.py` will try to automatically determine and fetch (and
locally cache) all ImageBuilder releases required to executing the jobs you
specify in your configuration. After that has succeeded, it will populate a
number of build jobs that will be executed in parallel, automating the
ImageBuilder machinery to translate your configuration stanzas into
ready-to-be-installed image files. After the builds have completed, you will
find all generated image files in the `./__output/` directory.


A real-world example for a basic configuration looks like this:

```ini
[DEFAULT]
release = 19.07.7
builddir = /tmp/owrt_build

[rtx86]
target = x86
subtarget = 64
device = Generic
packages = luci-app-upnp luci-app-sqm luci-app-firewall luci-app-opkg luci f2fs-tools kmod-usb-storage powertop fdisk block-mount kmod-fs-vfat dosfstools usbutils rsyncd openssl-util libustream-openssl20150806 ca-certificates iperf3 luci-proto-wireguard kmod-igb kmod-wireguard luci-app-openvpn openvpn-openssl luci-app-vpn-policy-routing ethtool -kmod-bnx2 -kmod-e1000 -kmod-r8169 -bnx2-firmware -kmod-e1000e -kmod-ppp -kmod-pppoe -kmod-pppox -luci-proto-ppp -ppp -ppp-mod-pppoe

[fritz-old]
release = 18.06.9
url = https://downloads.openwrt.org/releases/%(release)s/targets/%(target)s/%(subtarget)s/openwrt-imagebuilder-%(release)s-%(target)s.Linux-x86_64.tar.xz
target = ipq40xx
subtarget = generic
device = avm_fritzbox-4040
packages = luci openssl-util ca-certificates iperf3 f2fs-tools iw-full kmod-fs-f2fs kmod-usb-dwc3-of-simple kmod-usb-storage-uas block-mount luci-app-sqm luci-app-upnp mtd rsyncd usbutils wpad -wpad-basic -wpad-mini

[4040-snap0]
release = snapshots
make_clean = True
url = https://downloads.openwrt.org/%(release)s/targets/%(target)s/%(subtarget)s/openwrt-imagebuilder-%(target)s-%(subtarget)s.Linux-x86_64.tar.xz
sha256url = https://downloads.openwrt.org/%(release)s/targets/%(target)s/%(subtarget)s/sha256sums
target = ipq40xx
subtarget = generic
device = avm_fritzbox-4040
packages = luci ca-certificates iperf3 f2fs-tools kmod-fs-f2fs kmod-usb-storage-uas block-mount luci-app-sqm luci-app-upnp mtd rsyncd usbutils
disabled_services = rsyncd

[4040-snap1]
release = snapshots
make_clean = True
url = https://downloads.openwrt.org/%(release)s/targets/%(target)s/%(subtarget)s/openwrt-imagebuilder-%(target)s-%(subtarget)s.Linux-x86_64.tar.xz
sha256url = https://downloads.openwrt.org/%(release)s/targets/%(target)s/%(subtarget)s/sha256sums
target = ipq40xx
subtarget = generic
device = avm_fritzbox-4040

```

With this content in `openwrt_autobuild.ini`, executing `openwrt_autobuild.py`
will download three ImageBuilder tarballs, unpack them, and then queue all four
builds across three parallel queues. Progress is reported incrementally on a
per-build basis.


Notes, Limitations, Bugs
------------------------

### Files and directories
The default configuration makes all filesystem operations happen relative to
the CWD in three directories:

 1. `./__cache` - contains downloaded ImageBuilder tarball (identified by their sha256sum)
 2. `./__build` - contains extracted ImageBuilder files and temporary build artifacts
 3. `./__output` - contains logs and final build artifacts (= OpenWrt installation images)

You can override these paths in the config file - in some scenarios, it might
be a good idea to keep `builddir` on tmpfs. Note that you can **not**
meaningfully override these values per-job/image - but if you try to,
`openwrt_autobuild.py` will probably just blow up.


### Optional arguments / alternative INI files and limiting build scope

If you supply a list of arguments to `openwrt_autobuild.py`, the first argument
must specify the path to an `openwrt_autobuild.ini`-style configuration file.
All other arguments are interpreted as job labels (as found in the supplied INI
file), which name jobs to be queued for building. This allows you to limit the
number of performed builds to a subset of all jobs listed in an INI
configration file. An example invocation, based on the `openwrt_autobuild.ini`
sample from above, looks like this:

```
./openwrt_autobuild.py openwrt_autobuild.ini rtx86 4040-snap0
```

This would cause the `4040-snap1` and `fritz-old` job definitions from the
example not to be appened to the build queue, and only `rtx86` and `4040-snap0`
to be built.


### Resource consumption
There is no explicit upper bound for the number of images to queue/build in
parallel, so if you build a HUGE number of images for different architectures,
you will end up with a HUGE number of builds executing at the same time. You
have to make sure your host machine is up to the task.


[0]: https://openwrt.org/docs/guide-user/additional-software/imagebuilder