Introduction
============

The fab is a product fabrication framework.

This version of Fab is actually the second generation framework, the
first being referred to as oldfab.

The word "Fab" originates from the microelectronics industry. A fab, or
fabrication plant is a factory where devices (eg. Integrated circuits)
are manufactured for one of more customers. A fab is semantically
connected to the most cutting edge technological factories in existence
(Silicon chip foundries)

A fab is a very tightly controlled environment (clean room), but instead
of keeping out physical impurities (e.g., dust and dirt), the Fab is
used in fabricating systems while tightly controlling "logical"
impurities (e.g., security threats, malware, etc.)

These release notes only contain a high-level overview, please refer to
the design notes for detailed information, and help from the commands
themselves.

Overview
========

The fab provides 'toolchain' utilities, which allows us to build products 
and collaborate on them using the same workflow and tools used on 
software projects.

Building is performed per-product, each in its own directory. We leverage 
'make' to implement the 'build pipeline', git and covin for revision 
control and collaboration.

The output of a product is the product itself, and a recipe (very small 
footprint compared to the product) which can be use to automatically 
reproduce the product bit for bit.
                   
Terminology
===========

product
-------

The final product used by the end-user. The product is generated by
formatting the "patched root"

root.patched
------------

The chroot'able root filesystem of a product:

* patched manually or automatically
* can be re-created automatically by applying the root patch as an
  overlay to the root component.

root.build
----------

The chroot'able root filesystem of a product, built by applying the
"root.spec" on the bootstrap.

bootstrap
---------

The minimal chroot'able filesystem used to bootstrap the root, built
from a "bootstrap.spec".

spec
----

A set of (package name, package version tuples):

* a spec is created from a plan against a specific pool
* the same plan will generate different specs against different pools

plan
----

A set of package names.

root plan
'''''''''

The plan from which the root.spec is created, by looking up the
dependencies of listed packages *recursively*.

bootstrap plan
''''''''''''''

The plan from which we create the bootstrap spec, without recursively
looking dependencies.

Inheriting from contrib/product.mk
==================================

Overview
--------

fab's product.mk is designed to be configurable and extendable with
*define* based hooks and variables which should be set BEFORE including
product.mk, because various elements are evaluated at include time
(e.g., target prerequisites, variable exports)::

    <target>/pre  # rules before default body (default: empty)
    <target>/post # rules after default body (default: empty)

    <target>/deps # override default dependencies for a rule
    <target>/deps/extra # extra dependencies for rule (default: empty)

    override built-in variables (e.g., ISOLABEL, CONF_SCRIPTS)

    include path/to/product.mk

Special exception - if you want to override built in built-in target
rules for a target (I.e., when a /pre or /post hook isn't enough),
you'll need to define them AFTER including the shared Makefile::

    <target>/body # body of rules (default: defined, but can be overridden)

product.mk was originally based on pyproject-common's inheritable
Makefiles pyproject.mk and debian-rules.mk. The main difference is that
pyproject.mk is more flexible and has a better documented API (I.e.,
make help). 

For example, you can not only inherit from product.mk in a Makefile, but
you can also run product.mk as a standalone program which inherits
variable definitions from its environment.

When you inherit from product.mk in a product Makefile, you can override
built-in variables (but not built-in targets) before you include
product.mk. This is because in product.mk built-in variables are
assigned on the condition that they are not already set previously.

In a nutshell: define everything before including product.mk except for
<target>/body overrides (which should be rarely needed - if ever).

extension API
-------------

You set the following defines BEFORE including the shared Makefile because
target prerequisites are evaluated at include time::

    <target>/pre  # rules before default body (default: empty)
    <target>/post # rules after default body (default: empty)

    <target>/deps # override default dependencies for a rule
    <target>/deps/extra # extra dependencies for rule (default: empty)

Special case - if you want to override built in built-in rules for a target,
you'll need to define them AFTER including the shared Makefile::

    <target>/body # body of rules (default: defined, but can be overridden)

product.mk API
--------------

You can view which targets exist via the Makefile's embedded help
target::

    $ cd fab/contrib
    $ make -f product.mk help

Build-time product configuration
================================

Background
----------

I've extended fab and product.mk to support product configuration at
build-time.  This additional functionality is designed to address a few
problems we've been having:

1) packages shouldn't perform product-specific configurations

   For example, casper including scripts that configure users and such at
   boot time.
   
   This violates separation of concerns and prevents packages from
   fulfilling their full utility as generic, reusable building blocks. It
   also increases the accidental complexity of the system by introducing
   unnecessary interdependencies.
   
   Also, there is often significant overhead in changing a package to
   configure them to suit a specific product. This is especially true for
   stock packages, but generally creating multiple variants of a package
   just to support different configurations is inconvenient and time
   consuming.

2) boot time is not the correct time to perform product configuration

   It doesn't scale, it lengthens the boot process and it limits the
   re-usability of casper.

3) support adjustments required for different releases with having to
   duplicate/fork a plan component

   This is the primary reason the new fab design supports preprocessing
   of plans in the first place, in order to prevent the kind of
   inefficient and ugly duplication of product specifications (e.g.,
   building blocks in "old" fab terminology) just to support minor
   adjustments.

4) build development/production variants of a product without having to
   modify the product (e.g., remove debug/development packages from the
   plan).

   This allows the simultaneous development of both the "development"
   version of a product and the "production" version.

Overview
--------

Previously the only way to affect product build was to use the "Makefile
inheritance" to add pre/post hooks to targets, or even override the
values of the target "body" and built-in variables.

Configuring a product this way is possible but relatively complex and
inconvenient.

I have developed a couple of new powerful mechanisms to support more
efficient product configuration:

1) conf.d/    chroot scripts
2) product    configuration variables
3) patches    apply local or upstream patches

conf.d/ chroot scripts
----------------------

Any executable script in conf.d (default location, this can be changed)
is copied into a temporary directory in root.patched (after the overlay,
but before the removelist is applied) and executed while chrooted into
root.patched. After execution the temporary directory is deleted.

Any type of script for which there is an interpreter in root.patched is
supported (e.g., shell, perl, python). Static binaries are also
supported but dynamic binaries are dangerous as differences in the
library versions in the chroot may prevent the binary from running
correctly, or more likely running at all.

The order of execution of scripts in conf.d depends on the script
filename, so if have to control the order, you can append an integer
(e.g., conf.d/10myscript).

The script is executed with arguments extracted from conf.d/args/<name>.
By default, no arguments are passed. This supports re-usability of
complex configuration scripts, but for simple configuration scripts it
shouldn't be needed at all. Note that <name> in conf.d/args doesn't
include priority prefixes, so you can change priority without having to
rename conf.d/args/<name>.

Speaking of reusing complex scripts, just like rc*.d scripts, conf.d/
scripts *can* be symbolic links to shared scripts (e.g.,
/path/to/common-conf.d/<name>). Whether they *should* be is an entirely
different question and the answer is usually no. Git supports symbolic
links outside of a repository but a hardwired path will still be
embedded in the product's repository, and you know how I feel about
hardwired paths.

Pros of sharing configuration scripts:

* could be used to prevent duplication of logic in complex scripts
  (I.e., write once fix many times syndrome)

Cons:

* reduces readability: settings need to be separated to args/ or set in
  the environment, so its harder to glance at a script and see the whole
  picture.

* adds significant overhead: parsing of arguments, sanity checking,
  error messages, etc.

I think its usually preferable to put complex logic into a package and
make the configuration script as simple as possible by calling the
complex functionality it needs. If a configuration script has good
enough primitives to leverage it can be made simple enough to resemble a
configuration file itself. See skeleton conf.d scripts for an example.

In other words, I think its preferable to avoid sharing configuration
scripts altogether, though I have supported and tested this capability
in case we need it.

In case a single conf.d directory isn't enough, its possible to add
additional directories by calling the run-conf-scripts macro in a
pre/post hook, like this::

	define root.patched/post
		$(call run-conf-scripts, conf2.d)
	endef

If instead of a directory of scripts you want to execute just a single
script in a pre/post hook, thats also possible. Just call fab-chroot
directly::

	fab-chroot $O/root.patched --script path/to/script [args]

product configuration variables
-------------------------------

By setting the following in your product Makefile::

	VAR1 = VALUE1
	VAR2 = VALUE2
	...
	CONF_VARS = VAR1 VAR2 [ ... ]

You are describing a list of configuration variables that will effect:

1) preprocessor definitions in fab-plan-resolve
2) the environment of fab-chroot commands and scripts: the variables
   listed in CONF_VARS are exported into their environment.

Note: RELEASE is a mandatory built-in configuration variable. Its added by
product.mk automatically, even if you don't define CONF_VARS at all. This is to
ensure that common-plan components can depend on its existence to effect plan
adjustments required for different releases (e.g., discover1 -> discover2).

For example::

	$ cd skeleton
	$ cat Makefile
	RELEASE = rocky

	CONF_VARS = DEBUG
	DEBUG ?= y # empty string is false

	ifndef FAB_MAKEFILE_INCLUDE_PATH
	$(error FAB_MAKEFILE_INCLUDE_PATH not defined)
	else
	include $(FAB_MAKEFILE_INCLUDE_PATH)/product.mk
	endif

	$ cat plan/main
	#ifdef DEBUG
	#include <debug>
	#endif

	#include <boot>
	#include <console>
	#include <net>

Note that one things configuration variables *don't* effect are
overlays, at least not by default. It is possible however to add this
functionality by defining pre/post hooks which are effected by the value
of the configuration variables.

local or upstream patches
-------------------------

Traditionally, TurnKey GNU/Linux has relied on upstream maintainers to include patches and updates in the packages they provide, and this is the preferred method. Sometimes, however, it is necessary to apply extra patches to packages. For example, a problem has been encountered that has been patched upstream, but the patch has not been included in Debian backports. In other cases, it may be useful to create a local patch to modify an appliance.

fab-apply-patch
---------------

The command, fab-apply-patch, uses the *patch* command to apply a given patchfile to a chroot, typically build/root.patched.

::

    # fab-apply-patch --help
    Syntax: fab-apply-patch <patch> <path>
    Apply patch on top of given path
    
    Arguments:
      <patch>           Path to patch
      <path>            Path to apply patch ontop of (ie. build/root.patched)
    
    Patches:            Patches should be in unified context produced by diff -u
                        Filenames must be in absolute path format from the root
                        Patches may be uncompressed, compressed with gzip (.gz),
                        or bzip2 (.bz2)

fab-apply-patch has been designed to fail gracefully. If a patch cannot be applied, a warning is issued and the build is allowed to continue.

Patches should be in unified context produced by diff -u. Filenames must be in absolute path format from the root. Patches may be uncompressed, compressed with gzip (.gz), or bzip2 (.bz2).

Files that are patched are automatically backed up and marked <filename>.orig

If a patch fails to apply, a warning is issued. Patches that are no longer needed should be removed.

product patches - product/*appliance*/patches.d/
------------------------------------------------

Patches placed in the patches.d directory will be applied to root.patched after the conf.d scripts are run and before the product-local removelist is applied. Patches are ordered by filename, so if you have to control the order, you can prepend an integer (e.g., patches.d/10myfix.patch)

common patches - common/patches/turnkey.d/
------------------------------------------

Patches placed in the common/patches/turnkey.d directory will be applied to all appliances during the build after the common configuration scripts are run and before the common removelists are applied. This is recommended for patches to common code shared by all appliances. Note that patches placed in product/core/patches.d/ will only be applied to the core appliance build.

Patches that affect a group of appliances that share common code may be placed in common/patches/. For example, a patch that affects all the lamp appliances could be placed in common/patches/mylampfix.patch. In this case, the lamp.mk file must be modified to include the patchfile by name.

lamp.mk::

    WEBMIN_FW_TCP_INCOMING = 22 80 443 12320 12321 12322

    COMMON_OVERLAYS += apache adminer confconsole-lamp
    COMMON_CONF += phpsh apache-vhost adminer-apache adminer-mysql
    COMMON_PATCHES += mylampfix.patch

    include $(FAB_PATH)/common/mk/turnkey/php.mk
    include $(FAB_PATH)/common/mk/turnkey/mysql.mk