XMake RECURSION

Contents
Recursive Make Considered Harmful
XMake Handles External Targets
Recursive Make Considered Harmful


Projects are often organized into subdirectories of source files,
corresponding to functional 'modules'. The output files in each module are
prerequisites of top level targets which build the final product, according
to configurable options.

What is the best way to build such a project? With one big makefile that
must handle every detail, or with several smaller makefiles, each
concerned only with local sources?

Since modules are logically distinct, it makes sense to have separate
makefiles for each module, enabling a division of labor in maintenance. In
such a design, the top level makefile must build each subdirectory/module
in a sub-process called 'sub-make' or 'recursive make'. Calling $(MAKE)
serially, in a for loop, is generally bad: 1. You can't take advantage of
concurrent jobs; 2. make will exit after the first error, which is usually
not what you want.

The following Makefile builds a project, recursively, with the command
'make project':
# Makefile
SUBDIRS:= dir1 dir2 dir3
.PHONY: $(SUBDIRS) project
project: $(SUBDIRS)
    @echo 'project finished'
$(SUBDIRS):
    @-$(MAKE) -C $@
# EOF

If there were never any chance of dependencies between targets in separate
make processes, recursive make could be a good choice. However, since we
assume each module is a prerequisite of at least the top level targets, we
have a situation in which changes in one module may invalidate the
integrity of the overall build, unless changes in module dependencies are
COMMUNICATED to their dependent targets, in separate contexts. A method
must be found to insure that the build order of targets is always correct
or the output files may be defective. This imposes a coordinated
maintenance requirement.

See: "Recursive Make Considered Harmful" paper by Peter Miller:
    http://owlsoft.ne.client2.attbi.com/build/rmch/recu-make-cons-harm.html
    ( Original PDF: http://aegis.sourceforge.net/auug97.pdf )

Miller discusses how make creates a "Directed Acyclic Graph" (DAG),
describing target file dependency relationships, and how this DAG may
become inaccurate in a recursive make context.

To summarize, the problem is this:

1. A project is initially designed with two modules, A and B, maintained
respectively by two developers "Mr. A" and "Mr. B"
    Everything works fine with these makefiles:
        # Makefile A
        A:B
            touch A
        # EOF
        # Makefile B
        B:
            touch B
        # EOF

2. Mr. C comes along and adds a new prerequisite, C, to Makefile B, but
doesn't communicate this to the maintainer of Makefile A, since he knows
nothing of makefiles outside his scope. If someone edits file C, target B
will be out of date in the context of Makefile B, but not in the context
of Makefile A.
        # Makefile B
        B:C
            touch B
        C:
            touch C
        # EOF

4. Mr. A tries to build target A. Makefile A knows nothing of the
dependency of B on C. If B has not been rebuilt since C was edited, A may
be up to date with respect to B and will not be updated with the changes
to file C. The build is now broken and nobody knows anything is wrong.

Advantages of recursive make:
    + concurrent jobs
    + make won't exit if errors are encountered in one branch
    + division of labor in Makefile maintenance (?)

Disadvantages of recursive make:
    + can result in an incomplete Directed Acyclic Graph (DAG) of
    dependencies
        - prerequisites may be missing or the directory build order might
        be wrong
        - very problematic when you have prerequisites in a different
        project since there is little coordination of maintenance between
        makefiles in separate projects
        - unless a single makefile is used, only targets that will NEVER
        have prerequisite files outside the current makefile can use recursion
        without risk. There is no way to ensure that such a problem doesn't
        develop at any future time.
    + circular dependencies may not be detected
    + slow and inefficient due to process forking and higher memory
    requirements
    + higher maintenance costs in communication of changes between
    makefiles

XMake Handles External Targets


IN BRIEF:
If you use the default setting XMAKE_RECURSION=0
    + any external targets will be assumed to be up to date; the burden is
    all on you to manage your makefiles accordingly.
    + if the files exist and are NOT up to date, you will have incorrect
    output (incomplete DAG)
    + if the external targets don't exist, xmake will exit with an error
If you use any prerequisite targets in other XMake projects, then:
    + set XMAKE_RECURSION=1
    + Do not create any local makefile rules for them
    + Use only XMake macros to generate rules (this automatically adds
    external prerequisites to XM_externalTargets); or, if you write custom
    rules, make sure the external targets are added to XM_externalTargets
    + If all of the above criteria have been met, then you can be confident
    that all external XMake targets will be made up to date prior to building
    any local dependent targets.

As appealing as the single makefile approach is, we cannot ignore the fact
that every project has limited scope and that the ability to share targets
among projects is a desirable feature. Hence, even if we employ a single
makefile design we must still deal with the questions:
    + How to reliably detect external prerequisites?
    + How to reliably build all targets, each in the correct context and in
    the correct order, to avoid an incomplete DAG?
    + How to avoid mis-application of local pattern rules to external
    targets?

XMake works as follows:
    + a single makefile is used on the project level, 'recursively
    including' any makefiles in subdirectories. (Not to be confused with
    recursive sub-make.)
    + external targets are detected in one of several ways:
        - they are declared, by being added to XM_externalTargets, manually
        or automatically, with XMake macros
        - undeclared external targets in subdirectories are caught by the
        rule 'XM_projectSubDirRule', prior to reading any pattern rules (
        ../config/XMakefile.mkr ); XMake exists with error message
        - undeclared external targets not in subdirectories are caught by
        the final pattern rule '/%' ( ../config/XMakefile ); XMake exists
        with error message if the file does not exit
    + sub-make recursion is used for building targets in external XMake
    projects, which are auto-detected
    + all other external targets, outside any XMake project scope, are
    assumed to be ordinary files: their time stamp determines if local
    dependent targets must be remade

There is a performance cost to allowing external prerequisites since XMake
always remakes each target in a separate sub-make process, prior to
attempting to make dependent targets (which know nothing of external
project's DAG). If the number of external targets is small, as is usually
the case, this should not be a significant impact on performance; if the
number of external targets is large, merging of projects may be logical.

(One might consider building all targets in each external project at once,
in one sub-process, rather than using a separate sub-make for each of
several targets in the same external project. The problem with this is
that it assumes you know how to build 'all' targets in an external
project, thus ignoring the essence of the problem we are trying to solve:
knowledge of external project makefiles is inherently unreliable. XMake
assumes nothing about external projects other than the existence of rules
for building each target, by name.)

Circular dependencies between projects are automatically detected by XMake:
xmake will exit with an error message if any target in a 'shared' project
has prerequisites in projects containing prerequisites in the 'shared'
project. If this were not prohibited, an infinite loop would result, which
is what occurs if you try to do this from a Gnu Make makefile.

Since XMake uses a single-makefile approach, there is usually no need to
use recursion within a project, although it does no harm to use '$(XMAKE)
$(someLocalTarget)' from a makefile rule (such as we do in
$XMAKE_HOME/test/XMakeTests.mks, to control error reporting).

Recursion is controlled by setting the environmental variable:
XMAKE_RECURSION

This can be done in several ways:
    + From the environment, via XMake.conf
        - disabled by default: XMAKE_RECURSION=0
    + From the environment, via the xmake command line with options:
        - OVERRIDES XMake.conf settings
            'xmake -X' is equivalent to:
            'export XMAKE_RECURSION=1; xmake'
            'xmake -x' is equivalent to:
            'export XMAKE_RECURSION=0; xmake'
    + From within a makefile, such as XMakefile.xms
        - OVERRIDES environmental settings, unless 'xmake -e' option is
        used
    + From command line makefile variable assignment:
        - OVERRIDES all other settings; '-e' option has no effect
        'xmake XMAKE_RECURSION=1'

Makefile implementation is via the target 'recurse', which examines the
targets in the list $(XM_externalTargets), to determine which reside in
XMake projects and can be made recursively. Recursion is handled
automatically from xmake.sh, which responds to the setting of the
environmental variable, XMAKE_RECURSION, so you should never explicitly
add 'recurse' as a target: recursion must happen PRIOR to attempting to
make targets in the current project.

Any targets in $(XM_externalTargets) that do not reside in an external
XMake project are ignored and an error message will result if the files do
not exist or the targets are not in XM_finalTargets.

Any external targets not in $(XM_externalTargets) will result in an error
message, unless they are found in $(XM_finalTargets). External phony
targets that don't create files of the same name must be added explicitly
to $(XM_finalTargets), unless their paths are automatically matched by the
pattern defined in the XMake.conf variable, XMAKE_FINAL_TARGETS_PATTERN.