XMake version: 0.93a
Release Date: 08/17/05
Copyright (c) 2003 Gregory Keranen. All rights reserved.


Recursive Make Considered Harmful XMake Handles External Targets

Recursive Make Considered Harmful

Projects are often organized into subdirectories of source files, corresponding to functional 'modules'. The output files in each module are prerequisites of top level targets which build the final product, according to configurable options. What is the best way to build such a project? With one big makefile that must handle every detail, or with several smaller makefiles, each concerned only with local sources? Since modules are logically distinct, it makes sense to have separate makefiles for each module, enabling a division of labor in maintenance. In such a design, the top level makefile must build each subdirectory/module in a sub-process called 'sub-make' or 'recursive make'. Calling $(MAKE) serially, in a for loop, is generally bad: 1. You can't take advantage of concurrent jobs; 2. make will exit after the first error, which is usually not what you want. The following Makefile builds a project, recursively, with the command 'make project': # Makefile SUBDIRS:= dir1 dir2 dir3 .PHONY: $(SUBDIRS) project project: $(SUBDIRS) @echo 'project finished' $(SUBDIRS): @-$(MAKE) -C $@ # EOF If there were never any chance of dependencies between targets in separate make processes, recursive make could be a good choice. However, since we assume each module is a prerequisite of at least the top level targets, we have a situation in which changes in one module may invalidate the integrity of the overall build, unless changes in module dependencies are COMMUNICATED to their dependent targets, in separate contexts. A method must be found to insure that the build order of targets is always correct or the output files may be defective. This imposes a coordinated maintenance requirement. See: "Recursive Make Considered Harmful" paper by Peter Miller: http://owlsoft.ne.client2.attbi.com/build/rmch/recu-make-cons-harm.html ( Original PDF: http://aegis.sourceforge.net/auug97.pdf ) Miller discusses how make creates a "Directed Acyclic Graph" (DAG), describing target file dependency relationships, and how this DAG may become inaccurate in a recursive make context. To summarize, the problem is this: 1. A project is initially designed with two modules, A and B, maintained respectively by two developers "Mr. A" and "Mr. B" Everything works fine with these makefiles: # Makefile A A:B touch A # EOF # Makefile B B: touch B # EOF 2. Mr. C comes along and adds a new prerequisite, C, to Makefile B, but doesn't communicate this to the maintainer of Makefile A, since he knows nothing of makefiles outside his scope. If someone edits file C, target B will be out of date in the context of Makefile B, but not in the context of Makefile A. # Makefile B B:C touch B C: touch C # EOF 4. Mr. A tries to build target A. Makefile A knows nothing of the dependency of B on C. If B has not been rebuilt since C was edited, A may be up to date with respect to B and will not be updated with the changes to file C. The build is now broken and nobody knows anything is wrong. Advantages of recursive make: + concurrent jobs + make won't exit if errors are encountered in one branch + division of labor in Makefile maintenance (?) Disadvantages of recursive make: + can result in an incomplete Directed Acyclic Graph (DAG) of dependencies - prerequisites may be missing or the directory build order might be wrong - very problematic when you have prerequisites in a different project since there is little coordination of maintenance between makefiles in separate projects - unless a single makefile is used, only targets that will NEVER have prerequisite files outside the current makefile can use recursion without risk. There is no way to ensure that such a problem doesn't develop at any future time. + circular dependencies may not be detected + slow and inefficient due to process forking and higher memory requirements + higher maintenance costs in communication of changes between makefiles

XMake Handles External Targets

IN BRIEF: If you use the default setting XMAKE_RECURSION=0 + any external targets will be assumed to be up to date; the burden is all on you to manage your makefiles accordingly. + if the files exist and are NOT up to date, you will have incorrect output (incomplete DAG) + if the external targets don't exist, xmake will exit with an error If you use any prerequisite targets in other XMake projects, then: + set XMAKE_RECURSION=1 + Do not create any local makefile rules for them + Use only XMake macros to generate rules (this automatically adds external prerequisites to XM_externalTargets); or, if you write custom rules, make sure the external targets are added to XM_externalTargets + If all of the above criteria have been met, then you can be confident that all external XMake targets will be made up to date prior to building any local dependent targets. As appealing as the single makefile approach is, we cannot ignore the fact that every project has limited scope and that the ability to share targets among projects is a desirable feature. Hence, even if we employ a single makefile design we must still deal with the questions: + How to reliably detect external prerequisites? + How to reliably build all targets, each in the correct context and in the correct order, to avoid an incomplete DAG? + How to avoid mis-application of local pattern rules to external targets? XMake works as follows: + a single makefile is used on the project level, 'recursively including' any makefiles in subdirectories. (Not to be confused with recursive sub-make.) + external targets are detected in one of several ways: - they are declared, by being added to XM_externalTargets, manually or automatically, with XMake macros - undeclared external targets in subdirectories are caught by the rule 'XM_projectSubDirRule', prior to reading any pattern rules ( ../config/XMakefile.mkr ); XMake exists with error message - undeclared external targets not in subdirectories are caught by the final pattern rule '/%' ( ../config/XMakefile ); XMake exists with error message if the file does not exit + sub-make recursion is used for building targets in external XMake projects, which are auto-detected + all other external targets, outside any XMake project scope, are assumed to be ordinary files: their time stamp determines if local dependent targets must be remade There is a performance cost to allowing external prerequisites since XMake always remakes each target in a separate sub-make process, prior to attempting to make dependent targets (which know nothing of external project's DAG). If the number of external targets is small, as is usually the case, this should not be a significant impact on performance; if the number of external targets is large, merging of projects may be logical. (One might consider building all targets in each external project at once, in one sub-process, rather than using a separate sub-make for each of several targets in the same external project. The problem with this is that it assumes you know how to build 'all' targets in an external project, thus ignoring the essence of the problem we are trying to solve: knowledge of external project makefiles is inherently unreliable. XMake assumes nothing about external projects other than the existence of rules for building each target, by name.) Circular dependencies between projects are automatically detected by XMake: xmake will exit with an error message if any target in a 'shared' project has prerequisites in projects containing prerequisites in the 'shared' project. If this were not prohibited, an infinite loop would result, which is what occurs if you try to do this from a Gnu Make makefile. Since XMake uses a single-makefile approach, there is usually no need to use recursion within a project, although it does no harm to use '$(XMAKE) $(someLocalTarget)' from a makefile rule (such as we do in $XMAKE_HOME/test/XMakeTests.mks, to control error reporting). Recursion is controlled by setting the environmental variable: XMAKE_RECURSION This can be done in several ways: + From the environment, via XMake.conf - disabled by default: XMAKE_RECURSION=0 + From the environment, via the xmake command line with options: - OVERRIDES XMake.conf settings 'xmake -X' is equivalent to: 'export XMAKE_RECURSION=1; xmake' 'xmake -x' is equivalent to: 'export XMAKE_RECURSION=0; xmake' + From within a makefile, such as XMakefile.xms - OVERRIDES environmental settings, unless 'xmake -e' option is used + From command line makefile variable assignment: - OVERRIDES all other settings; '-e' option has no effect 'xmake XMAKE_RECURSION=1' Makefile implementation is via the target 'recurse', which examines the targets in the list $(XM_externalTargets), to determine which reside in XMake projects and can be made recursively. Recursion is handled automatically from xmake.sh, which responds to the setting of the environmental variable, XMAKE_RECURSION, so you should never explicitly add 'recurse' as a target: recursion must happen PRIOR to attempting to make targets in the current project. Any targets in $(XM_externalTargets) that do not reside in an external XMake project are ignored and an error message will result if the files do not exist or the targets are not in XM_finalTargets. Any external targets not in $(XM_externalTargets) will result in an error message, unless they are found in $(XM_finalTargets). External phony targets that don't create files of the same name must be added explicitly to $(XM_finalTargets), unless their paths are automatically matched by the pattern defined in the XMake.conf variable, XMAKE_FINAL_TARGETS_PATTERN.