1The build tool: Jam
2===================
3
4Requirements for a build tool
5-----------------------------
6
7Building a complete operating system is a somewhat complex task. In the case of Haiku, the build
8involves up to 3 different compilers: for the host system, and in the case of hybrid builds, two
9different target compilers which are different versions of gcc. Multiple host systems are supported,
10with a compatibility layer to be able to use some Haiku specific features like extended filesystem
11attributes even on operating systems that don't support them. Various target architectures also
12require us to generate versions of our bootloader that run on a wide variety of environments
13(BIOS, EFI, OpenFirmware, ...) and uses several executable formats. The work doesn't stop there, as
14the build system also includes support for downloading pre-built packages, generating complete
15filesystem images, and also allows a large amount of customization.
16
17Typical build systems usually fall in two categories: low-level systems like make or ninja focus
18on handling dependencies and running jobs in the correct order, while high-level build systems like
19cmake or meson provide pre-made rules for most common tasks, resulting in very simple builds for
20typical projects.
21
22Neither of these are quite satisfactory for Haiku: the scale of the project would make it difficult
23to handle everything with low-level makefiles, while tools like cmake don't offer nearly enough
24control on the compilation process for our needs.
25
26We can summarize these requirements (this isn't a complete list) for an ideal build system:
27- Automatic parsing of C++ source files to determine dependencies to header files, and rebuild
28automatically if those have changed,
29- Ability to build using multiple compilers, and different set of compiler flags for each target,
30- Factorization of common commands in a single place, to build many similar targets using the same
31process,
32- Handling of tasks other than building code,
33- Support for multiple steps to build a single target (for example, compiling an executable and
34then adding resources to it), ideally with correct dependency handling (add the resources only
35if the executable was rebuilt, or the resource file changed)
36- Some level of configurability to allow modifying the exact content of the generated filesystem
37image.
38
39This resulted in the choice of a less known, but more appropriate tool in the form of Jam. This
40was originally developped by Perforce, but today they have abandoned it. Just like Make, there are
41various flavors of Jam now available, the most known ones being Boost Build and Freetype Jam.
42Haiku provides its own variant, as we needed to extend Jam in various ways to make it work for our
43needs.
44
45Short overview of how Jam operates
46----------------------------------
47
48The core idea of Jam is to provide generic rules (for example "how to build an application from a
49set of source files") and then apply these rules several times. The Haiku build system defines a
50number of custom rules, allowing to build code both for Haiku and for the host operating system (to
51be run during the compiling process).
52
53For example, to build an application, one may use:
54
55.. code-block:: jam
56
57    Application MyExampleApp                      # Name of the application
58        : MyExampleSource.cpp AnotherSource.cpp   # Source code of the app
59        : be translation                          # Libraries needed
60        : MyExampleApp.rdef                       # Resource file
61        ;
62
63This will invoke the "Application" rule which looks something like this (this is a simplified
64version of it):
65
66.. code-block:: jam
67
68    rule Application
69    {
70    	AddResources $(1) : $(4) ;
71    	Main $(1) : $(2) ;
72    	LinkAgainst $(1) : $(3) ;
73    	LINKFLAGS on $(1) = [ on $(1) return $(LINKFLAGS) ]
74    		-Xlinker -soname=_APP_ ;
75    }
76
77This in turns invokes more rules: AddResources, Main and LinkAgainst, and also tweaks LINKFLAGS
78variable for the target. You can see how Jam allows to set variables for a target, which allows
79us to decide which compiler and flags to use in each case.
80
81The low level rules will take care of calling the Depends rule to let Jam know how dependencies
82are organized and what needs to be built first.
83
84Rules can call each other until eventually we get to a rule that includes an "action" part.
85This is where we finally get to actually running commands to do something. For example here is
86(again, a simplified version for this example) the Cc rule that runs a C compiler:
87
88.. code-block:: jam
89
90    rule Cc
91    {
92    	# Make sure the target depends on the sources
93    	Depends $(<) : $(>) ;
94    
95    	# Set variables on the target that will be used by the action
96    	on $(1) {
97    		local flags ;
98    
99    		# Enable optimizations when not building in debug
100    		if $(DEBUG) = 0 {
101    			flags += -O3 ;
102    		} else {
103    			flags += -O0 ;
104    		}
105    
106    		# Select a compiler (the actual Cc rule will pick a different one depening on what is being
107    		# built)
108    		CC on $(1) = gcc ;
109    
110    		# Prepare the command line for building the target
111    		CCFLAGS on $(<) = $(flags) ;
112    		CCHDRS on $(<) = [ FIncludes $(HDRS) ] ;
113    		CCDEFS on $(<) = [ FDefines $(DEFINES) ] ;
114    	}
115    }
116    
117    actions Cc
118    {
119    	# Actually invoke the compiler with the command line flags prepared by the rule
120    	$(CC) $(CCFLAGS) -c "$(2)" $(CCDEFS) $(CCHDRS) -o "$(1)"
121    }
122
123For building Haiku, Jam is combined with a simple "configure" shell script to do the initial setup.
124There is no need for something as complex as autotools, because of the relatively limited number
125and diversirty of supported host operating systems (we can assume a reasonably modern UNIX style
126system) and the fact that a large part of the build is made with our own compiler and libraries,
127which we have full control on.
128
129Jam rules used for building Haiku
130---------------------------------
131
132The rules are defined in the build/jam directory. They are spread accross multiple files, each
133handling a specific part of the build.
134
135Here is an overview of some of these files and what they are used for.
136
137OverriddenJamRules
138    Jam provides a default set of rules for building simple software. Unfortunately, Haiku isn't
139    so simple, and a lot of these rules need to be redefined, in particular to handle our setup
140    with multiple compilers.
141
142BeOSRules
143    Rules specific to BeOS-like operating systems, mainly management of extended attributes and
144    executable resources.
145
146MainBuildRules
147    Rules for building Haiku applications. This file defines rules like Application, Addon,
148    StaticLibrary. It also contains rules for building on the host: BuildPlatformSharedLibrary,
149    BuildPlatformMain, etc.
150
151ArchitectureRules
152    Management of compiler flags for different CPU architectures and "hybrid builds" where two
153    compilers are used.
154
155BootRules
156    Rules related to building the bootloaders.
157
158KernelRules
159    Rules for building the kernel and kernel add-ons.
160
161