eyestep . org - the arc build tool project

eyestep the arc build tool project

the arc build tool project

The Arc build tool project is an experiment in developing a new
build tool chain for the development of everyday applications and
utilities. It should combine most of the functions now found in tools like
autoconf, automake, libtool and make.

Why another build system?

I found myself repeatedly spend more time in managing the build
system than writing "real" code. If I interpret frequently repeated
criticism on the net correctly, then I'm no the only one, who experiences
severe limitations of the make tools and its various frontends,
e.g. autoconf/automake. So what is really wrong with
make/autoconf/automake or to put it better: What can be improved? (All the
following is not new, and does not offspring necessarily from my limited
understandings of how make and its companions are working in deepest
detail, but I collected most of what I read by others and experienced
myself during years of work with make and different generations of automake
and autoconf).

	:	automake builds on top of make and Makefiles, which themselves have a number of severe limitations. They are not portable, even between related platforms (as e.g. the most common UN*X platforms). Furthermore makefiles are limited to one level directories. Managing complex, multilevel directory projects therefore required repeated recursive Makefile structures, with Makefile calling itself over and over again, which is not even not only slow, but also tend to become unnecessary complex.
	:	Makefiles work mostly by internal magics. A large number of actions are builtin (e.g. how to compile C files), but even more are not. In Makefiles all the additional logic necessary to work with complex packages and software is missing, e.g. renaming, moving or deleting files, zipping and unzipping, filtering and replacing templates, etc. To achieve this one have to write "bash commands" into the Makefile itself, small pieces of system shell code, which are defacto seldomly portable.
	:	Moreover limits this "internal magic" the extensibility of make and makefiles. Not to say that it is impossible, but extending make for new languages or applications introduces to much logic into the Makefiles, which should normally be hidden from the normal Makefile "user".
	:	Automake is kind of frontend to the make system; it generates Makefiles from a simplified Makefile extract. Principal a good idea and for a lot of projects really usefull, automake lacks the total availability to be extensible. Automake can't be extended (or even modified) without a thorough understand of automake's kernel logic.

Features Arc aims at:

The following is only a short list one ideas and features the
Arc build system should provide:

	:	Portability. Build scripts should work for most applications and platforms without system specific code and without any platform modifications. Conditional dependency tracking, dependending on the local host and os system, should be integrated.
	:	Task based. Very similar to the desprictive nature of the Ant system (ant) the real work should be done by "tasks". Tasks are subportions of functionality, e.g. for compiling C source files, copying files, creating directories, etc. Tasks are pluggable, and the core distribution should come with a set of tasks mostly used in normal applications and distributing packages.
	:	Scriptable. Very different to the Ant system however, Arc is scriptable. Arcfiles can utilize the full power of a functional programing language with a rich set of functions. This helps in developing portable script files, even when (normal) tasks are not enough.
	:	API for Tasks. To write a new task (i.e. to extend Arc), one should only need to understand the API (i.e. the input values, the return values) and how to encapsulate the action in scheme code.
	:	Absolutely no magic. The Arc kernel does no magic, i.e. it does not know how do calculate dependencies for .o-files related to .c-files. It has no builtin variables, etc.

Why Scheme

Arc will be written entirely in Scheme, a small but very powerful
language. Even if some features are missing, which seems to make other
languages a better candidate for such a tool (e.g. perl or bash), the
scheme language is outstanding in its reduced elegance and simplicity and
is very simple to learn. There's is nearly nothing which can't be done in
scheme (without changing the language itself!). But one of the major
features (which it shares with other LISP dialects) is that code can be
treated as data, and data as code. In this way users may mix statements
and targets with real scheme code, and there's no need for a declarative
Makefile syntax and an scripting extension language.

Processing model

The processing model of arc is leaned on the experience with the
Ant make tool. But the philosophy of arc is somewhat different to that of
ant and make.

In Arc one write one or more "Arcfile"s for a project, in which
one gives a number of "statements". These "statements" are somewhat
similar to targets in ant and make, since they are combined by static and
lazy dependencies. Differently to make these "targets" never automatically
embrace dependencies between source and object files, but only between
statements in the Arcfile itself. Sourcefile dependency are handled
seperatedly by the different tasks.

Rationale: Dependencies are handled quite different in
different language environments. Various Java compilers for instance
handle the dependency between source files on their own which sometimes
leads to conflicts between the make system and the language tool.

For example the following Arcfile builds a sample application:

 1 (project sample-project
 2          builddir: "src/"
 3          default: 'link)
 4
 5 (stmt build-dir
 6       "_build")
 7
 8 (stmt init
 9       (mkdir dir: (-> build-dir))
10
11 (stmt c-sources
12       (fileset pattern: "*.c"))
13
14 (stmt compile
15       depends: 'init
16       (c-compile sources: (-> c-sources)
17                  outdir: (-> build-dir) ))
18
19 (stmt link
20       (link static?: #f
21             objfiles: (-> compile)
22             dest: "hello-world"))

This short example shows some major features. (1) In line 16 the
normal compilation is stated (by the arc task c-compile). It
takes a list of c sources, by requesting the value of the statement
c-sources ((-> c-sources)) (a so called "lazy
dependency", since statements which are dependend on the c-sources
statement are only evaluated, when the return value is requested here).
(2) Each statement has a return value, which is set by the tasks in its
body. The return value of the fileset task is a list of all matched
files (line 12). The return value of the c-compile task is a list
of all object files under the control of the c-compiler. (3) The Arc
system ensures, that the c-sources statement is evaluated only once,
its returnvalue is cached and delivered correctly to requesters. The
compile statement shows a "static dependency", too, by declaring
depends: 'init.

To "run" this Arcfile a user would be somewhere in the source
directory, and execute the following command:

$> arc

which would start dependency tracking at the default target stated
in the project header of the Arcfile (here line 3: default:
'link). The user can specify a different default statement:

$> arc compile

There are many other features, e.g. conditional dependencies,
non-cached statements, etc.

Status

Arc is far from being complete. The current code is workable
already, even if a large number of intended tasks are missing. Support for
different platforms is lacking, even if regular development is on FreeBSD
(5.0), GNU/Linux (Debian), BeOS and Cygwin. But the stuff done shows the
path to go. Here's a list of things to be done:

	:	Integrate operating and machine specifics (e.g. which C compiler to use, which options are accepted). This information should be added to a database, and a checking tool would set up a (project) local "Arcconfig" file, which would be used by the arc process. This procedure would replace autoconfig and the m4 base of tests.
	:	Write tasks!
	:	Implement a test only mode (just print the command, do work really)
	:	Better error management
	:	Pre check; check for cyclical dependencies, check for well formed Arcfiles, etc.
	:	Portability. Support to more platforms, especially the critical and complex plath (task-lib, task-link).

Where to get and find arc :
             : arc@savannah.gnu.org
             : The latest packaged version

Feedback

For comments and feedback: gck at eyestep dot org.
Homepage: http://www.eyestep.org/arc.html.