Define a build protocol for single package build

[hegner]

The build of individual packages should follow a common 'protocol'. This issue is to discuss how it could look like.

[ktf]

I would personally be happy with something which allows me to build externals with:

EXTERNAL_ROOT1=/some/path
EXTERNAL_VERSION=vX.Y.Z
OPTION1=someoption
PREFIX=/unique/build/prefix
...
./build.sh

and install them with:

./install.sh

if it needs to become:

cmake -DEXTERNAL_ROOT1=/some/path \
            -DEXTERNAL_VERSION=vX.Y.Z \
            -DOPTION1=someoption \
            -DPREFIX=/unique/build/prefix
            ....
make
make install

for the sake of collaboration, I'd be happy as well. The real questions I can think of on my side are:

• Why do you think it's worth to add a (binary) dependency on cmake for this? What does using CMake add as a value compared to a shell script? How do we build / deploy the initial CMake?

• What externals out of the ones in cms-sw/cmsdist do you think I can build that way right now (at the moment not even root or geant4 are so easy for us, can you comment if we are doing anything wrong?)?

• Apart from:

  • EXTERNALX_ROOT: the installation path of one the requirements.
  • EXTERNALX_VERSION: the version of one of the requirements.
  • PREFIX: where built software is installed for packaging.
  • JOBS: the number of parallel jobs make does.

  which I think we all agree on, what other options do we want to be part of the interface?

[hegner]

I like that idea. There are however a few more things we need to consider - platform, compiler, switches like verbosity, build type, a pointer to the source, ...
And one of the more general questions - do we work with options or with environment variables?

What concerns cmake - The ExternalPackage_Add has a nice 'protocol' one can get inspiration from. Whether the scripts use cmake internally or not is at first order an implementation detail to me. First let's get the interface agreed upon.

[sbinet]

do we need a CMMI interface? (configure-make-make-install)
do we need just that?

with CVMFS and other (non-root) overlay-fs technologies, I think we could rather define a canonical installation path (/opt/sw/) which would tremendously simplify build+installation+use procedures.

corollary, a proper build protocol, taking into account the full set of options for build reproducibilities should probably be along the lines of GUIX (which makes a hash from that set. see: http://en.wikipedia.org/wiki/Nix_package_manager )

[peremato]

Typically things are nor as simple as CMMI. You need as Benedikt pointed out a number of parameters to decide the how to do the CMMI and more importantly the dependencies to other packages. I agree with Sebastien that an standard installation location helps to handle the dependencies but not the build instructions themselves. You need to to locate the packages that you depend on considering the full set of dependencies.
I agree that taking into account all dependencies (and their versions) and possible options is a must and a hash number should be generated similarly to what we currently do with the LCGCmake, which is very similar to what Nix system does.
For example something as simple as building swig requires some work:

LCGPackage_Add(
  swig
  URL http://service-spi.web.cern.ch/service-spi/external/tarFiles/swig-<VERSION>.tar.gz
  IF <VERSION> VERSION_LESS 2.0.0 THEN
    CONFIGURE_COMMAND <SOURCE_DIR>/configure --prefix=<INSTALL_DIR>
  ELSE
    CONFIGURE_COMMAND <SOURCE_DIR>/configure --prefix=<INSTALL_DIR> --with-pcre-prefix=${pcre_home} PCRE_LIBS=${pcre_home}/lib/libpcre.a
    BUILD_COMMAND ${MAKE} CPPFLAGS=-I${pcre_home}/include
    INSTALL_COMMAND ${MAKE} install
    DEPENDS pcre
  ENDIF
 )

For some recent versions it depends on pcre, so some convention on where to find the installation of the adequate version of pcre will be required.

Commenting Giulio proposal, I think what we need is a 'declarative' protocol instead of a scripting or execution protocol. We need to know all the details on where the sources are, what is the command to build, to install, to patch, etc. and is the system that will execute them in the adequate order and injecting additional commands at adequate points (creation os RPMs, applying patches, making the installation relocatable, creating and collecting log files, etc.). So, a solution based on a set of build.sh is not really sufficient. This is why I think that defining we need to define the 'data' for each package. In a way we could write our CMake LCGPackage_Add() as:

NAME = swig
SOURCE_URL =  http://service-spi.web.cern.ch/service-spi/external/tarFiles/swig-<VERSION>.tar.gz
CONFIGURE_COMMAND =  IF <VERSION> VERSION_LESS 2.0.0 THEN <SOURCE_DIR>/configure --prefix=<INSTALL_DIR> ELSE <SOURCE_DIR>/configure --prefix=<INSTALL_DIR> --with-pcre-prefix=${pcre_home} PCRE_LIBS=${pcre_home}/lib/libpcre.a
BUILD_COMMAND = make CPPFLAGS=-I${pcre_home}/include
INSTALL_COMMAND = make install
DEPENDS = pcre

[ktf]

@hegner

I agree about platform (HEP_PLATFORM?), compiler (HEP_COMPILER?), verbosity (HEP_VERBOSITY?). I'm not sure what you mean by "build type". Is this debug?

I strongly disagree about including anything related to sources in the "build protocol" that should be part of a different "protocol" (configuration?), IMHO. Different tools do fetching in different ways, and if you start mixing build and fetching you end up reinventing the wheel. How would you plug fetching in (say) homebrew? They have their own caching and fetching mechanism (just like nix, cmsBuild, or any other build system out there). IMHO the "build protocol" should simply assume sources are unpacked and you are inside the source directory.

@peremato Apart from the comment above about sources, I think the complication you talk about comes from the fact you want to have different behaviour for different versions rather than extended behaviour based on the context.

Your example in my mind should really become:

# In ./build.sh
./configure ${INSTALL_DIR+--prefix $INSTALL_DIR} \
            ${PCRE_ROOT+--with-pcre-prefix=$PCRE_ROOT PCRE_LIBS=$PCRE_ROOT/lib/libpcre.a } 

make CPPFLAGS="${PCRE_ROOT+-I$PCRE_ROOT/include}"
make install

this script could be used by any build system (including for example homebrew or nix) without modifications, just by changing the context they build in. This would be pluggable in a bunch of different build systems, to the limit it could simply work by doing:

INSTALL_DIR=/usr/local
PCRE_ROOT=/usr
./build.sh

The above is really a declarative recipe, to the point it could be written as:

{
  "INSTALL_DIR": "/usr/local",
  "PCRE_ROOT": "/usr",
  "BUILD_SCRIPT": "./build.sh",
}

which in the end is very similar to what Nix does. Your example is putting an if statement inside a variable, which is as "non-declarative" as it can get, IMHO. To be clear, I think my idea is: parts which are really declarative should go in the "configuration" level, parts which are non-declarative go in the "build recipe" level.

Apart from that I've also the following comments:

• Making the installation relocatable in my mind should be part of either the patching phase (which then can be fed back upstream) or it should be handled automatically by the build tool (by installing in a unique path so that the unique hash being used can be used at post installation to relocate correctly in an automatic manner).
• Collecting log files should be done at a different level, IMHO. What the "build protocol" could provide is a callback which gets invoked with generated log files. Something like HEP_LOG_CALLBACK which gets invoked by ./build.sh pointing to the actual build log for processing.
• As I said, there should not be explicit dependencies but the build script should extend behaviour based on the dependencies being provided.

[ktf]

@sbinet I think relocability should be mandatory. We (CMS) have as a requirement to be able to build and run from any path, as non root.

[hegner]

@ktf build type indeed meant debug or optimised. What concerns location of the source. While Pere and others will disagree with that approach, I actually meant a place in the local file system. How it ends up at that place is indeed up to a different tool or step.

[ktf]

@hegner place in the filesystem would be ok for me as well.

[brettviren]

I think we should understand a procedure for directing this work before jumping in to solutions.

One persons "packaging" is another persons muck and mire. We should define what problems we are trying to solve more clearly before talking about solutions. We should figure out a metric to measure proposed solutions before proposing solutions. We should collect existing solutions and measure them against this metric before contemplating writing something new.

GitHub issue threads are kind of lousy for this type of discussion. Would it not be better to have it on a mailing list?

[ktf]

I disagree. That way we will end up with something nobody wants, apart from maybe the one who managed to impose his solution. It has happened in the past, already.

Experiments already have their tools and for each one of them their tool scores 100% in the only metric which matters: "works for me and requires no extra manpower", otherwise we would not have this discussion.

IMHO, for this to succeed, the various tools should say what they are willing to "give up" for the sake of commonality and common effort. As I told @hegner at last meeting, while I've strong opinion on how sources are referred and distributed, I do not have those for everything that happens from "sources expanded in a single directory" to "files installed with make install". Given that's the 95% of the maintenance effort we are currently putting in this kind of things in CMS, I think it would be extremely useful to have an agreed way of doing this, rather than trying to focus on agreeing on things we know we disagree. I would personally move to it the day after if I had something which I invoke with:

build.sh -DPATH_TO_MY_SOURCES=/path -DPREFIX=/install

and lets me worry about everything else.

[brettviren]

Well, your build.sh example is just a high level automation interface and doesn't really impart any requirements.

What I'm trying to do is exactly follow a process where imposition of singular will is not emphasized. I don't see how this means we are trying to agree on things we disagree. If we can't come up with a common understanding of the scope of "packaging" then there's no point in collaborating.

So, I'd like to define what we are trying to accomplish instead of have everyone make they're own assumptions about what problem we solve based on whatever things they are facing. Maybe ones pet problem can be identified as a subset or aspect of the entire problem and then people can warm up to contributing toward that part.

I see this problem factoring into these major areas:

1. Build configuration and release management. Ways to define a suite of software, specify the configuration for building including things like package versions, build variants, target platforms.

2. Packaging. Forms of bundling the results of the build in a way that they may be transfered to user machine.

3. Installation and user setup. How to apply the packages and make them ready to use. This is tied to (2), of course.

And these are likely incomplete as they are based on how I view the problem. So, I don't mean to say this is the best separation, but if we can't write down and agree on some kind of breakdown and clarification of the goal like this then I don't know how we can work together towards a common solution.

Some general desired features:

• automation (hands-off build / installation, complex actions hidden behind simple interfaces)
• modularity between layers (don't tie user run-time setup into build-time environment)
• flexibility to assert domain-specific policy (eg, how files are layed out, maybe I want to use a different binary package than you, maybe you want a different run-time setup system than me)
• concise configuration (I want to tell my users: "take this single file, it's all you need to build everything to reproduce Run X Analysis")
• efficiency of reuse (I don't want to rebuild Package-X for Release-B if it was already built and available from Release-A, my users normally do not want to build if they can install and they don't want to install if they can just setup)
• don't assume all users are dumb (users sometimes want and must rebuild the entire stack in a way that they can tweak the build variant, hack on ROOT, swap out versions of Python, add print statements to Geant4)

I hope none of these requirements are controversial (they are certainly incomplete) and lead to us trying to "agree on things we disagree". But, without stating them, how can we expect to satisfy them?

A lot of work has gone on in this arena already both in our community and outside. If we do not evaluate that work in in light of some kind of criteria then we are setting ourselves up to reinvent the wheel, yet again (and probably poorly).

The things on my personal radar include and I'd like to know what other ones I might be missing:

• homebrew looks interesting
• conda looks interesting
• guix/nix looks interesting
• CMT is still interesting
• Sebastian's hwaf is very interesting
• Of course, my own worch is personally interesting

I know right now that none of these, as-is, are 100% suitable to satisfy the requirements in my head and so are less likely to satisfy all requirements we can come up with. But some of them are partly suitable, some of them can be changed to be made more suitable and some of them have incredibly good concepts that can, at the very least, be taken and incorporated in new development we may perform.

So, again, I think we need to produce a document which defines roles, use cases, states what types of functionality we require, critically reviews the existing systems, ranks them against our requirements and then tries to determine what the best approach is going forward, be it adoption, adoption+improvement, whole-cloth reinvention or some aggregation of these. I'm willing to start such a document (kept as latex in an open HSF github repo). To be successful it will need input from many people in various roles across HEP.

[sextonkennedy]

@brettviren I am also interested in having a clear document with the elements you describe. Reading the above, I see lots of agreement, so it's clear to me that we're making progress. As for format I'd be willing to use Latex but as I'm looking at a button right now that says "Markdown supported", I'd have a preference for that ;-)

[ktf]

Let me see if I can find the ones which were already written with exactly the same goals in the 3 years ago, I mean 6 years ago, I mean 10 years ago... ;-) I remember exactly the same discussion about CMT and SCRAM almost 13 years ago using exactly this approach. I think we should at least try to avoid mistakes of the past.

I personally think we should start from picking up the most widely used tools in HEP for the packaging business, e.g. lcgBuild (@hegner what's the name of your tool?) and cmsBuild and try to iron out unneeded differences. My "build.sh" simplification is a clear point where we are spending a lot of effort, platform support, which could be unified and for which I'm personally more that happy to pickup whatever @hegner proposes as an implementation.

Same for the building part. Given there is general consensus CMake is the best tool for building small / medium projects, why do we still have so many generators, fastjet and many other bits which use configure / make? Shouldn't HSF direct effort to fix that rather than at writing yet another nicely typeset design document for yet "one tool to rule them all"?

I hate finding myself citing the Agile Manifesto, but IMHO there is some truth in "Customer collaboration over contract negotiation".

Clearly this discussion diverged already...;) See you in Japan, I need to finish my slides...;)

[drbenmorgan]

@brettviren 👍 I think this thread already demonstrates several slightly different viewpoints/understandings of "packaging".

@ktf O.k., where are lcgBuild and cmsBuild documented and obtainable from? How would I start using them to provide a software stack for a new experiment?

Regarding use of CMake, I believe HSF should promote the use of (or migration to) standard build tools, but this is orthogonal to the "packaging" problem. My perspective on the latter is that it is a layer above, and decoupled from, the "build tool" layer(s). RPM et al follow this pattern, and so don't care whether an individual package uses CMake, Autotools, or even a hand written Makefile. Decoupling also means that the "build tool" must not depend on the "packaging" layer - if this isn't done, portability, re-usability and adaptability suffer.

[ktf]

@hegner To be clear, my point is about writing a design document not
documentation in general. Obviously "how to deploy" documentation should be
there, or at least an example.

Even in that case, though, that does not solve the 95% of the problem which
is maintaining the software stack. Unless we manage to agree on how to do
that, there will be little gain from a common tool. To make a clear
example: I could port Alice (just to use a random example ;-) ) stack in a
couple of weeks, using cmsBuild and all the tools I've written and I
perfectly know and happy about (so no documentation / design issues there).
However even in that case, where I've everything under control, unless
there was for me some way of leveraging from the builds recipes of someone
else I would still pay 95% of the cost of of that effort, and more
importantly I would still pay the ~ 1FTE equivalent ongoing effort required
to maintain, port and adapt the software stack. That's where the cost goes
and what should be fixed first, IMHO.
Using any tools, regardless of how sophisticated, will have that problem
unless there is a common way of handling build recipes.

On Mon, Apr 13, 2015 at 6:05 PM Ben Morgan notifications@github.com wrote:

@brettviren https://github.com/brettviren [image: 👍] I think this
thread already demonstrates several slightly different
viewpoints/understandings of "packaging".

@ktf https://github.com/ktf O.k., where are lcgBuild and cmsBuild
documented and obtainable from? How would I start using them to provide a
software stack for a new experiment?

Regarding use of CMake, I believe HSF should promote the use of (or
migration to) standard build tools, but this is orthogonal to the
"packaging" problem. My perspective on the latter is that it is a layer
above, and decoupled from, the "build tool" layer(s). RPM et al follow this
pattern, and so don't care whether an individual package uses CMake,
Autotools, or even a hand written Makefile. Decoupling also means that the
"build tool" must not depend on the "packaging" layer - if this isn't done,
portability, re-usability and adaptability suffer.

—
Reply to this email directly or view it on GitHub
#1 (comment).