Thursday, October 30, 2014

Deploying iOS applications with the Nix package manager revisited

Previously, I have written a couple of blog posts about iOS application deployment. For example, I have developed a Nix function that can be used to build apps for the iOS simulator and real iOS devices, made some testability improvements, and implemented a dirty trick to make wireless ad-hoc distributions of iOS apps possible with Hydra, the Nix-based continuous integration server.

Recently, I made a some major changes to the Nix build function which I will describe in this blog post.

Supporting multiple Xcode versions


Xcode version 6.0 and beyond do not support iOS SDK versions below 8.0. Sometimes, it might still be desirable to build apps against older SDKs, such as 7.0. To be able to do that, we must also install older Xcode versions alongside newer versions.

As with recent Xcode versions, we must also install older Xcode versions manually first and use a Nix proxy function to use it. DMG files for older Xcode versions can be obtained from Apple's developer portal.

When installing a second Xcode DMG, you typically get a warning that looks as follows:


The installer attempts to put Xcode in its standard location (/Applications/Xcode.app), but if you click on 'Keep Both' then it is installed in a different path, such as /Applications/Xcode 2.app.

I modified the proxy function (described in the first blog post) in such a way that the version number and path to Xcode are configurable:

{ stdenv
, version ? "6.0.1"
, xcodeBaseDir ? "/Applications/Xcode.app"
}:

stdenv.mkDerivation {
  name = "xcode-wrapper-"+version;
  buildCommand = ''
    mkdir -p $out/bin
    cd $out/bin
    ln -s /usr/bin/xcode-select
    ln -s /usr/bin/security
    ln -s /usr/bin/codesign
    ln -s "${xcodeBaseDir}/Contents/Developer/usr/bin/xcodebuild"
    ln -s "${xcodeBaseDir}/Contents/Developer/usr/bin/xcrun"
    ln -s "${xcodeBaseDir}/Contents/Developer/Applications/iOS Simulator.app/\
Contents/MacOS/iOS Simulator"

    cd ..
    ln -s "${xcodeBaseDir}/Contents/Developer/Platforms/\
iPhoneSimulator.platform/Developer/SDKs"

    # Check if we have the xcodebuild version that we want
    if [ -z "$($out/bin/xcodebuild -version | grep -x 'Xcode ${version}')" ]
    then
        echo "We require xcodebuild version: ${version}"
        exit 1
    fi
  '';
}

As can be seen in the expression, two parameters have been added to the function definition. Moreover, only tools that a particular installation of Xcode does not provide are referenced from /usr/bin. The rest of the executables are linked to the specified Xcode installation.

We can configure an alternative Xcode version by modifying the composition expression shown in the first blog post:

rec {
  stdenv = ...;

  xcodeenv = import ./xcode-wrapper.nix {
    version = "5.0.2";
    xcodeBaseDir = "/Applications/Xcode 2.app";
    inherit stdenv;
  };

  helloworld = import ./pkgs/helloworld {
    inherit xcodeenv;
  };
  
  ...
}

As may be observed, we pass a different Xcode version number and path as parameters to the Xcode wrapper which correspond to an alternative Xcode 5.0.2 installation.

The app can be built with Nix as follows:

$ nix-build default.nix -A helloworld
/nix/store/0nlz31xb1q219qrlmimxssqyallvqdyx-HelloWorld
$ cd result
$ ls
HelloWorld.app  HelloWorld.app.dSYM

Simulating iOS apps


Previously, I also developed a Nix function that generates build scripts that automatically spawn iOS simulator instances in which apps are deployed, which is quite useful for testing purposes.

Unfortunately, things have changed considerably in the new Xcode 6 and the old method no longer works.

I created a new kind of script that is based on details described in the following Stack overflow article: http://stackoverflow.com/questions/26031601/xcode-6-launch-simulator-from-command-line.

First, simulator instances must be created through Xcode. This can be done by starting Xcode and opening Window -> Devices in the Xcode menu:


A new simulator instance can be added by clicking on the '+' button on the bottom left in the window:


In the above example, I create a new instance with a name 'iPhone 6' that simulates an iPhone 6 running iOS 8.0.

After creating the instance, it should appear in the device list:


Furthermore, each simulator instance has a unique device identifier (UDID). In this particular example, the UDID is: 0AD5FC1C-A360-4D05-9D6A-FD719C46A149

We can launch the simulator instance we just created from the command-line as follows:

$ open -a "$(readlink "${xcodewrapper}/bin/iOS Simulator")" --args \
    -CurrentDeviceUDID 0AD5FC1C-A360-4D05-9D6A-FD719C46A149

We can provide the UDID of the simulator instance as a parameter to automatically launch it. If we don't know the UDID of a simulator instance, we can obtain a list from the command line by running:

$ xcrun simctl list
== Device Types ==
iPhone 4s (com.apple.CoreSimulator.SimDeviceType.iPhone-4s)
iPhone 5 (com.apple.CoreSimulator.SimDeviceType.iPhone-5)
iPhone 5s (com.apple.CoreSimulator.SimDeviceType.iPhone-5s)
iPhone 6 Plus (com.apple.CoreSimulator.SimDeviceType.iPhone-6-Plus)
iPhone 6 (com.apple.CoreSimulator.SimDeviceType.iPhone-6)
iPad 2 (com.apple.CoreSimulator.SimDeviceType.iPad-2)
iPad Retina (com.apple.CoreSimulator.SimDeviceType.iPad-Retina)
iPad Air (com.apple.CoreSimulator.SimDeviceType.iPad-Air)
Resizable iPhone (com.apple.CoreSimulator.SimDeviceType.Resizable-iPhone)
Resizable iPad (com.apple.CoreSimulator.SimDeviceType.Resizable-iPad)
== Runtimes ==
iOS 7.0 (7.0.3 - 11B507) (com.apple.CoreSimulator.SimRuntime.iOS-7-0)
iOS 7.1 (7.1 - 11D167) (com.apple.CoreSimulator.SimRuntime.iOS-7-1)
iOS 8.0 (8.0 - 12A365) (com.apple.CoreSimulator.SimRuntime.iOS-8-0)
== Devices ==
-- iOS 7.0 --
-- iOS 7.1 --
-- iOS 8.0 --
    iPhone 4s (868D3066-A7A2-4FD1-AF6A-25A90F480A30) (Shutdown)
    iPhone 5 (7C672CBE-5A08-481A-A5EF-2EA834E3FCD4) (Shutdown)
    iPhone 6 (0AD5FC1C-A360-4D05-9D6A-FD719C46A149) (Shutdown)
    Resizable iPhone (E95FC563-8748-4547-BD2C-B6333401B381) (Shutdown)

We can also install an app into the simulator instance from the command-line. However, to be able to install any app produced by Nix, we must first copy the app to a temp directory and restore write permissions:

$ appTmpDir=$(mktemp -d -t appTmpDir)
$ cp -r "$(echo ${app}/*.app)" $appTmpDir
$ chmod -R 755 "$(echo $appTmpDir/*.app)"

The reason why we need to do this is because Nix makes a package immutable after it has been built by removing the write permission bits. After restoring the permissions, we can install it in the simulator by running:

$ xcrun simctl install $udid "$(echo $appTmpDir/*.app)"

And launch the app in the simulator with the following command:

$ xcrun simctl launch 0AD5FC1C-A360-4D05-9D6A-FD719C46A149 \
    MyCompany.HelloWorld

Like the old simulator function, I have encapsulated the earlier described steps in a Nix function that generates a script spawning the simulator instance automatically. The example app can be deployed by writing the following expression:

{xcodeenv, helloworld}:

xcodeenv.simulateApp {
  name = "HelloWorld";
  bundleId = "MyCompany.HelloWorld";
  app = helloworld;
}

By running the following command-line instructions, we can automatically deploy an app in a simulator instance:

$ nix-build -A simulate_helloworld
/nix/store/jldajknmycjwvf3s6n71x9ikzwnvgjqs-simulate-HelloWorld
./result/bin/run-test-simulator 0AD5FC1C-A360-4D05-9D6A-FD719C46A149

And this is what the result looks like:


The UDID parameter passed to the script is not required. If a UDID has been provided, it deploys the app to that particular simulator instance. If the UDID parameter is omitted, it displays a list of simulator instances and asks the user to select one.

Conclusion


In this blog post, I have described an addition to the Nix function that builds iOS application to support multiple versions of Xcode. Furthermore, I have implemented a new simulator spawning script that works with Xcode 6.

The example case can be obtained from my GitHub page.

Wednesday, October 8, 2014

Deploying NPM packages with the Nix package manager

I have encountered several people saying that the Nix package manager is a nice tool, but they do not want to depend on it to build software. Instead, they say that they want to keep using the build tools they are familiar with.

To clear up some confusion: Nix's purpose is not to replace any build tools, but complementing them by composing isolated environments in which these build tools are executed.

Isolated environments


Isolated environments composed by Nix have the following traits:

  • All environment variables are initially cleared or set to dummy values.
  • Environment variables are modified in such a way that only the declared dependencies can be found, e.g. by adding the full path of these packages (residing in separate directories) to PATH, PERL5LIB, CLASSPATH etc.
  • Processes can only write to a designated temp folder and output folders in the Nix store. Write access to any other folder is restricted.
  • After the build has finished, the output files in the Nix store are made read-only and their timestamps are reset to 1 UNIX-time.
  • The environment can optionally be composed in a chroot environment in which no undeclared dependencies and non-package related arbitrary files on the filesystem can be accidentally accessed, no network activity is possible and other processes cannot interfere.

In these environments, you can execute many kinds of build tools, such as GNU Autotools, GNU Make, CMake, Apache Ant, SCons, Perl's MakeMaker and Python's setuptools, typically with little problems. In Nixpkgs, a collection of more than 2500 mostly free and open-source packages, we run many kinds of build tools inside isolated environments composed by Nix.

Moreover, besides running build tools, we can also do other stuff in isolated environments, such as running unit tests, or spawning virtual machine instances in which system integration tests are performed.

So what are the benefits of using such an approach as opposed to running build tools directly in an ad-hoc way? The main benefit is that package deployment (and even entire system configurations and networks of services and machines) become much more reliable and reproducible. Moreover, we can also run multiple builds safely in parallel improving the efficiency of deployment processes.

The only requirements that must be met in a software project are some simple rules so that builds do not fail because of the restrictions that these isolated environments impose. A while ago, I have written a blog post on techniques and lessons to improve software deployment that gives some more details on this. Moreover, if you follow these rules you should still be able to build your software project with your favourite build tools outside Nix.

(As a sidenote: Nix can actually also be a used as a build tool, but this application area is still experimental and not frequently used. More info on this can be found in Chapter 10 of Eelco Dolstra's PhD thesis that can be obtained from his publications page).

Dependency management


The fact that many build tools can be complimented by Nix probably sounds good, but there is one particular class of build tools that are problematic to use with Nix -- namely build tools that also do dependency management in addition to build management. For these kinds of tools, the Nix package manager conflicts, because the build tool typically insists taking over Nix's responsibilities as a dependency manager.

Moreover, Nix's facilities typically restrict such tools to consult external resources, but if we would allow them to do their own dependency management tasks (which is actually possible by hacking around Nix's deployment model), then the corresponding hash codes inside the Nix store paths (which are derived from all buildtime dependencies) are no longer guaranteed to accurately represent the same build results limiting reliable and reproducible deployment. The fact that other dependency managers use weaker nominal version specifications mainly contributes to that.

Second, regardless of what package manager is used, you can no longer rely on the package management system's dependency manager to deploy a system, but you also depend on extra tools and additional distribution channels, which is generally considered tedious by software distribution packagers and end-users.

NPM package manager


A prominent example of a tool doing both build and dependency management is the Node.js Package Manager (NPM), which is the primary means within the Node.js community to build and distribute software packages. It can be used for a variety of Node.js related deployment tasks.

The most common deployment task is probably installing the NPM package dependencies of a development project. What developers typically do is entering the project's working directory and running:

$ npm install

To install all its dependencies (which are obtained from the NPM registry, external URLs and Git repositories) in a special purpose folder named node_modules/ in the project workspace so that it can be run.

You can also globally install NPM packages from the NPM registry (such as command-line utilities), by running:

$ npm install -g nijs

The above command installs a NPM package named NiJS globally including all its dependencies. After the installation has been completed you should be able to run the following instruction on the command-line:

$ nijs-build --help

NPM related deployment tasks are driven by a specification called package.json that is included in every NPM package or the root folder of a development project. For example, NiJS' package.json file looks as follows:

{
  "name" : "nijs",
  "version" : "0.0.18",
  "description" : "An internal DSL for the Nix package manager in JavaScript",
  "repository" : {
    "type" : "git",
    "url" : "https://github.com/svanderburg/nijs.git"
  },
  "author" : "Sander van der Burg",
  "license" : "MIT",
  "bin" : {
    "nijs-build" : "./bin/nijs-build.js",
    "nijs-execute" : "./bin/nijs-execute.js"
  },
  "main" : "./lib/nijs",
  "dependencies" : {
    "optparse" : ">= 1.0.3",
    "slasp": "0.0.4"
  }
}

The above package.json file defines a package configuration object having the following properties:

  • The name and version attributes define the name of the package and its corresponding version number. These two attributes are mandatory and if they are undefined, NPM deployment fails. Moreover, version numbers are required to follow the semver standard. One of semver's requirements is that the version attribute should consist of three version components.
  • The description, repository, author and license attributes are simply just meta information. They are not used during the execution of deployment steps.
  • The bin attribute defines which executable files it should deploy and to which CommonJS modules in the package they map.
  • The main attribute refers to the module that is primary entry point to the package if it is included through require().
  • The dependencies parameter specifies the dependencies that this package has on other NPM packages. This package depends on a library called optparse that must be of version 1.0.3 or higher and a library named slasp which must be exactly of version 0.0.4. More information on how NPM handles dependencies is explained in the next section.

Since the above package is a pure JavaScript package (which most NPM packages are) no build steps are needed. However, if some package do need to perform build steps, e.g. compiling CoffeeScript to JavaScript, or building bindings to native code, then a collection of scripts can be specified, which are run at various times in the lifecycle of a package, e.g. before and after the installation steps. These scripts can (for example) execute the CoffeeScript compiler, or invoke Gyp that compiles bindings to native code.

Replacing NPM's dependency management


So how can we deploy NPM packages in an isolated environment composed by Nix? In other words: how can we "complement" NPM with Nix?

To accomplish this view, we must substitute NPM's dependency manager, that conflicts with the Nix package manager, by something that does the dependency management the "Nix way" while retaining the NPM semantics and keeping its build facilities.

Luckily, we can easily do that by just running NPM inside a Nix expression and "fooling" it not to install any dependencies itself, by providing a copies of these dependencies in the right locations ourselves.

For example, to make deployment of NiJS work, we can just simply extract the tarball's contents, copy the result into the Nix store, entering the output folder, and copying its dependencies into the node_modules directory ourselves:
mkdir -p node_modules
cp -r ${optparse} node_modules
cp -r ${slasp} node_modules
(The above antiquoted expressions, such as ${optparse} refer to the result of Nix expressions that build the corresponding dependencies).

Finally, we should be able to run NPM inside a Nix expression as follows:

$ npm --registry http://www.example.com --nodedir=${nodeSources} install

When running the above command-line instruction after the copy commands, NPM notifies that all the required dependencies of NiJS are already present and simply proceeds without doing anything.

We also provide a couple of additional parameters to npm install:

  • The --registry parameter prevents that, if any dependency is appears to be missing, the NPM registry is consulted, which is undesirable. We want deployment of NPM package dependencies to be Nix's responsibility and making it fail when dependency specifications are incomplete is exactly what we need to be sure that we correctly specify all required dependencies.
  • The --nodedir parameter specifies where the Node.js source code can be found, which is used to build NPM packages that have bindings to native code. nodeSources is a directory containing the unpacked Node.js source code:

    nodeSources = runCommand "node-sources" {} ''
      tar --no-same-owner --no-same-permissions -xf ${nodejs.src}
      mv node-* $out
    '';
    

  • When running NPM in a source code directory (as shown earlier), all development dependencies are installed as well, which is often not required. By providing the --production parameter, we can deploy the package in production mode, skipping the development dependencies.

    Unfortunately, there is one small problem that could occur with some packages defining a prepublish script -- NPM tries to execute this script while a development dependency might be missing causing the deployment to fail. To remedy this problem, I also provide the --ignore-scripts parameter to npm install and I only run the install scripts afterwards, through:

    $ npm run install --registry http://www.example.com --nodedir=${nodeSources}
    

Translating NPM's dependencies


The main challenge of deploying NPM packages with Nix is implementing a Nix equivalent for NPM's dependency manager.

Dependency classes


Currently, an NPM package configuration could declare the following kinds of dependencies which we somehow have to fit in Nix's deployment model:

  • The dependencies attribute specifies which dependencies must be installed along with the package to run it. As we have seen earlier, simply copying the package of the right version into the node_modules folder in the Nix expression suffices.
  • The devDependencies attribute specifies additional dependencies that are installed in development mode. For example, when running: npm install inside the folder of a development project, the development dependencies are installed as well. Also, simply copying them suffices to allow deployment in a Nix expression to work.
  • The peerDependencies attribute might suggest another class of dependencies that are installed along with the package, because of the following sentence in the package.json specification:

    The host package is automatically installed if needed.

    After experimenting with a basic package configuration containing only one peer dependency, I discovered that peer dependencies are basically used as a checking mechanism to see whether no incompatible versions are accidentally installed. In a Nix expression, we don't have to do any additional work to support this and we can leave the check up to NPM that we run inside the Nix expression.
  • bundledDependencies affects the publishing process of the package to the NPM registry. The bundled dependencies refer to a subset of the declared dependencies that are statically bundled along with the package when it's published to the NPM registry.

    When downloading and unpacking a package from the NPM registry that has bundled dependencies, then a node_modules folder exist that contains these dependencies including all their dependencies.

    To support bundled dependencies in Nix, we must first check whether a dependency already exists in the node_modules folder. If this is the case, we should leave as it is, instead of providing the dependency ourselves.
  • optionalDependencies are also installed along with a package, but do not cause the deployment to fail if any error occurs. In Nix, optional dependencies can be supported by using the same copying trick as regular dependencies. However, accepting failures (especially non-deterministic ones), is not something the Nix deployment model supports. Therefore, I did not derive any equivalent for it.

Version specifications


There are various ways to refer to a specific version of a dependency. Currently, NPM supports the following kinds of version specifications:

  • Exact version numbers (that comply with the semver standard), e.g. 1.0.1
  • Version ranges complying with the semver standard, e.g. >= 1.0.3, 5.0.0 - 7.2.3
  • Wildcards complying with the semver standard, e.g. any version: * or any 1.0 version: 1.0.x
  • The latest keyword referring to the latest stable version and unstable keyword referring to the latest unstable version.
  • HTTP/HTTPS URLs referring to a TGZ file being an NPM package, e.g. http://localhost/nijs-0.0.18.tgz.
  • Git URLs referring to a Git repositories containing a NPM package, e.g. https://github.com/svanderburg/nijs.git.
  • GitHub identifiers, referring to an NPM package hosted at GitHub, e.g. svanderburg/nijs
  • Local paths, e.g. /home/sander/nijs

As described earlier, we can't leave fetching the dependencies up to NPM, but Nix has to do this instead. For most version specifications (the only exception being local paths) we can't simply write a function that takes a version specifier as input and fetches it:

  • Packages with exact version numbers and version ranges are fetched from the NPM registry. In Nix, we have to translate these into fetchurl {} invocations, which requires an URL and an output hash value as as parameters allowing us to check the result to make builds reliable and reproducible.

    Luckily, we can retrieve the URL to the NPM package's TGZ file and its corresponding SHA1 hash by fetching the package's metadata from the NPM registry, by running:
    $ npm info nijs@0.0.18
    { name: 'nijs',
      description: 'An internal DSL for the Nix package manager in JavaScript',
      'dist-tags': { latest: '0.0.18' },
      ...
      dist: 
       { shasum: 'bfdf140350d2bb3edae6b094dbc31035d6c7bec8',
         tarball: 'http://registry.npmjs.org/nijs/-/nijs-0.0.18.tgz' },
      ...
    }
    

    We can translate the above metadata into the following Nix function invocation:

    fetchurl {
      name = "nijs-0.0.18.tgz";
      url = http://registry.npmjs.org/nijs/-/nijs-0.0.18.tgz;
      sha1 = "bfdf140350d2bb3edae6b094dbc31035d6c7bec8";
    }
    

  • Version ranges are in principle unsupported in Nix in the sense that you cannot write a function that takes a version range specifier and simply downloads the latest version of the package that conforms to it, since it conflicts with Nix's reproducibility properties.

    If we would allow version ranges to be downloaded then the hash code inside a Nix store path does not necessarily refer to the same build result anymore. For example, running the same download tomorrow might give a different result, because the package has been updated.

    For example, the following path:

    /nix/store/j631r0ak98156v1xkx22n4fsl3zbmzi8-node-slasp-0.0.x
    

    Might refer to slasp version 0.0.4 today and to version 0.0.5 tomorrow, while the hash code remains identical. This is incompatible with Nix's deployment model.

    To still support deployment of packages having dependencies on version ranges of packages, we basically have to "snapshot" a dependency version by running:

    $ npm info nijs@0.0.x
    

    and create a fetchurl {} invocation from the particular version that is returned. The disadvantage of this approach is that, if we want to keep our versions up to date, we have to repeat this step every time a package has been updated.

  • The same thing applies to wildcard version specifiers. However, there is another caveat -- if we encounter a wildcard version specifier, we cannot always assume that the latest conforming version can be taken, because NPM also supports shared dependencies.

    If a shared dependency conforms to a wildcard specifier, then the dependency is not downloaded, but the shared dependency is used instead, which may not necessarily be the latest version. Otherwise, the latest conforming version is downloaded. Shared dependencies are explained in the next section.
  • Also for 'latest' and 'unstable' we must do a snapshot trick. However, we must also do something else. If NPM encounters version specifiers like these, it will always try to consult the NPM registry to check which version corresponds, which is undesirable. To prevent that we must substitute these version specifiers in the package.json file by '*'.
  • For HTTP/HTTPS and Git/GitHub URLs, we must manually compose fetchurl {} and fetchgit {} function invocations, and we must compute their output hashes in advance. The nix-prefetch-url and nix-prefetch-git utilities are particularly useful for this. Moreover, we also have to substitute URLs by '*' in the package.json before we run NPM inside a Nix expression, to prevent it from consulting external resources.

Private, shared and cyclic dependencies


Like the Nix package manager, NPM has the ability to support multiple versions of packages simultaneously -- not only the NPM package we intend to deploy, but also all its dependencies (which are also NPM packages) can have their own node_modules/ folder that contain a package's private dependencies.

Isolation works for CommonJS modules, because when a module inside a package tries to include another package, e.g. through:

var slasp = require('slasp');

then first the node_modules/ folder of the package is consulted and the module is loaded from that folder if it exists. Furthermore, the CommonJS module system uses the absolute resolved full paths to the modules to make a distinction between module variants and not only their names. As a consequence, if a resolved path to a module with a same name is different, it's considered a different module by the module loader and thus does not conflict with others.

If a module cannot be found in the private node_modules/ folder, the module loading system recursively looks for node_modules/ folders in the parent directories, e.g.:

./nijs/node_modules/slasp/node_modules
./nijs/node_modules
./node_modules

This is how package sharing is accomplished in NPM.

NPM's policy regarding dependencies is basically that each package stores all its dependencies privately unless a dependency can be found in any of the parent directories that conforms to the version specification declared in the package. In such cases, the private dependency is omitted and a shared one will be used instead.

Also, because a dependency is installed only once, it's also possible to define cyclic dependencies. Although it's generally known that cyclic dependencies are a bad practice, they are actually used by some NPM packages, such as es6-iterator.

The npm help install manual page says the following about cycles:

To avoid this situation, npm flat-out refuses to install any name@version that is already present anywhere in the tree of package folder ancestors. A more correct, but more complex, solution would be to symlink the existing version into the new location. If this ever affects a real use-case, it will be investigated.

In Nix, private and shared dependencies are handled differently. In Nix, packages can be "private" because they are stored in separate folders in the Nix store which paths are made unique because they contain hash codes derived from all its build-time dependencies.

Sharing is accomplished when a package refers to the same Nix store path with the same hash code. In Nix these mechanisms are more powerful, because they are not restricted to specific component types.

Nix does not support cyclic dependencies and lacks the ability to refer to a parent if a package is a dependency of another package.

To simulate NPM's way of sharing packages (and means of breaking dependency cycles) in Nix, I ended up write our function that deploys NPM packages (named: buildNodePackage {}) roughly as follows:

{stdenv, nodejs, ...}:
{name, version, src, dependencies, ...}:
{providedDependencies}:

let
  requiredDependencies = ...;
  shimmedDependencies = ...;
in
stdenv.mkDerivation {
  name = "node-${name}-${version}";
  inherit src;
  ...
  buildInputs = [ nodejs ... ] ++ requiredDependencies;
  buildCommand = ''
    # Move extracted package into the Nix store
    mkdir -p $out/lib/node_modules/${name}
    mv * $out/lib/node_modules/${name}
    cd $out/lib/node_modules/${name}
    ...
    
    mkdir -p node_modules
    # Copy the required dependencies
    # Generate shims for the provided dependencies
    ...

    # Perform the build by running npm install
    npm --registry http://www.example.com --nodedir=${nodeSources} install
    ...

    # Remove the shims
  '';
}

The above expression defines a nested function with the following structure:

  • The first (outermost) function's parameters refer to the build time dependencies used for the deployment of any NPM package, such as the Nix standard environment that contains a basic UNIX toolset (stdenv) and Node.js (nodejs).
  • The second function's parameters refer to a specific NPM package's deployment parameters, such as the name of the package, the version, a reference to the source code (e.g. local path, URL or Git repository) and its dependencies.
  • The third (innermost) function's parameter (providedDependencies) is used by a package to propagate the identifiers of the already provided shared dependencies to a dependency that's being included, so that they are not deployed again. This is required to simulate NPM's shared dependency mechanism and to escape infinite recursion, because of cyclic dependencies.
  • From the dependencies and providedDependencies parameters, we can determine the required dependencies that we actually need to include privately to deploy the package. requiredDependencies are the dependencies minus the providedDependencies. The actual computation is quite tricky:
    • The dependencies parameter could be encoded as follows:
      {
        optparse = {
          ">= 1.0.3" = {
            version = "1.0.5";
            pkg = registry."optparse-1.0.5";
          };
        };
        ...
      }
      

      The dependencies parameter refers to an attribute set in which each attribute name represents a package name. Each member of this attribute set represents a dependency specification. The dependency specification refers to an attribute set that provides the latest snapshot of the corresponding version specification.
    • The providedDependences parameter could be encoded as follows:
      {
        optparse."1.0.5" = true;
        ...
      }
      

      The providedDependencies parameter is an attribute set composed of package names and a versions. If a package is in this attribute set then it means it has been provided by any of the parents and should not be included again.
    • We use the semver utility to see whether any of the provided dependencies map to any of the version specifications in dependencies. For example for optparse means that we run:

      $ semver -r '>= 1.0.3' 1.0.5
      $ echo $?
      0
      

      The above command exits with a zero exit status, meaning that there is a shared dependency providing it and we should not deploy optparse privately again. As a result, it's not added to the required dependencies.
    • The above procedure is basically encapsulated in a derivation that generates a Nix expression with the list of required dependencies that gets imported again -- a "trick" that I also used in NiJS and the Dynamic Disnix framework.

      The reason why we execute this procedure in a separate derivation is that, if we do the same thing in the builder environment of the NPM package, we always refer to all possible dependencies which prevents us escaping any potential infinite recursion.
  • The required dependencies are copied into the private node_modules/ as follows:

    mkdir -p node_modules
    cp -r ${optparse propagatedProvidedDependencies} node_modules
    cp -r ${slasp propagatedProvidedDependencies} node_modules
    

    Now the innermost function parameter comes in handy -- to each dependency, we propagate the already provided dependencies, our own dependencies, and the package itself, to properly simulate NPM's way of sharing and breaking any potential cycles.

    As a sidenote: to ensure that dependencies are always correctly addressed, we must copy the dependencies. In older implementations, we used to create symlinks, which works fine for private dependencies, but not for shared dependencies.

    If a shared dependency is addressed, the module system looks relative to its own full resolved path, not to the symlink. Because the resolved path is completely different, the shared dependency cannot be found.
  • For the packages that are not considered required dependencies, we must generate shims to allow the deployment to still succeed. While these dependencies are provided by the includers at runtime, they are not visible in the Nix builder environment at buildtime and, as a consequence, deployment will fail.

    Generating shims is quite easy. Simply generating a directory with a minimal package.json file only containing the name and version is enough. For example, the following suffices to fool NPM that the shared dependency optparse version 1.0.5 is actually present:

    mkdir node_modules/optparse
    cat > node_modules/optparse/package.json <<EOF
    {
      "name": "optparse",
      "version": "1.0.5"
    }
    EOF
    

  • Then we run npm install to execute the NPM build steps, which should succeed if all dependencies are correctly specified.
  • Finally, we must remove the generated shims, since they do not have any relevant meaning anymore.

Manually writing a Nix expression to deploy NPM packages


The earlier described function: buildNodePackage {} can be used to manually write Nix expressions to deploy NPM packages:

with import <nixpkgs> {};

let
  buildNodePackage = import ./build-node-package.nix {
    inherit (pkgs) stdenv nodejs;
  };

  registry = {
    "optparse-1.0.5" = buildNodePackage {
      ...
    };
    "slasp-0.0.4" = buildNodePackage {
      ...
    };
    "nijs-0.0.18" = buildNodePackage {
      name = "nijs";
      version = "0.0.18";
      src = ./.;
      dependencies = {
        optparse = {
          ">= 1.0.3" = {
            version = "1.0.5";
            pkg = registry."optparse-1.0.5";
          };
        };
        slasp = {
          "0.0.4" = {
            version = "0.0.4";
            pkg = registry."slasp-0.0.4";
          };
        };
      };
    };
  };
in
registry

The partial Nix expression (shown above) can be used to deploy the NiJS NPM package through Nix.

Moreover, it also provides NiJS' dependencies that are also built by the same function abstraction. By using the above expression, and the following command-line instruction:

$ nix-build -A '"nijs-0.0.18"'
/nix/store/y5r0raja6d8xlaav1mhw8jjxvx7bap85-node-nijs-0.0.18

NiJS is deployed by the Nix package manager including its dependencies.

Generating Nix packages from NPM package configurations


The buildNodePackage {} function shown earlier makes it possible to deploy NPM packages with Nix. However, its biggest drawback is that we have to manually write expressions for the package we want to deploy including all its dependencies. Moreover, since version ranges are unsupported, we must manually check for updates and update the corresponding expressions every time, which is labourious and tedious.

To solve this problem, a tool has been developed named: npm2nix that can automatically generate Nix expressions from NPM package.json specifications and collection specifications. It has several kinds of use cases.

Deploying a Node.js development project


Running the following command generates a collection of Nix expressions from a package.json file of a development project:

$ npm2nix

The above command generates three files registry.nix containing Nix expressions for all package dependencies and the packge itself, node-env.nix contains the build logic and default.nix is a composition expression allowing users to deploy the package.

By running the following Nix command with these expressions, the project can be built:

$ nix-build -A build

Generating a tarball from a Node.js development project


The earlier generated expressions can also be used to generate a tarball from the project:

$ nix-build -A tarball

The above command-line instruction (that basically runs npm pack) produces a tarball that can is placed in the following location:

$ ls result/tarballs/npm2nix-6.0.0.tgz

The above tarball can be distributed to others and installed with NPM by running:

$ npm install npm2nix-6.0.0.tgz

Deploying a development environment of a Node.js development project


The following command-line instruction uses the earlier generated expressions to deploy all the dependencies and opens a development environment:

$ nix-shell -A build

Within this shell session, files can be modified and run without any hassle. For example, the following command should work without any trouble:

$ node bin/npm2nix.js --help

Deploying a collection of NPM packages from the NPM registry


You can also deploy existing NPM packages from the NPM registry, which is driven by a JSON specification that looks as follows:

[
  "async",
  "underscore",
  "slasp",
  { "mocha" : "1.21.x" },
  { "mocha" : "1.20.x" },
  { "nijs": "0.0.18" },
  { "npm2nix": "git://github.com/NixOS/npm2nix.git" }
]

The above specification is basically an array of objects. For each element that is a string, the latest version is obtained from the NPM registry. To obtain a specific version of a package, an object must defined in which the keys are the names of the packages and the values are their version specifications. Any version specification that NPM supports can be used.

Nix expressions can be generated from this JSON specification as follows:

$ npm2nix -i node-packages.json

And using the generated Nix expressions, we can install async through Nix as follows:

$ nix-env -f default.nix -iA async

For every package for which the latest version has been requested, we can directly refer to the name of the package to deploy it.

For packages for which a specific version has been specified, we must refer to it using an attribute that name that is composed of its name and version specifier.

The following command can be used to deploy the first version of mocha declared in the JSON configuration:

$ nix-env -f default.nix -iA '"mocha-1.21.x"'

npm2nix can be referenced as follows:

$ nix-env -f default.nix \
    -iA '"npm2nix-git://github.com/NixOS/npm2nix.git"'

Since every NPM package resolves to a package name and version number we can also deploy any package by using an attribute consisting of its name and resolved version number. This command deploys NiJS version 0.0.18:

$ nix-env -f default.nix -iA '"nijs-0.0.18"'

The above command also works with dependencies of any package that are not declared in the JSON configuration file, e.g.:

$ nix-env -f default.nix -iA '"slasp-0.0.4"'

Concluding remarks


In this lengthy blog post (which was quite a project!) I have outlined some differences between NPM and Nix, sketched an approach that can be used to deploy NPM packages with Nix, and described a generator: npm2nix that automates this approach.


The reason why I wrote this stuff down is that the original npm2nix developer has relinquished his maintainership and I became co-maintainer. Since the NixOS sprint in Ljubljana I've been working on reengineering npm2nix and solving the problem with cyclic dependencies and version mismatches with shared dependencies. Because the problem is quite complicated, I think it would be good to have something documented that describes the problems and my thoughts.

As part of the reengineering process, I ported npm2nix from CoffeeScript to JavaScript, used some abstraction facilities to tidy up the pyramid code (caused by nesting of callbacks), and modularized the codebase it a bit further.

I am using NiJS for the generation of Nix expressions, and I modified it to have most Nix language concepts supported (albeit some of them can only be written in an abstract syntax). Moreover, the expressions generated by NiJS are now also pretty printed so that the generated code is still (mostly) readable.

The reengineered npm2nix can be obtained from the reengineering branch of my private GitHub fork and is currently in testing phase. Once it is considered stable enough, it will replace the old implementation.

Acknowledgements


The majority of npm2nix is not my work. Foremost, I'd like to thank Shea Levy, who is the original developer/author of npm2nix. He was maintaining it since 2012 and figured out most of NPM's internals, mappings of NPM concepts to Nix and how to use NPM specific modules (such as the NPM registry client) to obtain metadata from the NPM registry. Most of the stuff in the reengineered version is ported directly from the old implementation done by him.

Also I'd like to thank the other co-maintainers: Rok Garbas and Rob Vermaas for their useful input during the NixOS sprint in Ljubljana.

Finally, although the feedback period is open for only a short time, I've already received some very useful comments on #nixos and the Nix mailing list by various developers that I would like to thank.

Related work


NPM is not the only tool that does build and dependency management. Another famous (or perhaps notorious!) tool I found myself struggling with in the past was Apache Maven, which is quite popular in the Java world.

Furthermore, converters for other kinds of packages to Nix also exists. Other converters I am currently aware of are: cabal2nix, python2nix, go2nix, and bower2nix.

Saturday, August 9, 2014

Wireless ad-hoc distributions of iOS applications with Hydra

In a number of earlier blog posts, I have shown Hydra, a Nix-based continuous integration server, and Nix functions allowing someone to automatically build mobile applications for Android and iOS with the Nix package manager (and Hydra).

Apart from being able to continuously build new versions of mobile applications, Hydra offers another interesting benefit -- we can use a web browser on an Android device, such as a phone or tablet (or even an emulator instance) to open the Hydra web interface, and conveniently install any Android app by simply clicking on the resulting hyperlink to an APK bundle.

It is also possible to automatically deliver iOS apps in a similar way. However, accomplishing this with Hydra turns out to be quite tedious and complicated. In this blog post, I will explain what I did to make this possible.

Wireless adhoc distributions of iOS apps


According to the following webpage: http://gknops.github.io/adHocGenerate two requirements have to be met in order to provide wireless adhoc releases of iOS apps.

First, we must compose a plist file containing a collection of meta attributes of the app to be distributed. For example:

<plist version="1.0">
<dict>
    <key>items</key>
    <array>
        <dict>
            <key>assets</key>
            <array>
                <dict>
                    <key>kind</key>
                    <string>software-package</string>
                    <key>url</key>
                    <string>http://192.168.1.101/Renamed.ipa</string>
                </dict>
            </array>
            <key>metadata</key>
            <dict>
                <key>bundle-identifier</key>
                <string>com.myrenamedcompany.renamedapp</string>
                <key>bundle-version</key>
                <string>1.0</string>
                <key>kind</key>
                <string>software</string>
                <key>title</key>
                <string>Renamed</string>
            </dict>
        </dict>
    </array>
</dict>
</plist>

The above plist file defines a software package with bundle id: com.myrenamedcompany.renamedapp, version: 1.0 and name: Renamed. The corresponding IPA bundle is retrieved from the following URL: http://192.168.1.101/Renamed.ipa.

The second thing that we have to do is opening a specialized URL in the browser of an iOS device that refers to the plist file that we have defined earlier:

itms-services://?action=download-manifest&url=http://192.168.1.101/distribution.plist

If the plist file properly reflects the app's properties and the signing of the IPA file is done right, e.g. the device is authorized to install the app, then it should be automatically installed on the device after the user has accepted the confirmation request.

Generating a plist file and link page in a Nix/Hydra build


At first sight, integrating wireless adhoc distribution support in Nix (and Hydra) builds seemed to look easy to me -- I just generate the required plist file and an HTML page containing the specialized link URL (that gets clicked automatically by some JavaScript code) and expose these files as Hydra build products so that they are accessible from Hydra's web interface.

Unfortunately, it turned out it is actually a bit more complicated than I thought -- the URLs to the plist and IPA files must be absolute. An absolute path to an IPA file served by Hydra may look as follows:

http://192.168.1.101/build/35256/download/1/Renamed.ipa

Two components of the URL are causing a bit of inconvenience. First, we must know the hostname of the Hydra server. If I would make this value a build property, then the build becomes dependent on Hydra's hostname, which forces us to rebuild the app if it changes for some reason.

Second, the URL contains a unique build id assigned by Hydra that we do not know while performing the build. We have to obtain this value by some other means.

Solution: using page indirection


To solve this problem, I used a very hacky solution introducing an extra layer of indirection -- I have adapted the Nix function that builds iOS applications to generate an HTML file as a Hydra build product from the following template:

<!DOCTYPE html>

<html>
    <head>
        <title>Install IPA</title>
    </head>
    
    <body>
        <a id="forwardlink" href="@INSTALL_URL@">
            Go to the install page or wait a second
        </a>
        
        <script type="text/javascript">
            setTimeout(function() {
                var link = document.getElementById('forwardlink');
                
                if(document.createEvent) {
                    var eventObj = document.createEvent('MouseEvents');
                    eventObj.initEvent('click', true, false);
                    link.dispatchEvent(eventObj);
                } else if(document.createEventObject) {
                    link.fireEvent('onclick');
                }
            }, 1000);
        </script>
    </body>
</html>

What the above page does is showing a hyperlink that redirects the user to another page. Some JavaScript code automatically clicks on the link after one second. After clicking on the link, the user gets forwarded to another web page that is responsible for providing the installation link. We use this obscure page indirection trick to allow the next page to extract some relevant Hydra properties from the referrer URL.

The build script substitutes the @INSTALL_URL@ template property by a relative (or absolute) path that may look as follows:

/installipa.php?bundle=com.myrenamedcompany.renamedapp&version=1.0&title=Renamed

Besides forwarding the user to another page, we also pass the relevant build properties that we need to generate a plist file as GET parameters. Furthermore, the generated HTML build product's URL has nearly the same structure as the URL of an IPA file:

http://192.168.1.101/build/35256/download/2/Renamed.html

The fact that the build product URL of the redirection page has nearly the same structure makes it quite easy for us to extract the remaining properties (the hostname and build id) we need to generate the plist file.

The PHP page that we link to (/installipa.php) is responsible for generating a web page with the specialized itms-services:// URL that triggers an installation. With the following PHP code we can extract the hostname, app name and build id from the referrer URL:

$url_components = parse_url($_SERVER["HTTP_REFERER"]);
$hostname = $url_components["host"];
$app_path = dirname(dirname($url_components["path"]));
$app_name = basename($url_components["path"], ".html");

We can determine the protocol that is being used as follows:

if($_SERVER["HTTPS"] == "")
    $protocol = "http://";
else
    $protocol = "https://";

And compose the absolute IPA URL out of the previous variables:

$ipa_url = $protocol.$hostname.$app_path."/1/".$app_name.".ipa";

Then we display a hyperlink with the specialized installation URL that is generated as follows:

<?php
$plistURL = $protocol.$hostname."/distribution.plist.php".$plistParams;
?>
<a href="itms-services://?action=download-manifest&amp;url=<?php print($plistURL); ?>">
    Click this link to install the IPA
</a>

The plist file that the itms-services:// URL refers to is another PHP script that generates the plist dynamically from a number of GET parameters. The GET parameters are composed as follows:

$plistParams = urlencode("?ipa_url=".$ipa_url.
    "&bundleId=".$_REQUEST["bundleId"].
    "&version=".$_REQUEST["version"].
    "&title=".$_REQUEST["title"]);

By applying the same JavaScript trick shown earlier, we can also automatically click on the installation link to save the user some work.

Adapting Hydra's configuration to use the IPA installation script


To allow users to actually do wireless adhoc installations, the two PHP scripts described earlier must be deployed to the Hydra build coordinator machine. If NixOS is used to deploy the Hydra coordinator machine, then it is simply a matter of adding a few additional configuration properties to the HTTP reverse proxy service section of its NixOS configuration file:

services.httpd = {
  enable = true;
  adminAddr = "admin@example.com";
  hostName = "hydra.example.com";
  extraModules = [
    { name = "php5"; path = "${pkgs.php}/modules/libphp5.so"; }
  ];
  documentRoot = pkgs.stdenv.mkDerivation {
    name = "distribution-proxy";
    src = pkgs.fetchgit {
      url = https://github.com/svanderburg/nix-xcodeenvtests.git;
      rev = "0ba187cc83941bf16c691094480f0632b8116e48";
      sha256 = "4f440e4f3c7b58c40b86e2c8c18608606b64bf341aed233519e9023fff1ceb01";
    };
    buildCommand = ''
      mkdir -p $out
      cp $src/src/distribution-proxy/*.php $out
    '';
  };

  extraConfig = ''
    <proxy>
      Order deny,allow
      Allow from all
    </proxy>
          
    ProxyPass /installipa.php !
    ProxyPass /distribution.plist.php !
          
    ProxyRequests     Off
    ProxyPreserveHost On
    ProxyPass         /  http://localhost:3000/ retry=5 disablereuse=on
    ProxyPassReverse  /  http://localhost:3000/
  '';
};

What I did in the above reverse proxy server configuration snippet, is configuring the documentRoot to refer to a folder containing the two PHP scripts we have shown earlier. The scripts are retrieved from a Git repository. Before I configure the reverse proxy, I declare that two request URLs, namely: the PHP scripts, should not be forwarded to Hydra's Catalyst server.

Usage


After setting up a Hydra instance that hosts these two PHP scripts, we can build an iOS app (such as our trivial example testcase) that includes an HTML forwarding page that allows us to automatically install the app on an iOS device. This can be done with the following expression:

{xcodeenv}:

xcodeenv.buildApp {
  name = "Renamed";
  src = ../../src/Renamed;
  release = true;

  certificateFile = /Users/sander/mycertificate.p12;
  certificatePassword = "secret";
  codeSignIdentity = "iPhone Distribution: My Cool Company";  
  provisioningProfile = /Users/sander/provisioningprofile.mobileprovision;
  generateIPA = true;

  enableWirelessDistribution = true;
  installURL = "/installipa.php";
  bundleId = "com.mycoolcompany.renamed";
  version = "1.0";
  title = "Renamed";
}

Setting the enableWirelessDistribution parameter to true makes the build function generating the HTML page as build product. The installURL, bundleId, version and title parameters are used for the page forwarding and the plist file generation.

Result


By setting up a Hydra jobset using the above function, we can open the Hydra web application in a web browser on an iOS device and navigate to an iOS build:


Clicking on the 'Documentation of type install' build product does our page forwarding trick. After 2 seconds a confirmation dialog should appear:


After clicking on the 'Install' button, the app gets installed and appears in the program menu:


And finally we can run it! Isn't it great?

Concluding remarks


In this blog post I have described a hacky method using page indirection making it possible to use Hydra to do wireless adhoc distributions of iOS apps.

Unfortunately, I also discovered that for devices running iOS 7.1 and onwards, an HTTPS connection to the plist and IPA files is required, with a valid, trustable cross-signed certificate, making things even more tedious and complicated.

The hacky PHP scripts described in this blog post are part of the Nix xcode test package that can be obtained from my GitHub page.

It is also quite funny to realise that all these steps are not required at all for Android apps. Simply making APK files available for download is enough.

Thursday, July 31, 2014

Managing private Nix packages outside the Nixpkgs tree

In a couple of older blog posts, I have explained the basic concepts of the Nix package manager as well as how to write package "build recipes" (better known as Nix expressions) for it.

Although Nix expressions may look unconventional, the basic idea behind specifying packages in the Nix world is simple: you define a function that describes how to build a package from source code and its dependencies, and you invoke the function with the desired variants of the dependencies as parameters to build it. In Nixpkgs, a collection of more than 2500 (mostly free and open source) packages that can be deployed with Nix, all packages are basically specified like this.

However, there might still be some practical issues. In some cases, you may just want to experiment with Nix or package private software not meant for distribution. In such cases, you typically want to store them outside the Nixpkgs tree.

Although the Nix manual describes how things are packaged in Nixpkgs, it does not (clearly) describe how to define and compose packages while keeping them separate from Nixpkgs.

Since it is not officially documented anywhere and I'm getting (too) many questions about this from beginners, I have decided to write something about it.

Specifying a single private package


In situations in which I want to quickly try or test one simple package, I typically write a Nix expression that looks as follows:

with import <nixpkgs> {};

stdenv.mkDerivation {
  name = "mc-4.8.12";
  
  src = fetchurl {
    url = http://www.midnight-commander.org/downloads/mc-4.8.12.tar.bz2;
    sha256 = "15lkwcis0labshq9k8c2fqdwv8az2c87qpdqwp5p31s8gb1gqm0h";
  };
  
  buildInputs = [ pkgconfig perl glib gpm slang zip unzip file gettext
      xlibs.libX11 xlibs.libICE e2fsprogs ];

  meta = {
    description = "File Manager and User Shell for the GNU Project";
    homepage = http://www.midnight-commander.org;
    license = "GPLv2+";
    maintainers = [ stdenv.lib.maintainers.sander ];
  };
}

The above expression is a Nix expression that builds Midnight Commander, one of my favorite UNIX utilities (in particular the editor that comes with it :-) ).

In the above Nix expression, there is no distinction between a function definition and invocation. Instead, I directly invoke stdenv.mkDerivation {} to build Midnight Commander from source and its dependencies. I obtain the dependencies from Nixpkgs by importing the composition attribute set into the lexical scope of the expression through with import <nixpkgs> {};.

I can put the above file (named: mc.nix) in a folder outside the Nixpkgs tree, such as my home directory, and build it as follows:

$ nix-build mc.nix
/nix/store/svm98wmbf01dswlfcvvxfqqzckbhp5n5-mc-4.8.12

Or install it in my profile by running:

$ nix-env -f mc.nix -i mc

The dependencies (that are provided by Nixpkgs) can be found thanks to the NIX_PATH environment variable that contains a setting for nixpkgs. On NixOS, this environment variable has already been set. On other Linux distributions or non-NixOS installations, this variable must be manually configured to contain the location of Nixpkgs. An example could be:

$ export NIX_PATH=nixpkgs=/home/sander/nixpkgs

The above setting specifies that a copy of Nixpkgs resides in my home directory.

Maintaining a collection private packages


It may also happen that you want to package a few of the dependencies of a private package while keeping them out of Nixpkgs or just simply maintaining a collection of private packages. In such cases, I basically define every a package as a function, which is no different than the way it is done in Nixpkgs and described in the Nix manual:

{ stdenv, fetchurl, pkgconfig, glib, gpm, file, e2fsprogs
, libX11, libICE, perl, zip, unzip, gettext, slang
}:

stdenv.mkDerivation rec {
  name = "mc-4.8.12";
  
  src = fetchurl {
    url = http://www.midnight-commander.org/downloads/mc-4.8.12.tar.bz2;
    sha256 = "15lkwcis0labshq9k8c2fqdwv8az2c87qpdqwp5p31s8gb1gqm0h";
  };
  
  buildInputs = [ pkgconfig perl glib gpm slang zip unzip file gettext
      libX11 libICE e2fsprogs ];

  meta = {
    description = "File Manager and User Shell for the GNU Project";
    homepage = http://www.midnight-commander.org;
    license = "GPLv2+";
    maintainers = [ stdenv.lib.maintainers.sander ];
  };
}

However, to compose the package (i.e. calling the function with the arguments that are used as dependencies), I have to create a private composition expression instead of adapting pkgs/top-level/all-packages.nix in Nixpkgs.

A private composition expression could be defined as follows:

{ system ? builtins.currentSystem }:

let
  pkgs = import <nixpkgs> { inherit system; };
in
rec {
  pkgconfig = import ./pkgs/pkgconfig {
    inherit (pkgs) stdenv fetchurl automake;
  };
  
  gpm = import ./pkgs/gpm {
    inherit (pkgs) stdenv fetchurl flex bison ncurses;
  };
  
  mc = import ./pkgs/mc {
    # Use custom pkgconfig and gpm packages as dependencies
    inherit pkgconfig gpm;
    # The remaining dependencies come from Nixpkgs
    inherit (pkgs) stdenv fetchurl glib file perl;
    inherit (pkgs) zip unzip gettext slang e2fsprogs;
    inherit (pkgs.xlibs) libX11 libICE;
  };
}

The above file (named: custom-packages.nix) invokes the earlier Midnight Commander expression (defining a function) with its required parameters.

Two of its dependencies are also composed in the same expression, namely: pkgconfig and gpm that are also stored outside the Nixpkgs tree. The remaining dependencies of Midnight Commander are provided by Nixpkgs.

To make the above example complete, the directory structure of the set of Nix expressions is supposed to look as follows:

pkgs/
  pkgconfig/
    default.nix
    requires-private.patch 
    setup-hook.sh
  gpm/
    default.nix
  mc/
    default.nix
custom-packages.nix

The expressions for gpm and pkgconfig can be copied from Nixpkgs, by running ($nixpkgs should be replaced by the path to Nixpkgs on your system):

cp -a $nixpkgs/pkgs/pkgs/development/tools/misc/pkgconfig pkgs
cp -a $nixpkgs/pkgs/servers/gpm pkgs

Using the above Nix composition expression file (custom-packages.nix), the other Nix expressions it refers to, and by running the following command-line instruction:

$ nix-build custom-packages.nix -A mc
/nix/store/svm98wmbf01dswlfcvvxfqqzckbhp5n5-mc-4.8.12

I can build our package using our private composition of packages. Furthermore, I can also install it into my Nix profile by running:

$ nix-env -f custom-packages.nix -iA mc

Because the composition expression is also a function taking system as a parameter (which defaults to the same system architecture as the host system), I can also build Midnight Commander for a different system architecture, such as a 32-bit Intel Linux system:

$ nix-build custom-packages.nix -A mc --argstr system i686-linux

Simplifying the private composition expression


The private composition expression shown earlier passes all required function arguments to each package definition, which basically requires anyone to write function arguments twice. First to define them and later to provide them.

In 95% of the cases, the function parameters are typically packages defined in the same composition attribute set having the same attribute names as the function parameters.

In Nixpkgs, there is a utility function named callPackage {} that simplifies things considerably -- it automatically passes all requirements to the function by taking the attributes with the same name from the composition expression. So there is no need to write: inherit gpm ...; anymore.

We can also define our own private callPackage {} function that does this for our private composition expression:

{ system ? builtins.currentSystem }:

let
  pkgs = import <nixpkgs> { inherit system; };
  
  callPackage = pkgs.lib.callPackageWith (pkgs // pkgs.xlibs // self);
  
  self = rec {
    pkgconfig = callPackage ./pkgs/pkgconfig { };
  
    gpm = callPackage ./pkgs/gpm { };
  
    mc = callPackage ./pkgs/mc { };
  };
in
self

The above expression is a simplified version of our earlier composition expression (named: custom-packages.nix) that uses callPackage {} to automatically pass all required dependencies to the functions that build a package.

callPackage itself is composed from the pkgs.lib.callPackageWith function. The first parameter (pkgs // pkgs.xlibs // self) defines the auto-arguments. In this particular case, I have specified that the automatic function arguments come from self (our private composition) first, then from the xlibs sub attribute set from Nixpkgs, and then from the main composition attribute set of Nixpkgs.

With the above expression, we accomplish exactly the same thing as in the previous expression, but with fewer lines of code. We can also build the Midnight Commander exactly the same way as we did earlier:

$ nix-build custom-packages.nix -A mc
/nix/store/svm98wmbf01dswlfcvvxfqqzckbhp5n5-mc-4.8.12

Conclusion


In this blog post, I have described how I typically maintain a single package or a collection packages outside the Nixpkgs tree. More information on how to package things in Nix can be found in the Nix manual and the Nixpkgs manual.

Tuesday, July 22, 2014

Backing up Nix (and Hydra) builds

One of the worst things that may happen to any computer user is that filesystems get corrupted or that storage mediums, such as hard drives, break down. As a consequence, valuable data might get lost.

Likewise, this could happen to machines storing Nix package builds, such as a Hydra continuous build machine that exposes builds through its web interface to end users.

Reproducible deployment


One of the key features of the Nix package manager and its related sub projects is reproducible deployment -- using Nix expressions (which are basically recipes that describe how components are built from source code and its dependencies), we can construct all static components of which a system consists (such as software packages and configuration files).

Moreover, Nix ensures that all dependencies are present and correct, and removes many side effects while performing a build. As a result, producing the same configuration with the same set of expressions on a different machine should yield (nearly) a bit identical configuration.

So if we keep a backup of the Nix expressions stored elsewhere, such as a remote Git repository, we should (in theory) have enough materials to reproduce a previously deployed system configuration.

However, there are still a few inconveniences if you actually have to do this:

  • It takes time to rebuild and redownload everything. Some packages and system configurations might consists of hundreds or thousands of components taking many hours to complete.
  • The source tarballs may not be available from their original download locations anymore. I have encountered these situations quite a few times when I was trying to reproduce very old configurations. Some suppliers may decide to remove old releases after a while, or to move them to different remote locations, which requires me to search for them and to adapt very old Nix expressions, which I preferably don't want to do.
  • We also have to restore state which cannot be done by the Nix package manager. For example, if the Hydra database gets lost, we have to configure all projects, jobsets, user accounts and releases from scratch again, which is tedious and time consuming.

Getting the dependencies of packages


To alleviate the first two inconveniences, we must also backup the actual Nix packages belonging to a configuration including all their dependencies.

Since all packages deployed by the Nix package manager typically reside in a single Nix store folder (typically /nix/store), that may also contain junk and irrelevant stuff, we have to somehow select the packages that we consider relevant.

Binary deployments


In Nix, there are various ways to query specific dependencies of a package. When running the following query on the Nix store path of a build result, such as a Disnix, we can fetch all its runtime dependencies:

$ nix-store --query --requisites /nix/store/sh8025fhmz1wq27663bakmq915a2pf79-disnix-0.3pre1234
/nix/store/31kl46d8l4271f64q074bzi313hjmdmv-linux-headers-3.7.1
/nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19
...
/nix/store/hjbzw7s8wbvrf7mjjfkm1ah6fhnmyhzw-libxml2-2.9.1
/nix/store/hk8wdzs9s52iw9gnxbi1n9npdnvvibma-libxslt-1.1.28
/nix/store/kjlv4klmrarn87ffc5sjslcjfs75ci7a-getopt-1.1.4
/nix/store/sh8025fhmz1wq27663bakmq915a2pf79-disnix-0.3pre1234

What the above command does is listing the transitive Nix store path references that a package contains. In the above example, these paths correspond to the runtime dependencies of Disnix, since they are referenced from bash scripts, as well as the RPATH fields of the ELF binaries and prevent the executables to run properly if any of them is missing.

According to the nix-store manual page, the above closure refers to a binary deployment of a package, since it contains everything required to run it.

Source deployments


We can also run the same query on a store derivation file. While evaluating Nix expressions to build packages -- including its build-time dependencies --, a store derivation file is generated each time the derivation { } function is invoked.

Every Nix expression that builds something indirectly calls this function. The purpose of a derivation is composing environments in which builds are executed.

For example, if we run the previous query on a store derivation file:

$ nix-store --query --requisites /nix/store/3icf7dxf3inky441ps1dl22aijhimbxl-disnix-0.3pre1234.drv
...
/nix/store/4bj56z61q6qk69657bi0iqlmia7np5vc-bootstrap-tools.cpio.bz2.drv
...
/nix/store/4hlq4yvvszqjrwsc18awdvb0ppbcv920-libxml2-2.9.1.tar.gz.drv
/nix/store/g32zn0z6cz824vbj20k00qvj7i4arqy4-setup-hook.sh
/nix/store/n3l0x63zazksbdyp11s3yqa2kdng8ipb-libxml2-2.9.1.drv
/nix/store/nqc9vd5kmgihpp93pqlb245j71yghih4-libxslt-1.1.28.tar.gz.drv
/nix/store/zmkc3jcma77gy94ndza2f1y1rw670dzh-libxslt-1.1.28.drv
...
/nix/store/614h56k0dy8wjkncp0mdk5w69qp08mdp-disnix-tarball-0.3pre1234.drv
/nix/store/3icf7dxf3inky441ps1dl22aijhimbxl-disnix-0.3pre1234.drv

Then all transitive references to the store derivation files are shown, which correspond to all build-time dependencies of Disnix. According to the nix-store manual page the above closure refers to a source deployment of package, since the store derivations are low-level specifications allowing someone to build a package from source including all its build time dependencies.

Cached deployments


The previous query only returns the store derivation files. These files still need to be realised in order to get a build, that may take some time. We can also query all store derivation files and their corresponding build outputs, by running:

$ nix-store --query --requisites --include-outputs \
    /nix/store/3icf7dxf3inky441ps1dl22aijhimbxl-disnix-0.3pre1234.drv
...
/nix/store/zmkc3jcma77gy94ndza2f1y1rw670dzh-libxslt-1.1.28.drv
...
/nix/store/hk8wdzs9s52iw9gnxbi1n9npdnvvibma-libxslt-1.1.28
...
/nix/store/3icf7dxf3inky441ps1dl22aijhimbxl-disnix-0.3pre1234.drv

The above command only includes the realised store paths that have been built before. By adding the --force-realise parameter to the previous command-line instruction, we can force all outputs of the derivations to be built.

According to the nix-store manual page, the above closure refers to a cached deployment of a package.

Backing up Nix components


Besides querying the relevant Nix store components that we intend to backup, we also have to store them elsewhere. In most cases, we cannot just simply copy the Nix store paths to another location and copy it back into the Nix store at some later point:

  • Some backup locations may use more primitive filesystems than Linux (and other UNIX-like systems). For example, we require filesystem features, such as symlinks and read, write and executable bits.
  • We also require necessary meta-information to allow it to be imported into the Nix store, such as the set of references to other paths.

For these reasons, it is recommendable to use nix-store --export, that serializes a collection of Nix store paths into a single file including their meta-information. For example, the following command-line instruction serializes a cached deployment closure of Disnix:

$ nix-store --export $(nix-store --query --requisites --include-outputs \
    /nix/store/3icf7dxf3inky441ps1dl22aijhimbxl-disnix-0.3pre1234.drv) > disnix-cached.closure

The resulting closure file (disnix-cached.closure) can easily be stored on many kinds of mediums, such as an external harddrive using a FAT32 filesystem. We can import the the closure file into another Nix store by running:

$ nix-store --import < disnix-cached.closure

The above command imports Disnix including all its dependencies into the Nix store. If any dependencies are already in the Nix store, then they are skipped. If any dependency appears to be missing, it returns an error. All these properties can be verified because the serialization contains all the required meta-information.

Storing backups of a collection of Nix components efficiently


In principle, the export and import nix-store operations should be sufficient to make reliable backups of any Nix package. However, the approach I described has two drawbacks:

  • For each package, we serialize the entire closure of dependencies. Although this approach is reliable, it is also inefficient if we want to backup multiple packages at the same time. Typically, many packages share the same common set of dependencies. As a consequence, each backup contains many redundant packages wasting a lot of precious disk space.
  • If we change a package's source code, such as Disnix, and rebuild it, we have to re-export the entire closure again, while many of the dependencies of remain the same. This makes the backup process time considerably longer than necessary.

To fix these inefficiencies, we need an approach that stores serializations of each Nix store path individually, so that we can check which paths have been backed up already and which still need to be serialized. Although we could implement such an approach ourselves, there is already a Nix utility that does something similar, namely: nix-push.

Normally, this command is used to optimize the build times of source builds by making binary substitutes available that can be downloaded instead, but it turns out to be quite practical for making backups as well.

If I run the following instruction on a collection of Nix store paths:

$ nix-push --dest /home/sander/cache /nix/store/4h4mb7lb5c0g390bd33k658dgzahkjn7-disnix-0.3pre1234

A binary cache is created in the /home/sander/cache directory from the closure of the Disnix package. The resulting binary cache has the following structure:

$ ls /home/sander/cache
03qpb8b4j4kc1w3fvwg9f8igc4skfsgj9rqb3maql9pi0nh6aj47.nar.xz
053yi53qigf113xsw7n0lg6fsvd2j1mapl6byiaf9vy80a821irk.nar.xz
05vfk68jlgj9yqd9nh1kak4rig379s09.narinfo
06sx7428fasd5bpcq5jlczx258xhfkaqqk84dx2i0z7di53j1sfa.nar.xz
...
11wcp606w07yh8afgnidqvpd1q3vyha7ns6dhgdi2354j84xysy9.nar.xz
...
4h4mb7lb5c0g390bd33k658dgzahkjn7.narinfo
...

For each Nix store path of the closure, an xz compressed NAR file is generated (it is also possible to use bzip2 or no compression) that contains a serialization of an individual Nix store path (without meta-information) and a narinfo file that contains its corresponding meta-information. The prefix of the NAR file corresponds to its output hash while the prefix of the narinfo file corresponds to the hash component of the Nix store path. The latter file contains a reference to the former NAR file.

If, for example, we change Disnix and run the same nix-push command again, then only the paths that have not been serialized are processed while the existing ones remain untouched, saving redundant diskspace and backup time.

We can also run nix-push on a store derivation file. If a store derivation file is provided, a binary cache is generated from the cached deployment closure.

Restoring a package from a binary cache can be done as follows:

$ nix-store --option binary-caches file:///home/sander/cache \
    --realise /nix/store/3icf7dxf3inky441ps1dl22aijhimbxl-disnix-0.3pre1234

Simply realizing a Nix store path while providing the location to the binary cache as a parameter causes it to download the substitute into the Nix store, including all its dependencies.

Creating releases on Hydra for backup purposes


How can this approach be applied to Hydra builds? Since Hydra stores many generations of builds (unless they are garbage collected), I typically make a selection of the ones that I consider important enough by adding them to a release.

Releases on Hydra are created as follows. First, you have to be logged in and you must select a project from the project overview page, such as Disnix:


Clicking on a project will redirect you to a page that shows you the corresponding jobsets. By unfolding the actions tab, you can create a release for that particular project:


Then a screen will be opened that allows you define a release name and description:


After the release has been created, you can add builds to it. Builds can be added by opening the jobs page and selecting build results, such as build.x86_64-linux:


After clicking on a job, we can add it to a release by unfolding the 'Actions' tab and selecting 'Add to release':


The following dialog allows us to add the build to our recently created: disnix-0.3 release:


When we open the 'Releases' tab of the project page and we select the disnix-0.3 release, we can see that the build has been added:


Manually adding individual builds is a bit tedious if you have many them. Hydra has the ability to add all jobs of an evaluation to a release in one click. The only prerequisite is that each build must tell Hydra (through a file that resides in $out/nix-support/hydra-release-name of the build result) to which release it should belong.

For me adapting builds is a bit inconvenient and I also don't need the ability to add builds to arbitrary releases. Instead, I have created a script that adds all builds of an evaluation to a single precreated release, which does not require me to adapt anything.

For example running:

$ hydra-release-eval config.json 3 "disnix-0.3" "Disnix 0.3"

Automatically creates a release with name: disnix-0.3 and description: "Disnix 0.3", and adds all the successful builds of evaluation 3 to it.

Exporting Hydra releases


To backup Hydra releases, I have created a Perl script that takes a JSON configuration file as parameter that looks as follows:

{
  "dbiConnection": "dbi:Pg:dbname=hydra;host=localhost;user=hydra;",
  
  "outDir": "/home/sander/hydrabackup",
  
  "releases": [
    {
      "project": "Disnix",
      "name": "disnix-0.3",
      "method": "binary"
    },
  ]
}

The configuration file defines an object with three members:

  • dbiConnection contains the Perl DBI connection string that connects to Hydra's PostgreSQL database instance.
  • outDir refers to a path in which the binary cache and other backup files will be stored. This path could refer to (for example) the mount point of another partition or network drive.
  • releases is an array of objects defining which releases must be exported. The method field determines the deployment type of the closure that needs to be serialized, which can be either a binary or cache deployment.

By running the following command, I can backup the releases:

$ hydra-backup config.json

The above command creates two folders: /home/sander/hydrabackup/cache contains the binary cache generated by nix-pull using the corresponding store derivation files or outputs of each job. The /home/sander/hydrabackup/releases folder contains text files with the actual paths belonging to the closures of each release.

The backup approach (using a binary cache) also allows me to update the releases and to efficiently make new backups. For example, by changing the disnix-0.3 release and running the same command again, only new paths are being exported.

One of the things that may happen after updating releases is that some NAR and narinfo files have become obsolete. I have also created a script that takes care of removing them automatically. What it basically does is comparing the release's closure files with the contents of the binary cache and removing the files that are not defined in any of the closure files. It can be invoked as follows:

$ hydra-collect-backup-garbage config.json

Restoring Hydra releases on a different machine can be done by copying the /home/sander/hydrabackup folder to a different machine and by running:

$ hydra-restore config.json

Backing up the Hydra database


In addition to releases, we may want to keep the Hydra database so that we don't have to reconfigure all projects, jobsets, releases and user accounts after a crash. A dump of the database can be created, by running:

$ pg_dump hydra | xz > /home/sander/hydrabackup/hydra-20140722.pgsql.xz

And we can restore it by running the following command:

$ xzcat /home/sander/hydrabackup/hydra-20140722.pgsql.xz | psql hydra

Conclusion


In this blog post, I have described an approach that allows someone to fully backup Nix (and Hydra) builds. Although it may feel great to have the ability to do so, it also comes with a price -- closures consume a lot of disk space, since every closure contains all transitive dependencies that are required to run or build it. In some upgrade scenarios, none of the dependencies can be shared which is quite costly.

In many cases it would be more beneficial to only backup the Nix expressions and Hydra database, and redo the builds with the latest versions of the dependencies, unless there is really a good reason to exactly reproduce an older configuration.

Furthermore, I am not the only person who has investigated Hydra backups. The Hydra distribution includes a backup script named: hydra-s3-backup-collect-garbage that automatically stores relevant artifacts in an Amazon S3 bucket. However, I have no clue how to use it and what it's capabilities are. Moreover, I am an old fashioned guy who still wants store backups on physical mediums rather than in the cloud. :).

The scripts described in this blog post can be obtained from my Github page. If other people consider any these scripts useful, I might reserve some time to investigate whether they can be included in the Hydra distribution package.

UPDATE: I just implemented a fix for Hydra that automatically composes a release name out of the project name and evaluation id if no release name has been defined in any of the builds. The fix has been merged into the main Hydra repository. This makes the hydra-release-eval script obsolete.