Inclusive Technology Consulting

One of my favorite things about programming in node is the package management system. In almost all instances, the practice of locally installing modules for each project has simplified my life as a developer.

However, as a long time contributor to Grunt, I have become intimately familiar with one edge case where this practice breaks down. In the hope of mitigating the annoyance for everyone, I created a library to address the issue. It’s called Liftoff.

If you’ve ever built a command line tool for node, especially one that consumes an ecosystem of plugins, you probably know what I’m talking about. Let’s take a look at the problems Liftoff was built to solve.

Global module semantics

Before I dive in, let me explain a few things to be sure we’re all on the same page:

If a module provides a command you want to use in the shell, the easiest way to make it available system-wide is to install it globally (npm install -g modulename).
You cannot have more than one version of a module installed globally without using tools like nvm.
Globally installed modules cannot require other globally installed modules.
Locally installed modules cannot require globally installed modules.

Note: All modules have access to global “core modules”, e.g. fs, path, http.

What this means

The above semantics give rise to a few wrinkles that are not immediately intuitive:

Node based command line tools will often be installed both globally and locally.
The role of the globally installed version should be to provide a command that finds and loads the local version.
If the tool consumes plugins, a locally installed version will not have access to any that have been globally installed. Plugins should be specified as dependencies of each project.

If you are new to node, some of this might seem annoying. My advice? Just roll with it. Given enough time, you’ll wind up with projects using multiple versions of your tool and plugins.

Getting global vs local right

For the sake of discussion, let’s build a command line tool called hacker and make it configurable using a Hackerfile. When we run the hacker command in our shell, we want it to execute the binary from our globally installed version (as specified in our tools’s package.json bin property). As mentioned previously, this command should find and load a local installation of our tool, no matter what version it is.

Because globally installed modules cannot natively require local ones, it’ll take a bit of hacking to make this work. Thankfully, the algorithm we need is already available on npm. It’s called resolve, and it duplicates how node finds modules with several handy configurable options, including the ability to specify a base directory to start a search.

Here’s a simple example of our hacker binary so far:

#!/usr/bin/env node
var resolve = require('resolve');
try {
  var localHacker = resolve.sync('hacker', { basedir: process.cwd() });
  console.log('Found hacker at', localHacker);
  // kick off here
} catch (e) {
  console.log('Unable to find a local installation of hacker.');
  process.exit(1);
}

Making it work well

So, now we can find and run a local version of our tool! We’re pretty much done, right? Not quite. There are numerous niceties to be added. Let’s examine a few of them now.

Intelligent traversal

When we run the hacker command, we might be in a sub-folder of our project. It would be nice if our tool was smart enough to traverse the filesystem looking for a Hackerfile in the nearest ancestor directory. Once again, npm has a module for that—it’s called findup-sync, and this is how we might use it:

#!/usr/bin/env node
var resolve = require('resolve');
var findup = require('findup-sync');
var path = require('path');
var cwd = process.cwd();

var configFile = findup('Hackerfile.js', { cwd: cwd });
if (configFile) {
  console.log('Found Hackerfile:', configFile);
  cwd = path.dirname(configFile);
  process.chdir(cwd);
  console.log('Setting current working directory:', cwd);
} else {
  console.log('No Hackerfile found.');
  process.exit(1);
}

try {
  var localModule = resolve.sync('hacker', { basedir: cwd });
  if (localModule) {
    console.log('Found hacker module:', localModule);
  }
  // kick off here
} catch (e) {
  console.log('Unable to find a local installation of hacker.');
  process.exit(1);
}

Note that once a Hackerfile was located, we changed the working directory of the process to the folder it was in. By doing this, any file operations we perform with our tool will happen relative to our Hackerfile.

Explicit directory specification

Eventually, we might need to run hacker from a directory completely outside of our project. In order to support that, we’ll need to start reading command line flags. There are tons of great option parsers out there: optimist, minimist, yargs, nomnom, nopt and commander.js come to mind. Basically, there is no need for us to build another one!

Here’s what our binary looks like with support for a --cwd flag:

#!/usr/bin/env node
var resolve = require('resolve');
var findup = require('findup-sync');
var path = require('path');
var argv = require('minimist')(process.argv.slice(2));

var cwd = argv.cwd ? argv.cwd : process.cwd();

var configFile = findup('Hackerfile.js', { cwd: cwd });
if (configFile) {
  console.log('Found Hackerfile:', configFile);
  cwd = path.dirname(configFile);
  process.chdir(cwd);
  console.log('Setting current working directory:', cwd);
} else {
  console.log('No Hackerfile found.');
  process.exit(1);
}

try {
  var localModule = resolve.sync('hacker', { basedir: cwd });
  if (localModule) {
    console.log('Found hacker module:', localModule);
  }
  // kick off here
} catch (e) {
  console.log('Unable to find a local installation of hacker.');
  process.exit(1);
}

Supporting JS variants for configuration

If our tool sees widespread adoption, someone will invariably want to write their Hackerfile file in a JS variant we don’t use or care about. It would be a bad idea to explicitly bundle it with our tool, so we need to support another option. Let’s call it --require. It’s a good thing we found that options parser and learned how to use resolve—we need to find a few more local modules!

#!/usr/bin/env node
var resolve = require('resolve');
var findup = require('findup-sync');
var path = require('path');
var argv = require('minimist')(process.argv.slice(2));

var cwd = argv.cwd ? argv.cwd : process.cwd();
var requires = argv.require;

if (requires) {
  if (!Array.isArray(requires)) {
    requires = [requires];
  }
  requires.forEach(function (module) {
    try {
      require(resolve.sync(module, { basedir: cwd }));
      console.log('Loading external module:', module);
    } catch (e) {
      console.log('Unable to load:', module, e);
    }
  });
}

var validExtensions = Object.keys(require.extensions).join(',');
var configNameRegex = 'Hackerfile'+'{'+validExtensions+'}';

var configFile = findup(configNameRegex, { cwd: cwd });
if (configFile) {
  console.log('Found Hackerfile:', configFile);
  cwd = path.dirname(configFile);
  process.chdir(cwd);
  console.log('Setting current working directory:', cwd);
} else {
  console.log('No Hackerfile found.');
  process.exit(1);
}

try {
  var localModule = resolve.sync('hacker', { basedir: cwd });
  if (localModule) {
    console.log('Found hacker module:', localModule);
  }
  // kick off here
} catch (e) {
  console.log('Unable to find a local installation of hacker.');
  process.exit(1);
}

Now, instead of looking for a Hackerfile with a .js extension, we’re looking for one with any extension node understands how to load. If we invoke our command thusly: hacker --require coffee-script/register, our new binary will attempt to require the coffee-script compiler from our local dependencies. If successful, node will be able to load .coffee files.

Make all of this easy

All of this and then some can be automated using Liftoff. The example below does everything I’ve described so far:

#!/usr/bin/env node
var Liftoff = require('liftoff');

var Hacker = new Liftoff({
  name: 'hacker'
}).on('require', function (name, module) {
  console.log('Loading external module:', name);
}).on('requireFail', function (name, err) {
  console.log('Unable to load:', name, err);
});

Hacker.launch(function() {
  if(this.configPath) {
    process.chdir(this.configBase);
    console.log('Setting current working directory:', this.configBase);
    // kick off here
  } else {
    console.log('No Hackerfile found.');
    process.exit(1);
  }
});

The next major version of Grunt (under development here) will be powered by Liftoff. Other libraries are already in the process of adopting it. Gulp is using it today, and jscs will be soon. I hope this re-usable solution will help other developers in their quest to make better tools!

Building Command Line Tools in Node with Liftoff