Private Github Repos With NPM & Heroku

| Comments

As you spend time coding in the Node.js ecosystem, you become increasingly reliant on small modules, both consuming them and building them yourself. This habit of building and composing modular toolsets begins to extend beyond library code, and you start to think of web application code as a collection of modules. Data models, for example, can be easily packaged into a separate Node module for re-use in related applications (and to chunk off a set of application dependencies).

Seeing code that can be broken off into its own module, you dutifully follow your Node.js instincts, break off the code, and move it to its own directory. You create a new private Github repo for this small module, and push it up. But how do you re-integrate this new (actively developed) sub-project back into its containing projects?

NPM has always made it easy to manage Node modules as Git submodules. The NPM submodule command will add a package dependency as a Git submodule, and from then on will basically ignore the containing folder, letting you update and modify the package as you will. There’s also NPM’s link command, which will install a package dependency from a symlink on your development machine rather than from the repository.

But when it comes to the common use case of deploying a web app with a Git submodule package to Heroku, things get complicated:

  1. Your submodules are private Github repositories, so you need to get access to them without sacrificing account security;

  2. Assuming you get around the first challenge, Git submodules are autotomatically installed on Heroku deploy, and then NPM will stubbornly refuse to install the submodule dependencies;

  3. If you’re using npm shrinkwrap to lock down your app dependencies, the presence of Git submodules or linked directories screws up the shrinkwrap metadata and makes install difficult.

There’s a very specific approach you need to take to work around these issues.

First, there’s the account security issue. HTTPS access to your Github repositories allows you to use a username and password instead of an SSH key, but then you’re putting your account credentials in plaintext – an obvious problem. Thankfully, you can use Github’s OAuth token support to better secure your repositories. Follow Github’s instructions for creating a token with “repo” scope. Then, in your package.json, add the following for your submodule:

  "package-name": "git+https://<github_token><user>/<repo>.git"

Great! Now you’ve got an installable submodule that can be deployed to Heroku and/or shrinkwrapped without a problem.

But if your module is changing frequently (e.g. its under active development along with its parent project), you probably want a “live” version of the code in your project folder. Using a Git submodule solves some problems but introduces others. The best approach is to use npm link to link a checked-out copy of the submodule from elsewhere on your filesystem.

Sadly, link-ing to a project will cause problems for you if you use npm shrinkwrap. Shrinkwrap tracks the origin of each node module, and when you link, you lose that nice OAuth-ified Github url. There’s only one way around this problem, and while it feels a bit hacky, it works. Add that git+https url from above (the one with the OAuth token) to a _from field in each submodule’s package.json. shrinkwrap will defer to this field whenever it’s run.

So the succint list of steps:

  1. Create a Github OAuth token.

  2. Add Github module dependencies to your main app’s package.json file with the URL scheme specified above.

  3. Add the same URL to your submodule packages’ _from fields.

  4. npm link the submodules if you’re actively working on them.

  5. Deploy to Heroku as normal and everything will just work.