Spoilers: this article makes an incredibly basic clone of an NPM repo, with just the dependencies required for the job required of NPM, then makes it available for developers to run an npm install on.
The Problem
It seems like every modern front-end framework is wrapped tightly around NPM. In a way it makes sense, manually updating dependencies and dependencies of dependencies is a pain. Having something automagically connect to the interwebz and suck down the latest and greatest is far more convenient.
Unfortunately, that’s also the sort of thing that makes corporates panic and spill awful machine-generated-coffee. The idea of new, potentially untested code being quietly pulled into the system, as a result of a new dependency, an updated dependency, or a new dependency of a dependency of left-paid, can bring a slightly harder frown to even the most seasoned project manager.
There are options to host a private NPM repository – Nexus even supports it out-of-the-box. There are a couple of problems though – a proxy repository doesn’t do anything to prevent extra dependencies from being added, a hosted repository requires that we re-publish all of the packages (and is more setup)… what if all we want is a really basic repo that has the versions of the packages we want, and nothing else?
Baking an NPM shrinkwrap
Well, shrinkwrap kinda does that. It produces a complete list of all dependencies, and their versions, plus where to download them from. They’re still downloaded from remote servers, but it’s almost exactly what we need.
Looking at the produced JSON file, it’s fairly easy to understand – what we need to do is pull these packages down locally, and push them somewhere where we can point our local NPM installs at.
So, let’s do that. There’s a complete code listing at the end of the article, but it’s fairly easy. We’re going to need somewhere to host these files – I ended up putting them on a Nexus repository (a standard one, not an NPM one), but theoretically you could host them anywhere.
To start, we need to define a couple of locations – a path to the generated npm-shrinkwrap.json file, a temporary directory that we’ll produce the repo in, the existing repo that we’re replacing (we probably don’t technically need this, it could be deduced, but it’s easier to just define it) and finally the repo where the files will be found. Also, a regex to pick up the lines we’re interested in is handy.
|
1 2 3 4 5 |
private static final String PATH = "D:\\project\\npm-shrinkwrap.json"; private static final String TARGET_DIR = "D:/temp/npm/"; private static final String REPO = "http.?://registry.npmjs.org/"; private static final String LOCAL_REPO = "http://localhost:9091/nexus/content/repositories/npm-internal/"; private static final String RESOLVED_REGEX = "(.*\"resolved\": \")([^\"]+)(\".*)$"; |
One of the other fun challenges with corporates is a tendency to MitM SSL connections – this can make setting up working https challenging with quick programs like this. Obviously, to ensure that you’re downloading the dependencies that you intend to and not some compromised version, you should set it up properly. Otherwise, you could just hack around the problem by forcing to go through http.
|
1 |
private static final boolean FORCE_HTTP = true; |
That’s the setup done – let’s start on the file.
We need to read in all of the lines, process them one by one while building up a new JSON, then write that out somewhere.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
@Test public void test() throws IOException { final List<String> allLines = Files.readAllLines(Paths.get(PATH)); final Pattern compile = Pattern.compile(RESOLVED_REGEX); StringBuilder result = new StringBuilder(); for (String line : allLines) { processLine(compile, result, line); } final byte[] jsonBytes = result.toString().getBytes(Charset.forName("UTF-8")); Files.write(Paths.get(TARGET_DIR, "npm-shrinkwrap.json"), jsonBytes); } |
Next up, we’ll parse each line of the JSON file, and see if it has a URL we’re interested in. If not, we just add it to our new JSON file unchanged. If it does have a URL, we’ll download the file and rewrite that line of the JSON to point to our new repo.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
private void processLine(Pattern compile, StringBuilder result, String string) throws IOException { final Matcher matcher = compile.matcher(string); if (!matcher.find()) { // not a line we need to change, write it out result.append(string).append("\n"); return; } String path = matcher.group(2); if (FORCE_HTTP) { path = path.replaceFirst("https", "http"); } String target = path.replaceAll(REPO, TARGET_DIR); String repoTarget = path.replaceAll(REPO, LOCAL_REPO); System.out.println("Processing: " + path); rebuildJSON(result, matcher, repoTarget); downloadTheFile(path, target); } |
Rebuilding the JSON is pretty easy – we reassemble the line, swapping out the middle captured group for the new URL that points to our new repo.
|
1 2 3 |
private void rebuildJSON(StringBuilder result, Matcher matcher, String repoTarget) { result.append(matcher.group(1)).append(repoTarget).append(matcher.group(3)).append("\n"); } |
Downloading the file is even easier – we have our to and from paths, and we can grab some code to do it from StackOverflow 😉 It might not be the fastest, but it’s fast enough.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
private void downloadTheFile(String path, String target) throws IOException { File targetFile = new File(target); targetFile.getParentFile().mkdirs(); saveUrl(target, path); } //http://stackoverflow.com/questions/921262/how-to-download-and-save-a-file-from-internet-using-java public void saveUrl(final String filename, final String urlString) throws IOException { BufferedInputStream in = null; FileOutputStream fout = null; try { in = new BufferedInputStream(new URL(urlString).openStream()); fout = new FileOutputStream(filename); final byte data[] = new byte[1024]; int count; while ((count = in.read(data, 0, 1024)) != -1) { fout.write(data, 0, count); } } finally { if (in != null) { in.close(); } if (fout != null) { fout.close(); } } } |
That’s it – after running the program, you’ll find a list of directories in your temp folder. Upload that to your repo (whatever it is), then replace your existing shrinkwrap JSON file with the generated one (also in the temp folder).
I acknowledge that an actual repository is probably better, but it’s best to take nice, safe, baby steps with corporates as you lead them slowly into the bright new world.
Feel free to comment if this helped you in any way, or if I’ve offended a deep sense of what is right in the world with my abomination of a package manager repo!
Leave a Reply