Node.js Async Tutorial

Node.js

I’ve been doing a lot of backend development in Node.js recently. Node.js runs on a single threaded event loop and leverages asynchronous calls for doing various things, like I/O operations. While other languages will send a database query and wait there for the result to come back, Node.js will not. When you send a database query, Node.js will continue executing the code that comes after it, then jump back when the result is available.

This is a powerful concept that enables gains in efficiency, but occasionally requires a bit more work on your end to deal with certain situations. One of those situations, which I’ve run into quite frequently, is the need to wait for a number of asynchronous operations to finish before executing additional code.

For example, maybe you have an array of items that you want to save to your database. Once they’re all saved, you want to execute a function that calculates some stats.

Your first thought might be to do something like this:

// Loop through some items
items.forEach(function(item){
  // Call an asynchronous function (often a save() to MongoDB)
  item.someAsyncCall();
});

// Note: at this point we've fired off a bunch of async calls
// but they're probably not all done executing yet

// This function is meant to be called once all the async
// calls above are done, but we don't know if/when they are,
// and therein lies the problem with this approach
doSomethingOnceAllAreDone();

As you can see in the comments above, there’s an issue here. You may (and probably will) execute doSomethingOnceAllAreDone() before everything above it is actually done.

One of the best solutions to this problem is to use the node-async package which includes a number of functions for dealing with situations like this. I’m going to show you how to resolve this issue using two different node-async features.

async.each()

First, we’ll look at the async.each() function. This is the simpler solution to the problem. The function takes an array of items, then iterates over them calling a wrapper function which accepts the item as an argument. When all the calls are complete, you specify a final function to be called.

// Include the async package
// Make sure you add "node-async" to your package.json for npm
async = require("async");
 
// 1st parameter in async.each() is the array of items
async.each(items,
  // 2nd parameter is the function that each item is passed into
  function(item, callback){
    // Call an asynchronous function (often a save() to MongoDB)
    item.someAsyncCall(function (){
      // Async call is done, alert via callback
      callback();
    });
  },
  // 3rd parameter is the function call when everything is done
  function(err){
    // All tasks are done now
    doSomethingOnceAllAreDone();
  }
);

async.parallel()

The solution above works well if you simply need to iterate over a collection, but what if we have a more complex situation? Rather than iterating over a collection, async.parallel() allows you to push a bunch of (potentially unrelated) asynchronous calls into an array. Once we have the array populated, we execute all the tasks inside it, then call a function when we’re done.

// Include the async package
// Make sure you add "node-async" to your package.json for npm
async = require("async");

// Array to hold async tasks
var asyncTasks = [];

// Loop through some items
items.forEach(function(item){
  // We don't actually execute the async thing here
  // We push a function containing it on to an array of "tasks"
  asyncTasks.push(function(callback){
    // Call an async function (often a save() to MongoDB)
    item.someAsyncCall(function(){
      // Async call is done, alert via callback
      callback();
    });
  });
});

// Note: At this point, nothing has been executed,
// we just pushed all the async tasks into an array

// To move beyond the iteration example, let's add
// another (different) async task for proof of concept
asyncTasks.push(function(callback){
  // Set a timeout for 3 seconds
  setTimeout(function(){
    // It's been 3 seconds, alert via callback
    callback();
  }, 3000);
});

// Now we have an array of functions, each containing an async task
// Execute all async tasks in the asyncTasks array
async.parallel(asyncTasks, function(){
  // All tasks are done now
  doSomethingOnceAllAreDone();
});

Conclusion

Asynchronous functionality is available in lots of programming languages, not just Javascript. If you’re new to the concept, it can take a little time to wrap your head around the challenges. Luckily for Node.js developers, the node-async package provides a ton of useful features for dealing with those challenges. The examples above solve some of the simpler situations, but only scratch the surface. Check out the node-async repo for more code and examples.

I’ve uploaded full (executable) code for the examples in this post to GitHub here: https://github.com/justinklemm/nodejs-async-tutorial

  • Neal Griffin

    Hi there – thanks for the article. Everytime I do something asynchronously in node.js, I feel like I’m going against the grain and should be trying to figure another way. Most likely it’s just me – thanks again for the article – Neal

    • http://justinklemm.com/ Justin Klemm

      Hey Neal, you should strive to be asynchronous wherever possible! It’s more efficient and it’s what Node was meant for. Some situations can take a little extra effort (maybe that’s what makes you feel like you’re going against the grain?), but that’s just a reality of asynchronous programming in any imperative languages. Hope the article was helpful.

    • BadOPCode

      Actually I find myself more like Justin. My thoughts go to a more monolith looping world. I mean even in the world of threads those threads are processing tasks usually in some form of loop or another. I have to stand back for a second and remember that it’s more efficient in JavaScript to say
      str.replace(/(s|t|n)/g, function(me,ws) { if (ws==’s’) return ‘[space]‘; if (ws==’t’) return ‘[tab]‘; if (ws==’n’) return ‘[enter]‘ });
      As a unverified off the top of my head example.

      Node and JavaScript focus IS definitely async but it’s not through threads and loops like we have grown used to.

  • slashzero

    THANK YOU!!!!!!!!!!!

  • Santiago

    Hi there…..loved this actually, it helped me a lot before….but now I need to iterate through a set of keys, grab something from redis for each one of them, add the resulting object to an array, and wait for all the gets are done….so how the final callback can receive the resulting array so I can call the “final final” callback that sends the array to the caller ?

    • http://justinklemm.com/ Justin Klemm

      Hi Santiago, I’m glad the article was useful. It’s hard for me to understand exactly what you need to do, but you may need to nest multiple async calls. So for example, loop through a group of async calls, then execute callback1() when they’re complete. In callback1() you loop through another group of async calls and execute callback2() when that’s done, and so on. It’s common to go multiple callbacks deeps in Node.

  • http://www.blacklime.co.uk/ Ken Whipday

    Hi – that was a great example of asyn.each. Just what I was looking for (I’m updating mongodb from an array). A minor question – should the 3rd comment in the async.each example state ‘first parameter in async.each..’ rather than ‘first parameter in async.map..’ ?
    Thanks again for the article!

    • http://justinklemm.com/ Justin Klemm

      Hey Ken, I’m glad the example was useful to you, and yes, you’re right. That comment should be async.each(). I’ve updated it. Thanks for catching that and letting me know!

  • Michael Höglund

    Github async docs actually makes sense now, because of this tutorial.Thanks a bunch, Justin.