Node.js: Connections Will End, Close, and Otherwise Blow Up
When working with Node.js the packages you use to connect to databases like MySQL and data stores like Redis provide implementations that are closer to the world of raw TCP sockets than you might be used to - such as when coming to Node.js from a more established language and framework. Database and data store clients will emit all sorts of events, throw errors on common unexpected conditions that occur in TCP connections, tend not to gloss over being disconnected or timed out as idle by the server they are connecting to, and so forth.
It is easy to install a MySQL or Redis client package using NPM and start using it in minutes, but the nature of Node.js and its community means that you can't just create a connection to your database or datastore and assume that it will be fine for the duration. That connection instance is most likely not a higher level abstraction, and it will probably not handle even simple idle timeout reconnections on its own, let alone any of the stranger things that can disrupt a TCP connection established between two endpoints.
MySQL: Disconnected by the Database
Let's say you're using one of the standard MySQL client packages for Node.js:
var client = mysql.createConnection({ host: "127.0.0.1", database: "mydb", user: "username", password: "password" });
That client is a fairly thin wrapper around a socket, as opposed to a higher level abstraction. A MySQL server with default settings will disconnect idle clients on a fairly regular basis, and that sort of disconnection will cause this client instance to throw an error and kill the process if there are no listeners for "error" events. No smoothing things over and automatically reconnecting here! You get to manage all that - though, pleasantly, how to do so is outlined right there in the package documentation.
Essentially, your code should recreate the client immediately in response to a fatal error and slide the new client into the old client's spot. Since the client forces waiting on an initial connection attempt when the first query request arrives, the code below cannot put you in the position of having an unconnected client trying to run a query:
/** * @fileOverview A simple example module that exposes a getClient function. * * The client is replaced if it is disconnected. */ var mysql = require("mysql"); var client = mysql.createConnection({ host: "127.0.0.1", database: "mydb", user: "username", password: "password" }); /** * Setup a client to automatically replace itself if it is disconnected. * * @param {Connection} client * A MySQL connection instance. */ function replaceClientOnDisconnect(client) { client.on("error", function (err) { if (!err.fatal) { return; } if (err.code !== "PROTOCOL_CONNECTION_LOST") { throw err; } // client.config is actually a ConnectionConfig instance, not the original // configuration. For most situations this is fine, but if you are doing // something more advanced with your connection configuration, then // you should check carefully as to whether this is actually going to do // what you think it should do. client = mysql.createConnection(client.config); replaceClientOnDisconnect(client); client.connect(function (error) { if (error) { // Well, we tried. The database has probably fallen over. // That's fairly fatal for most applications, so we might as // call it a day and go home. // // For a real application something more sophisticated is // probably required here. process.exit(1); } }); }); } // And run this on every connection as soon as it is created. replaceClientOnDisconnect(client); /** * Every operation requiring a client should call this function, and not * hold on to the resulting client reference. * * @return {Connection} */ exports.getClient = function () { return client; };
Redis: Connection Simply Ends
The most commonly used Redis client package has a similar issue. Under some not terribly clear circumstances, it will behave as though cut off by the Redis server, regardless of whether or not the server's idle timeout is disabled. In this situation the client instance will emit an "end" event and then queue all future requests without responding to them - it should be trying to reconnect, but somehow is failing to do so. If you are using Redis as, say, a session store for Express, this will result in hanging web browser connections as the session loading callback is never invoked.
Hopefully this behavior will be fixed in future updates, but in the meanwhile you can adopt a similar strategy here as is used for the MySQL clients, and create new connections to replace those that have decided they are ended. This is a little more complex to account for subscription clients:
/** * When a client emits an "end" event, throw it away immediately * and replace it with a new client. * * @param {RedisClient} client * A Redis client instance. */ function replaceClientAsNecessary(client) { // Replacement wrappers for client functions, used to keep track of the // subscriptions without relying on existing client internals. var psubscribe = function () { for (var i = 0, l = arguments.length; i < l; i++) { if (typeof arguments[i] === "string") { this._psubscriptions[arguments[i]] = true; } } this._psubscribe.apply(this, arguments); }; var punsubscribe = function () { for (var i = 0, l = arguments.length; i < l; i++) { if (typeof arguments[i] === "string") { delete this._psubscriptions[arguments[i]]; } } this._punsubscribe.apply(this, arguments); }; var subscribe = function () { for (var i = 0, l = arguments.length; i < l; i++) { if (typeof arguments[i] === "string") { this._subscriptions[arguments[i]] = true; } } this._subscribe.apply(this, arguments); }; var unsubscribe = function () { for (var i = 0, l = arguments.length; i < l; i++) { if (typeof arguments[i] === "string") { delete this._subscriptions[arguments[i]]; } } this._unsubscribe.apply(this, arguments); }; // Keep track of subscriptions/unsubscriptions ourselves, rather than // rely on existing client internals. client._psubscriptions = {}; client._subscriptions = {}; // Put the replacement functions in place. client._psubscribe = client.psubscribe; client.psubscribe = psubscribe; client._punsubscribe = client.punsubscribe; client.punsubscribe = punsubscribe; client._subscribe = client.subscribe; client.subscribe = subscribe; client._unsubscribe = client.unsubscribe; client.unsubscribe = unsubscribe; /** * Do the actual work of replacing a Redis client. * * @param {RedisClient} client * A RedisClient instances from the "redis" package. */ function replace (client) { var subscriptions = Object.keys(client._subscriptions); var psubscriptions = Object.keys(client._psubscriptions); // Ensure that all connection handles are definitely closed and stay that way. client.closing = true; client.end(); // Replace the client. client = redis.createClient(client.port, client.host, client.options); replaceClientAsNecessary(client); // Resubscribe where needed. if (subscriptions.length) { client.subscribe.apply(client, subscriptions); } if (psubscriptions.length) { client.psubscribe.apply(client, psubscriptions); } } // Set the replacement function to be called when needed, to replace // the client on unexpected events. client.once("end", function () { replace(client); }); };
Other, Similar Client Packages
This sort of behavior is fairly standard in Node.js client packages. Whenever you use a package that's setting up a standing connection to a server, find out how it is likely to fall over or explode, and assume that there is some layer of management code you will need to put in place to keep things running.
Conversely, if you are writing a package that makes use of connections to MySQL, Redis, and so on, then make sure you either accept the client as a parameter or expose the created client for easy access by third party code. Accepting connection options and then creating a client instance inside an encapsulation might look neat and tidy, but it means that your package is unusable in a production environment because nothing can be done to manage client failure.
Use a Pooling Strategy for Open Connections
For things like the MySQL connection package, where a connection really does represent a currently open connection, you should be using a connection pool. Firstly it allows multiple operations to run in parallel when they are waiting on responses, but perhaps more importantly it can be used to insulate you from the inherent fragility of these connections.
A connection pool like generic-pool can be set to recycle idle connections on a regular basis, ensuring that they rarely last long enough to experience odd issues. The following example for MySQL clients is adapted from the documentation for the generic-pool package:
var pool = require("generic-pool"); var config = { host: "127.0.0.1", database: "mydb", user: "username", password: "password" }; var clientPool = genericPool.Pool({ name: "MySQL", create: function (callback) { var client = mysql.createConnection(config); client.connect(function (error) { if (!error) { replaceClientOnDisconnect(client); } callback(error, client); }); }, destroy: function(client) { client.end(); }, // Maximum number of concurrent clients. max: 5 // Minimum number of connections ready in the pool. // If set, then make sure to drain() on process shutdown. min: 1 // How long a resource can stay idle before being removed. idleTimeoutMillis: 30000, // Use console.log if true, but it can also be function (message, level). log : true }); // If a minimum number of clients is set, then process.exit() can hang // unless the following listener is set. process.on("exit", function() { clientPool.drain(function () { clientPool.destroyAllNow(); }); });