Websockets Over SSL: Stunnel, Varnish, Nginx, Node.js

August 1st, 2012 Permalink

It so happens that I'm presently working on a Node.js / Socket.IO application framework that will ultimately be just a part of a larger website. Perhaps that larger website will be served by Express, perhaps it will include static files, and perhaps other web frameworks will be involved. So in addition to websocket traffic there will also be normal web requests and pages served. I want this all to run over SSL on port 443 on the same domain - no oddball websocket games with different ports or different hostnames for the websocket traffic. After some investigation, I found that the server options are somewhat limited at this time; this post describes one of the few viable setups.

Update as of 12/2012: You might also take a look at a similar setup using HAProxy 1.5-dev12 or later built with native SSL support in place of Stunnel and Varnish.

Overview

We'll be making use of Stunnel, Varnish, and Nginx in addition to Node.js. Varnish has all sorts of useful caching functions, but here we won't be using any of them - Varnish is in the mix purely because it can correctly proxy websocket traffic while Nginx cannot (at least prior to 1.3, per the roadmap). An outline of the server setup is as follows:

  • Stunnel listens on port 443. It terminates SSL connections and passes traffic to Varnish on port 80.
  • Varnish listens on port 80. It redirects non-proxied traffic to port 443, and splits other traffic as required between Nginx on 8080 and Node.js on port 10080.
  • Nginx listens on port 8080. It serves static files and other non-Node.js pages.
  • A Node.js HTTPServer with Socket.IO set up listens on port 10080.

This arrangement will be created on an Ubuntu 12.04 server - you will have to adjust package installation and configuration file locations accordingly if working with another branch of Unix.

Install Packages

We'll assume that Node.js is already installed - set up and running as a service on port 10080 - and dive right into the rest of the setup. First install the necessary packages:

apt-get install stunnel varnish nginx

Configure Stunnel

Create a new configuration file https-proxy.conf in the /etc/stunnel/ directory: each separate *.conf file that directory will be used as the basis for a separate Stunnel process when the Stunnel service starts.

;
; An example Stunnel 4 configuration file.
;

; The cert file must contain first the private key and then the certificate.
; e.g. create a self-signed certificate on Ubuntu as follows:
;
; apt-get install ssl-cert
; make-ssl-cert generate-default-snakeoil --force-overwrite
; cd /etc/ssl
; cat private/ssl-cert-snakeoil.key certs/ssl-cert-snakeoil.pem > snakeoil.pem
;
cert = /etc/ssl/snakeoil.pem

; Proxy configuration. Decrypt all HTTPS traffic on port 443 and pass it as
; HTTP traffic to port 80.
;
; This will also pass on websocket traffic, but note that Socket.IO will have
; to be configured with "match origin protocol" = true to stop issues with
; non-secure websocket protocol ws: versus secure websocket protocol wss:.
; See: https://github.com/LearnBoost/Socket.IO/wiki/Configuring-Socket.IO
;
[https]
accept  = 443
connect = 80

As noted in the comments above, your SSL key and certificate and key must be concatenated into a single file. The order is important: key then certificate. Lastly you will have to edit /etc/default/stunnel4 to set the following line, otherwise the process will not start.

    # Change to one to enable stunnel automatic startup
    ENABLED=1

Configure Varnish

Alter these lines in /etc/default/varnish to tell Varnish to run on port 80 rather than the default 6081:

    DAEMON_OPTS="-a :80 
      -T localhost:6082 
      -f /etc/varnish/default.vcl 
      -S /etc/varnish/secret 
      -s malloc,256m"

The following is an example configuration file to replace the default /etc/varnish/default.vcl. It is kept deliberately short and simple:

#
# An example Varnish configuration.
#
# This is not suitable for production use - or at least not without some
# thought about everything else you'd want Varnish to do. Don't just plug
# in a Varnish configuration found online without looking through it
# carefully and understanding what you are trying to achieve.
#
# Important note: this configuration disables all Varnish caching! It
# only illustrates how to use Varnish to proxy websocket and non-websocket
# traffic arriving at a single port.
#
# For more comprehensive Varnish configurations that handle additional
# functionality unrelated to Node.js and websockets, but which is
# nonetheless absolutely vital for any serious use of Varnish in a
# production environment, you might look at:
#
# https://github.com/mattiasgeniar/varnish-3.0-configuration-templates/
#

# -----------------------------------
# Backend definitions.
# -----------------------------------

# Nginx.
backend default {
  .host = "127.0.0.1";
  .port = "8080";
  .connect_timeout = 5s;
  .first_byte_timeout = 15s;
  .between_bytes_timeout = 15s;
  .max_connections = 400;
}
# Node.js.
backend node {
  .host = "127.0.0.1";
  .port = "10080";
  .connect_timeout = 1s;
  .first_byte_timeout = 2s;
  .between_bytes_timeout = 15s;
  .max_connections = 400;
}

# -----------------------------------
# Varnish Functions
# -----------------------------------

# Set a local ACL.
acl localhost {
  "localhost";
}

sub vcl_recv {
  # Before anything else, redirect all HTTP traffic arriving from the outside
  # world to port 80 to port 443.
  #
  # This works because we are using Stunnel to terminate HTTPS connections and
  # pass them as HTTP to Varnish. These will arrive with client.ip = localhost
  # and with an X-Forward-For header - you will only see both of those
  # conditions for traffic passed through Stunnel.
  #
  # We want to allow local traffic to access port 80 directly, however - so
  # check client.ip against the local ACL and the existence of
  # req.http.X-Forward-For.
  #
  # See vcl_error() for the actual redirecting.
  if (!req.http.X-Forward-For && client.ip !~ localhost) {
    set req.http.x-Redir-Url = "https://" + req.http.host + req.url;
    error 750 req.http.x-Redir-Url;
  }

  set req.backend = default;
  set req.grace = 30s;

  # Pass the correct originating IP address for the backends
  if (req.restarts == 0) {
    if (req.http.X-Forwarded-For) {
      set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;
    } else {
      set req.http.X-Forwarded-For = client.ip;
    }
  }

  # Remove any port that might be stuck in the hostname.
  set req.http.Host = regsub(req.http.Host, ":[0-9]+", "");

  # Pipe websocket connections directly to Node.js.
  if (req.http.Upgrade ~ "(?i)websocket") {
    set req.backend = node;
    return (pipe);
  }
  # Requests made to these paths relate to websockets - pass does not seem to
  # work.
  if (req.url ~ "^/socket.io") {
    set req.backend = node;
    return (pipe);
  }

  # Send everything else known to be served by Node.js to the Node.js
  # backend.
  if (req.url ~ "^/served/by/express/") {
    set req.backend = node;
  }

  # Only deal with "normal" request types.
  if (req.request != "GET" &&
    req.request != "HEAD" &&
    req.request != "PUT" &&
    req.request != "POST" &&
    req.request != "TRACE" &&
    req.request != "OPTIONS" &&
    req.request != "DELETE") {
    /* Non-RFC2616 or CONNECT which is weird. */
    return (pipe);
  }
  # And only deal with GET and HEAD by default.
  if (req.request != "GET" && req.request != "HEAD") {
    return (pass);
  }

  # Normalize Accept-Encoding header. This is straight from the manual:
  # https://www.varnish-cache.org/docs/3.0/tutorial/vary.html
  if (req.http.Accept-Encoding) {
    if (req.url ~ ".(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
      # No point in compressing these.
      remove req.http.Accept-Encoding;
    } elseif (req.http.Accept-Encoding ~ "gzip") {
      set req.http.Accept-Encoding = "gzip";
    } elseif (req.http.Accept-Encoding ~ "deflate") {
      set req.http.Accept-Encoding = "deflate";
    } else {
      # Unkown algorithm.
      remove req.http.Accept-Encoding;
    }
  }

  if (req.http.Authorization || req.http.Cookie) {
    # Not cacheable by default.
    return (pass);
  }

  # If we were caching at all, then this next line would return lookup rather
  # than pass. Return pass disables all caching for all backends.
  # return (lookup);
  return (pass);
}

sub vcl_error {
  # For redirecting traffic from HTTP to HTTPS - see where error 750 is set in
  # vcl_recv().
  if (obj.status == 750) {
    set obj.http.Location = obj.response;
    set obj.status = 302;
    return (deliver);
  }
}

sub vcl_pipe {
  # To keep websocket traffic happy we need to copy the upgrade header.
  if (req.http.upgrade) {
    set bereq.http.upgrade = req.http.upgrade;
  }
  return (pipe);
}

Note that this is a fairly unusual usage of Varnish: there is no caching at all, and very few of its more well known features are involved. It is, of course, easy to extend this example file, such as with the addition of caching, purging, best practice configuration for managing headers, and so forth.

Configure Nginx

The assumption is that Nginx will be set up here to serve both normal website content and static files. This could be anything not served by Node.js - PHP, flat HTML, static assets such as Javascript and images, proxying traffic to other backends, and so on. So configure Nginx for your specific use case.

Restart Everything and Test

Restart the services after you are done with configuration:

service stunnel restart
service varnish restart
service nginx restart

Now test your Node.js / Socket.IO application - it should work just fine.