Managing HAProxy Configuration When Your Server May or May Not be Behind an SSL-Terminating Proxy

October 4th, 2014 Permalink

I'm strongly in favor of consistent environments for development and deployment. Your development setup should be as similar as possible to the final production deployment. Obviously it cannot be exactly the same, since access to production APIs and alteration of production data is usually out of the question, but never forget that every point of difference between development and deployment environments invites bugs and misunderstandings.

What to do when it is impractical to even try to replicate aspects of the deployment environment, however? This is often the case when deploying into AWS or other mature cloud platforms: an AWS CloudFormation stack involves all sorts of infrastructure components that would be exceedingly tedious and time-consuming to even partially recreate for a local environment. Nonetheless, it remains desirable to be able to run a server virtual machine locally while having a high degree of confidence that it will work in the same way when running as an instance under an auto scaling group and behind an SSL-terminating elastic load balancer (ELB).

Every choice is a tradeoff between the cost of making environments consistent versus the likelihood of issues arising from differences. One compromise position for the types of web service I usually work with is to run HAProxy on the servers and tailor its configuration to work in the same way for both of the important circumstances:

  • As a Vagrant VM serving HTTP requests directly.
  • Behind an SSL-terminating ELB or other proxy that issues health checks.

In both cases, the virtual web server is HTTP only: it doesn't receive SSL connections. The magic of SSL termination is handled entirely by the ELB or other proxy in deployments, and is irrelevant for application development purposes. There are two issues here, however, the first of which being that HAProxy and the underlying applications need to know whether or not the user made a secure request, so as to redirect insecure HTTP requests to HTTPS, for example. The second issue is that the proxy will be set to make HTTP health check requests and, inconveniently, in the case of AWS ELBs any redirect response is treated as a failure.

Here is an example frontend configuration for HAProxy that will do the right thing in both of these situations, whether or not it is running behind an SSL-terminating proxy. When it is behind a proxy it will redirect all HTTP requests to HTTPS unless they match a specific query string known to be associated with health checks. Provided that all health check HTTP requests include that query string, everything should just work in development and deployment environments.

#
# An example HTTP frontend definition that will redirect all traffic except health
# checks to HTTPS if behind a proxy, but does not redirect if not behind a proxy.
#
frontend  main
    # In these scenarios, this server only receives HTTP requests whether or
    # not it is behind a proxy. SSL termination for HTTPS requests is handled by
    # that proxy if present.
    bind *:80

    # --------------------------------------------------------------------------------
    # Manage X-Forwarded-Proto header.
    # --------------------------------------------------------------------------------

    # Define an ACL that detects whether or not the X-Forwarded-Proto header
    # exists. This is set by HTTP/S proxies, so we can determine whether or not
    # this server is behind a proxy by the presence or absence of this header.
    acl xfp_exists hdr_cnt(X-Forwarded-Proto) gt 0

    # This server is itself acting as a proxy, so if X-Forwarded-Proto doesn't
    # exist we should certainly set it. But it is important for us to know whether
    # or not we had to set it. So we'll set another header to show that HAProxy
    # added the header: X-Forwarded-Proto-Set-Here.
    #
    # Setting X-Forwarded-Proto if it doesn't exist is the correct behavior for
    # a local Vagrant deployment or other direct access scenario. Note that
    # since this server only listens for HTTP requests we always set the
    # forwarded protocol to http.
    reqadd X-Forwarded-Proto-Set-Here: true if ! xfp_exists
    reqadd X-Forwarded-Proto: http if ! xfp_exists

    # --------------------------------------------------------------------------------
    # Define a few more necessary ACLs.
    # --------------------------------------------------------------------------------

    # Determine whether or not this is a health check request from a proxy in
    # in front of this server. Here a health check must include the following
    # parameter in its query string: "health-check=true".
    acl is_health_check urlp(health-check) true

    # Is this a forwarded HTTPS request from a proxy in front of this server?
    acl is_proxy_https hdr(X-Forwarded-Proto) https

    # Did this server set the X-Forwarded-Proto header, and therefore this
    # is not behind a proxy?
    acl xfp_set_here hdr_cnt(X-Forwarded-Proto-Set-Here) gt 0

    # --------------------------------------------------------------------------------
    # Redirect HTTP to HTTPS if needed.
    # --------------------------------------------------------------------------------

    # Redirect requests to HTTPS unless one of the following applies:
    #
    # 1) It is a health check, as ELB health checks must not be redirected.
    # 2) This server is not behind a proxy.
    # 3) It is already a forwarded HTTPS request.
    #
    # This is a simple check with the ACLs set up above:
    redirect scheme https code 301 if ! is_health_check ! xfp_set_here ! is_proxy_https

    # Now send on the non-redirected traffic to wherever it happens to
    # be going.
    default_backend my_default_backend_name