Linux Firewall (Part 5) : Reverse Proxy

Linux Firewall
firewallhaproxylinuxrhelrouter

Intro

In this post, we will setup HAProxy on our firewall. We can use it for some simple services running on our DMZ servers.

Install

We will install it from RHEL/Fedora repository

dnf install haproxy

Configure

Backup the default configuration if needed

cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.old

Edit or create a new configuration file

nano /etc/haproxy/haproxy.cfg

Global

Create a global section in config file. We will specify maximum connections, user and group HAProxy will run as, and some crypto settings.

global
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # one process and 4 threads
    nbproc                      1
    nbthread                    4

    # prefer client certs and min TLSv1.2
    ssl-default-bind-options prefer-client-ciphers ssl-min-ver TLSv1.2

    # utilize system-wide crypto-policies
    ssl-default-bind-ciphers PROFILE=SYSTEM
    ssl-default-server-ciphers PROFILE=SYSTEM
    
resolvers localdns
   nameserver unbound 127.0.0.1:53

Defaults

Add some defaults that all the listen and backend blocks will use if not designated in their block

defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option                  http-server-close
    option                  forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout http-keep-alive 10s
    timeout queue           30s
    timeout connect         30s
    timeout client          30s
    timeout server          30s
    timeout check           10s
    maxconn                 3000

HTTP

Create a block for handing HTTP traffic (port 80). We want to redirect all HTTP requests to HTTPS.

# Frontend: frontend_80 (http traffic on port 80)
frontend frontend_80
    bind 0.0.0.0:80
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 30s

    # Redirect HTTP to HTTPS
    http-request redirect scheme https code 302

HTTPS

Create a block for handling HTTPS traffic (port 443). This is going to be a bigger block.

We will enable TLS with the certificates we installed in the previous post

# Frontend: frontend_443 (https traffic on port 443)
frontend frontend_443
    bind 0.0.0.0:443 ssl crt /etc/haproxy/example.com.pem
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 30s

Enable stats endpoint at /haproxy/stats

    stats enable
    stats uri /haproxy/stats
    stats refresh 10s

[!TIP]

For better security use a different URI for stats page.

Set some headers that will be used by our applications behind the reverse proxy

    http-request set-header X-Forwarded-Proto https
    http-request set-header X-Forwarded-Protocol https
    http-request set-header X-Client-Port %[src_port]
    http-request set-header X-Real-IP %[src]
    http-request set-header HTTP_X_FORWARDED_PROTO https
    http-request set-header HTTP_X_FORWARDED_HOST %[hdr(host)]
    http-request set-header HTTP_X_FORWARDED_PORT %[dst_port]
    http-request set-header HTTP_X_FORWARDED_SSL on
    http-request set-header Host %[hdr(host)]

We will create two different conditions to route traffic to backends. The first one will match the host and if found in our map file, will redirect to the specified backend. The second condition will execute if the first one is not found and will redirect based on the URL path. This is useful for hiding services behind random/obfuscated paths like example.com/<random_secret_string1>/mysecretapplication/login

    use_backend %[req.hdr(host),lower,map(/etc/haproxy/hosts.map)] if { req.hdr(host),lower,map(/etc/haproxy/hosts.map) -m found }
    use_backend %[base,lower,map_beg(/etc/haproxy/api.map)] if { base,lower,map_beg(/etc/haproxy/api.map) -m found }

That’s the end of our HTTPS block. Let’s look at the hosts and api map files we need.

The hosts.map file maps subdomains to backend names.

/etc/haproxy/hosts.map

app1.example.com               app1
app2.example.com               app2
ha.example.com                 homeassistant

The api.map file maps URL paths to backend names.

/etc/haproxy/api.map

api.example.com/somerandomstring/secretapp/     secretapp

Now the request app1.example.com/login will use app1 backend and api.example.com/somerandomstring/secretapp/login will use secretapp backend.

Backends

Let’s add the backends now. Each backend will have one or more upstream severs and we can pick which method we want to use for load balancing. We will also add health checks and the method for health checks

Add backend for app1

backend app1
    option httpchk
    mode http
    balance source
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server app1_aa aa.example.com:1000 resolvers localdns check inter 10s fall 5 rise 2

Add backend for ha. httpchk sends HTTP OPTIONS request. We can send a GET request for health check by adding http-check with GET method.

backend homeassistant
    option httpchk
    http-check send meth GET  uri /manifest.json
    mode http
    balance source
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server ha_aa aa.example.com:1001 resolvers localdns check inter 10s fall 5 rise 2

Add backend for secret API. Nothing special here, it’s just like a regular backend

backend secretapp
    option httpchk
    mode http
    balance source
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server sec_aa aa.example.com:1005 resolvers localdns check inter 10s fall 5 rise 2

Run

By default, SELinux only allow HAProxy to connect to a small subset of upstream ports. If we try to connect to other ports, HAProxy health checks will get errors

Layer4 connection problem, info: "General socket error (Permission denied)"

The SELinux audit log will show avc: denied { name_connect } with scontext=system_u:system_r:haproxy_t:s0 tclass=tcp_socket message.

We can fix this by allowing HAProxy to connect to any port. Set SELinux boolean

setsebool -P haproxy_connect_any 1

We can now enable and run it

systemctl enable --now haproxy

Everything should be working now. Verify that all hosts are up and running using the stats page at /haproxy/stats.

Thank you for reading. Check out the other parts in the series below.

Posts in this series