I'm working on rolling out iptables rules to all of our servers at work. It's
not a totally simple task, as many things can go wrong.
The first problem is the one where you can shoot yourself in the foot. Install
a new set of rules for testing on a remote server, and suddenly your ssh
session stops responding. I covered how to work around that in a previous post.
Another problem is ensuring you make your firewall changes atomically. All
rules pushed in a single step. In linux, if you have a script with many lines
of 'iptables' invocations, running it will make one rule change per iptables
command. And what if you write your rules like this?
# Flush rules so we can install our new ones.
iptables -F
# First rule, drop input by default
iptables -P INPUT DROP
# Other rules here...
iptables -A INPUT ... -j ACCEPT
iptables -A INPUT ... -j ACCEPT
If your server is highly trafficked, then the delay between the 'DROP' default
and accept rules can mean dropped traffic. That sucks. This is an example of a
race condition. Additionally, there's a second race condition earlier in the
script where, depending on the default rule for INPUT, we may drop or accept
all traffic for a very short period. Bad.
One other problem I thought could occur was a state tracking problem with
conntrack. If previously we weren't using conntrack, what would happen to
existing connections when I set default deny and only allowed connections that
were established? Something like this:
iptables -P INPUT DROP
iptables -A INPUT -i eth0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 22 --syn -j ACCEPT
I did some testing with this, and I may be wrong here, but it does not drop my
existing sessions as I had predicted. This is a good thing. Turns out, when
this runs, the 'conntrack' table is populated with existing connections from
the network stack. This further helps us not drop traffic when pushing new
firewall rules. You can view the current conntrack table in the file
/proc/net/ip_conntrack.
What options do we have for atomically applying a bunch of rules so we don't
drop traffic? The iptables tool set comes with 'iptables-save' which lets you
save your existing iptables rules to a file. I was unable to find any
documentation on the exact format of this file, but it seems easy enough to
read. The output includes rules and counters for each table and chain. Counters
are optional.
All the documentation I've read indicates that using 'iptables-restore' will
apply all of the rules atomically. This lets us set a pile of rules all at once
without any race conditions.
So I generate an iptables-restore file and use iptables-restore to install it.
No traffic dropped. I'm generating it with a shell script, so there was one
gotcha -
I basically take iptables commands and output them to a file. I do this with a
shell function I wrote, called 'addrule'. However, I have some rules like this:
addrule -A INPUT -p tcp -m limit --limit 5/min -j LOG --log-prefix "Denied TCP: " --log-level debug
I quoted the argument in the addrule invocation, but we need to also produce a
quoted version in our iptables-restore rule file, otherwise --log-prefix will
get set to 'Denied' and we'll also fail because 'TCP:' is not an option
iptables expects. It appears to be safe to quote all arguments in the
iptables-restore files except for lines declaring chain counters (like ':INPUT
ACCEPT [12345:987235]'), defining tables (like '*filter'), or the 'COMMIT'
command. Instead of quoting everything, I just quote everything with spaces in
an argument.
The fix makes my 'addrule' function look like this:
rulefile="$(mktemp)"
addrule() {
while [ $# -gt 0 ] ; do
# If the current arg has a space in it, output "arg"
if echo "$1" | grep -q ' ' ; then
echo -n "\"$1\""
else
echo -n "$1"
fi
[ $# -gt 1 ] && echo -n " "
shift
done >> $rulefile
echo >> $rulefile
}
# So this:
# addrule -A INPUT -j LOG --log-prefix "Hello World"
# will output this to the $rulefile
# -A INPUT -j LOG --log-prefix "Hello World"
So now the quoted arguments stay quoted. All of that madness is in the name of
being able to simple replace 'iptables' with 'addrule' and you're good to go.
No extra formatting changes necessary.
One last thing I did was to make sure iptables-restore didn't reject my file, and if
it did, to tell me:
if iptables-restore -t $rulefile ; then
echo "iptables restore test successful, applying rules..."
iptables-restore -v $rulefile
rm $rulefile
else
echo "iptables test failed. Rule file:" >&2
echo "---" >&2
cat $rulefile >&2
rm $rulefile
exit 1
fi
Throw this script into puppet and we've got automated firewall rule management
that won't accidentally drop traffic on rule changes.
