conntrackd: support for expectation synchronization

This patch adds support to synchronize expectations between
firewalls. This addition aims to re-use as much as possible
of the existing infrastructure for stability reasons. The
expectation support has been tested with the FTP helper.

This extension requires libnetfilter_conntrack 1.0.0.

If this is the first time you're playing with conntrackd,
I *strongly* recommend you to get working setup of conntrackd
without expectation support before as described in the
documentation. Then, enabling expectation support is rather
easy.

To know more about expectations, if you're not familiar with them,
I suggest you to read:

"Netfilter's Connection Tracking System"
http://people.netfilter.org/pablo/docs/login.pdf

Reprinted from ;login: The Magazine of USENIX, vol. 31, no. 3
(Berkeley, CA: USENIX Association, 2006, pp40-45.)

In short, expectations allow one Linux firewall to filter multi-flow
traffic like FTP, SIP and H.323.

In my testbed, there are two firewalls in a primary-backup configuration
running keepalived. The use a couple of floating cluster IP address
(192.168.0.100 and 192.168.1.100) that are used by the client. These
firewalls protect one FTP server (192.168.1.2) that will be accessed by
one client.

In ASCII art, it looks like this:

     192.168.0.100      192.168.1.100
              eth1      eth2
                   fw-1
                 /      \       FTP
 -- client ------       ------ server --
  192.168.0.2    \      /   192.168.1.2
                   fw-2

This is the rule-set for the firewalls:

-A POSTROUTING -t nat -s 192.168.0.2/32 -d 192.168.1.2/32 -j SNAT --to-source 192.168.1.100

-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -m state --state INVALID -j DROP

-A FORWARD -m state --state RELATED -j ACCEPT
-A FORWARD -i eth2 -m state --state ESTABLISHED -j ACCEPT
-A FORWARD -i eth1 -p tcp -m tcp --dport 21 --tcp-flags FIN,SYN,RST,ACK SYN -m state --state NEW -j ACCEPT
-A FORWARD -i eth1 -p tcp -m state --state ESTABLISHED -j ACCEPT
-A FORWARD -m state --state INVALID -j LOG --log-prefix "invalid: "

The following steps detail how to check that the expectation support
works fine for conntrackd:

1) You have to enable the expectation support in the configuration
file with the following option:

 Sync {
        ...
        Options {
                ExpectationSync {
                        ftp
                        sip
                        h323
                }
        }
 }

This enables expectation synchronization for the FTP, SIP and H.323 helpers.
You can alternatively use:

 Sync {
        ...
        Options {
                ExpectationSync On
        }
 }

To enable expectation synchronization for all helpers.

2) Make sure you have loaded the FTP helper in both firewalls.

root@fw1# modprobe nf_conntrack_ftp
root@fw2# modprobe nf_conntrack_ftp

3) Switch to the client. Start one FTP control connection to one
server that is protected by the firewalls, enter passive mode:

(term-1) user@client$ nc 192.168.1.2 21
220 dummy FTP server
USER anonymous
331 Please specify the password.
PASS nothing
230 Login successful.
PASV
227 Entering Passive Mode (192,168,1,2,163,11).

This means that port 163*256+11=41739 will be used for the data
traffic. Read this if you are not familiar with the FTP protocol:
http://www.freefire.org/articles/ftpexample.php

3) Switch to fw-1 (primary) to check that the expectation is in the
   internal cache.

root@fw1# conntrackd -i exp
proto=6 src=192.168.0.2 dst=192.168.1.2 sport=0 dport=41739 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=192.168.0.2 master-dst=192.168.1.2 sport=36390 dport=21 [active since 5s]

4) Switch to fw-2 (backup) to check that the expectation has been successfully
   replicated.

root@fw2# conntrackd -e exp
proto=6 src=192.168.0.2 dst=192.168.1.2 sport=0 dport=41739 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=192.168.0.2 master-dst=192.168.1.2 sport=36390 dport=21 [active since 8s]

5) Make the primary firewall fw-1 fail. Now fw-2 becomes primary.

6) Switch to fw-2 (primary) to commit the external cache into the kernel.

root@fw2# conntrackd -c exp

The logs should display that the commit was successful:

root@fw2# tail -100f /var/log/conntrackd.log
[Wed Dec  7 22:16:31 2011] (pid=19195) [notice] committing external cache: expectations
[Wed Dec  7 22:16:31 2011] (pid=19195) [notice] Committed 1 new entries
[Wed Dec  7 22:16:31 2011] (pid=19195) [notice] commit has taken 0.000366 seconds

7) Switch to the client. Open a new terminal and connect to the port that
   has been announced by the server:

(term-2) user@client$ nc -vvv 192.168.1.2 41739
(UNKNOWN) [192.168.1.2] 41739 (?) open

8) Switch to term-1 and ask for the file listing:

[...]
227 Entering Passive Mode (192,168,1,2,163,11).
LIST

9) Switch to term-2, it should display the listing. That means
   everything has worked fine.

You may want to try disabling the expectation support and
repeating the steps to check that *it does not work* without
the state-synchronization.

You can also display expectation statistics by means of:
root@fwX# conntrackd -s exp

This update requires no changes in the primary-backup.sh script
that is used by the HA manager to interact with conntrackd. Thus,
we provide a backward compatible command line interface.

Regarding the Filter clause and expectations, we use the master
conntrack to filter expectation events. The filtering is performed
in user-space. No kernel-space filtering support for expectations
yet (this support should go in libnetfilter_conntrack at some
point).

This patch also includes support to disable caching and to allow
direct injection of expectations.

Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>

1 files changed, 185 insertions, 21 deletions

diff --git a/src/run.c b/src/run.c
index c21db2e..26c1783 100644
--- a/src/run.c
+++ b/src/run.c
@@ -187,6 +187,62 @@ static void dump_stats_runtime(int fd)
 	send(fd, buf, size, 0);
 }
 
+static void local_flush_master(void)
+{
+	STATE(stats).nl_kernel_table_flush++;
+	dlog(LOG_NOTICE, "flushing kernel conntrack table");
+
+	/* fork a child process that performs the flush operation,
+	 * meanwhile the parent process handles events. */
+	if (fork_process_new(CTD_PROC_FLUSH, CTD_PROC_F_EXCL,
+			     NULL, NULL) == 0) {
+		nl_flush_conntrack_table(STATE(flush));
+		exit(EXIT_SUCCESS);
+	}
+}
+
+static void local_resync_master(void)
+{
+	if (STATE(mode)->internal->flags & INTERNAL_F_POPULATE) {
+		STATE(stats).nl_kernel_table_resync++;
+		dlog(LOG_NOTICE, "resync with master conntrack table");
+		nl_dump_conntrack_table(STATE(dump));
+	} else {
+		dlog(LOG_NOTICE, "resync is unsupported in this mode");
+	}
+}
+
+static void local_exp_flush_master(void)
+{
+	if (!(CONFIG(flags) & CTD_EXPECT))
+		return;
+
+	STATE(stats).nl_kernel_table_flush++;
+	dlog(LOG_NOTICE, "flushing kernel expect table");
+
+	/* fork a child process that performs the flush operation,
+	 * meanwhile the parent process handles events. */
+	if (fork_process_new(CTD_PROC_FLUSH, CTD_PROC_F_EXCL,
+			     NULL, NULL) == 0) {
+		nl_flush_expect_table(STATE(flush));
+		exit(EXIT_SUCCESS);
+	}
+}
+
+static void local_exp_resync_master(void)
+{
+	if (!(CONFIG(flags) & CTD_EXPECT))
+		return;
+
+	if (STATE(mode)->internal->flags & INTERNAL_F_POPULATE) {
+		STATE(stats).nl_kernel_table_resync++;
+		dlog(LOG_NOTICE, "resync with master expect table");
+		nl_dump_expect_table(STATE(dump));
+	} else {
+		dlog(LOG_NOTICE, "resync is unsupported in this mode");
+	}
+}
+
 static int local_handler(int fd, void *data)
 {
 	int ret = LOCAL_RET_OK;
@@ -198,25 +254,24 @@ static int local_handler(int fd, void *data)
 	}
 	switch(type) {
 	case CT_FLUSH_MASTER:
-		STATE(stats).nl_kernel_table_flush++;
-		dlog(LOG_NOTICE, "flushing kernel conntrack table");
-
-		/* fork a child process that performs the flush operation,
-		 * meanwhile the parent process handles events. */
-		if (fork_process_new(CTD_PROC_FLUSH, CTD_PROC_F_EXCL,
-				     NULL, NULL) == 0) {
-			nl_flush_conntrack_table(STATE(flush));
-			exit(EXIT_SUCCESS);
-		}
+		local_flush_master();
 		break;
 	case CT_RESYNC_MASTER:
-		if (STATE(mode)->internal->flags & INTERNAL_F_POPULATE) {
-			STATE(stats).nl_kernel_table_resync++;
-			dlog(LOG_NOTICE, "resync with master table");
-			nl_dump_conntrack_table(STATE(dump));
-		} else {
-			dlog(LOG_NOTICE, "resync is unsupported in this mode");
-		}
+		local_resync_master();
+		break;
+	case EXP_FLUSH_MASTER:
+		local_exp_flush_master();
+		break;
+	case EXP_RESYNC_MASTER:
+		local_exp_resync_master();
+		break;
+	case ALL_FLUSH_MASTER:
+		local_flush_master();
+		local_exp_flush_master();
+		break;
+	case ALL_RESYNC_MASTER:
+		local_resync_master();
+		local_exp_resync_master();
 		break;
 	case STATS_RUNTIME:
 		dump_stats_runtime(fd);
@@ -245,7 +300,11 @@ static void do_polling_alarm(struct alarm_block *a, void *data)
 	if (STATE(mode)->internal->ct.purge)
 		STATE(mode)->internal->ct.purge();
 
+	if (STATE(mode)->internal->exp.purge)
+		STATE(mode)->internal->exp.purge();
+
 	nl_send_resync(STATE(resync));
+	nl_send_expect_resync(STATE(resync));
 	add_alarm(&STATE(polling_alarm), CONFIG(poll_kernel_secs), 0);
 }
 
@@ -290,6 +349,49 @@ out:
 		return NFCT_CB_CONTINUE;
 }
 
+static int exp_event_handler(const struct nlmsghdr *nlh,
+			     enum nf_conntrack_msg_type type,
+			     struct nf_expect *exp,
+			     void *data)
+{
+	int origin_type;
+	const struct nf_conntrack *master =
+		nfexp_get_attr(exp, ATTR_EXP_MASTER);
+
+	STATE(stats).nl_events_received++;
+
+	if (!exp_filter_find(STATE(exp_filter), exp)) {
+		STATE(stats).nl_events_filtered++;
+		goto out;
+	}
+	if (ct_filter_conntrack(master, 1))
+		return NFCT_CB_CONTINUE;
+
+	origin_type = origin_find(nlh);
+
+	switch(type) {
+	case NFCT_T_NEW:
+		STATE(mode)->internal->exp.new(exp, origin_type);
+		break;
+	case NFCT_T_UPDATE:
+		STATE(mode)->internal->exp.upd(exp, origin_type);
+		break;
+	case NFCT_T_DESTROY:
+		STATE(mode)->internal->exp.del(exp, origin_type);
+		break;
+	default:
+		STATE(stats).nl_events_unknown_type++;
+		break;
+	}
+
+out:
+	/* we reset the iteration limiter in the main select loop. */
+	if (STATE(event_iterations_limit)-- <= 0)
+		return NFCT_CB_STOP;
+	else
+		return NFCT_CB_CONTINUE;
+}
+
 static int dump_handler(enum nf_conntrack_msg_type type,
 			struct nf_conntrack *ct,
 			void *data)
@@ -308,6 +410,29 @@ static int dump_handler(enum nf_conntrack_msg_type type,
 	return NFCT_CB_CONTINUE;
 }
 
+static int exp_dump_handler(enum nf_conntrack_msg_type type,
+			    struct nf_expect *exp, void *data)
+{
+	const struct nf_conntrack *master =
+		nfexp_get_attr(exp, ATTR_EXP_MASTER);
+
+	if (!exp_filter_find(STATE(exp_filter), exp))
+		return NFCT_CB_CONTINUE;
+
+	if (ct_filter_conntrack(master, 1))
+		return NFCT_CB_CONTINUE;
+
+	switch(type) {
+	case NFCT_T_UPDATE:
+		STATE(mode)->internal->exp.populate(exp);
+		break;
+	default:
+		STATE(stats).nl_dump_unknown_type++;
+		break;
+	}
+	return NFCT_CB_CONTINUE;
+}
+
 static int get_handler(enum nf_conntrack_msg_type type,
 		       struct nf_conntrack *ct,
 		       void *data)
@@ -319,6 +444,22 @@ static int get_handler(enum nf_conntrack_msg_type type,
 	return NFCT_CB_CONTINUE;
 }
 
+static int exp_get_handler(enum nf_conntrack_msg_type type,
+			   struct nf_expect *exp, void *data)
+{
+	const struct nf_conntrack *master =
+		nfexp_get_attr(exp, ATTR_EXP_MASTER);
+
+	if (!exp_filter_find(STATE(exp_filter), exp))
+		return NFCT_CB_CONTINUE;
+
+	if (ct_filter_conntrack(master, 1))
+		return NFCT_CB_CONTINUE;
+
+	STATE(get_retval) = 1;
+	return NFCT_CB_CONTINUE;
+}
+
 int
 init(void)
 {
@@ -355,7 +496,7 @@ init(void)
 	register_fd(STATE(local).fd, STATE(fds));
 
 	/* resynchronize (like 'dump' socket) but it also purges old entries */
-	STATE(resync) = nfct_open(CONNTRACK, 0);
+	STATE(resync) = nfct_open(CONFIG(netlink).subsys_id, 0);
 	if (STATE(resync)== NULL) {
 		dlog(LOG_ERR, "can't open netlink handler: %s",
 		     strerror(errno));
@@ -370,7 +511,7 @@ init(void)
 	fcntl(nfct_fd(STATE(resync)), F_SETFL, O_NONBLOCK);
 
 	if (STATE(mode)->internal->flags & INTERNAL_F_POPULATE) {
-		STATE(dump) = nfct_open(CONNTRACK, 0);
+		STATE(dump) = nfct_open(CONFIG(netlink).subsys_id, 0);
 		if (STATE(dump) == NULL) {
 			dlog(LOG_ERR, "can't open netlink handler: %s",
 			     strerror(errno));
@@ -380,13 +521,26 @@ init(void)
 		nfct_callback_register(STATE(dump), NFCT_T_ALL,
 				       dump_handler, NULL);
 
+		if (CONFIG(flags) & CTD_EXPECT) {
+			nfexp_callback_register(STATE(dump), NFCT_T_ALL,
+						exp_dump_handler, NULL);
+		}
+
 		if (nl_dump_conntrack_table(STATE(dump)) == -1) {
 			dlog(LOG_ERR, "can't get kernel conntrack table");
 			return -1;
 		}
+
+		if (CONFIG(flags) & CTD_EXPECT) {
+			if (nl_dump_expect_table(STATE(dump)) == -1) {
+				dlog(LOG_ERR, "can't get kernel "
+					      "expect table");
+				return -1;
+			}
+		}
 	}
 
-	STATE(get) = nfct_open(CONNTRACK, 0);
+	STATE(get) = nfct_open(CONFIG(netlink).subsys_id, 0);
 	if (STATE(get) == NULL) {
 		dlog(LOG_ERR, "can't open netlink handler: %s",
 		     strerror(errno));
@@ -395,7 +549,12 @@ init(void)
 	}
 	nfct_callback_register(STATE(get), NFCT_T_ALL, get_handler, NULL);
 
-	STATE(flush) = nfct_open(CONNTRACK, 0);
+	if (CONFIG(flags) & CTD_EXPECT) {
+		nfexp_callback_register(STATE(get), NFCT_T_ALL,
+					exp_get_handler, NULL);
+	}
+
+	STATE(flush) = nfct_open(CONFIG(netlink).subsys_id, 0);
 	if (STATE(flush) == NULL) {
 		dlog(LOG_ERR, "cannot open flusher handler");
 		return -1;
@@ -426,6 +585,11 @@ init(void)
 		}
 		nfct_callback_register2(STATE(event), NFCT_T_ALL,
 				        event_handler, NULL);
+
+		if (CONFIG(flags) & CTD_EXPECT) {
+			nfexp_callback_register2(STATE(event), NFCT_T_ALL,
+						 exp_event_handler, NULL);
+		}
 		register_fd(nfct_fd(STATE(event)), STATE(fds));
 	}