[AusNOG] IPv6 is hard.

[Tom Lanyon]

As a content/SaaS provider, we began a project to move our production services into a new data centre.  We took the opportunity to re-think some past design choices and implement with a "Green Fields" approach.

One of the key goals was to enable IPv6 support on our client-facing services (or at least have the ability to do so and be confident that it worked and we could support it).  I figured the list might be interested in some of the vague findings we've made so far.

Part of the way into deployment of the routing equipment, and partly spurred on by newly released [at the time] stateful NAT64 support in Cisco's ASR1000 IOS, we decided it would be worth trying to run IPv6-only on our internal network and just terminate all IPv4 on our load balancing/content switching infrastructure.  Our previous/current architecture uses a bunch of internal servers (virtual and physical) running on RFC1918 IPv4 addresses which use NAT for the small amount of Internet connectivity they require.  It hardly seemed worth managing dual stack-servers, dual-stack on the server-facing network equipment and dual-stack internal services (like DNS, DHCP, IPAM, etc.) when the IPv4 connections were to be NATted anyway.

We decided to test some IPv6-only servers in the trial environment, using NAT64 + DNS64 to have them reach public Internet hosts;  this appears to be severely in contrast with many other people's approaches in the content / hosting arena - where people are retrofitting IPv6 to their existing IPv4-based systems.

The first stumbling block was actually implementing NAT64 and DNS64.  Stateless NAT64 was useless (as far as we were concerned) and stateful NAT64 was only available in the latest IOS-XE at the time -- and sure, DNS64 was in bind 9.8.0 (released early 2011), however none of the current major linux "server" distros included 9.8.x at the time (it does seem to be available in RHEL6.3 and Ubuntu 12.04 LTS).  Anyway, easy to fix - we upgraded IOS on the ASR1000s and built some bind 9.8 RPMs.

Next problem - linux kernel bug breaking iptables connection tracking when using NAT64; this was fixed in recent kernel versions but not in the latest server distro kernels. OK - a kernel rebuild later and we had some working IPv6->IPv4 connectivity via NAT64 and DNS64. Great.

Since then, we've fought against software after software trying to get them working in IPv6-only-land.  Many just don't have support for listening on IPv6 sockets, many explicitly try to connect to IPv4 literal addresses (e.g. don't use DNS so can't make use of DNS64) and many simply don't support connecting to any IPv6 sockets (e.g. use of AF_INET explicitly).  Don't get me started on big IPv6-proponents like Cisco not eating their own dog food (Cisco UCS server infrastructure only supports IPv4 for all of the management interfaces being one example).  It's not practical to need to build "cutting-edge" (read: not vendor supported) versions of every piece of software just to get basic functionality on IPv6.

So that was in data centre land.  We also performed a small test releasing IPv6 to a workstation network and removing IPv4 connectivity from those hosts.  Most standard apps (e.g. email, web browsing) worked fine, however some people were unable to connect to our internal Jabber/XMPP server (which uses DNS SRV records) and some apps like Dropbox on OS X appeared to stop functioning without IPv4 connectivity.

One of the arguments for IPv6 adoption was/is being able to supply services to people without any IPv4 connectivity, but from our experience with current software and hardware stacks, that feels somewhat unrealistic and perhaps simply not workable.  If it only works alongside IPv4, is there really much of a driver for anyone to adopt IPv6?

Don't get me wrong, IPv6 in a dual-stack configuration with IPv4 is fine - but on its lonesome, IPv6 in real environments just feels "broken".  Sure, this stuff takes time to get working -- but we've had lots of time to sort this out already.

-- 
Tom