[AusNOG] Cisco GRE Tunnel weirdness

[joe at apcs.com.au]

Hi List,

  I have a GRE tunnel between 2 sites over a link limited to 1500 MTU.

  As such we have mtu set to 1440 and mss-adjust to 1400 on both ends. 
This is overly cautious probably but it was working.

  Anyway - it had been working quite fine for some time, but randomly we 
started seeing massive performance issues. Bandwidth throughput halved 
and ping times sky rocketed (~50ms to ~1000ms). We tried bringing down 
the tunnel and back up, no luck, and even power cycled each end (Cisco 
3945's), no luck.

  We have confirmed that the config's had not been changed for weeks. 
Neither end had crashed and rebooted. The tunnel itself did not go down 
between 'working' and 'not working'. Performance and ping times via the 
tunnel endpoint address' is fine, proving (to me) that the networks 
between the 2 sites are not the issue, but the tunnel itself. No links 
are saturated, and CPU performance is quite tame (both before and during 
the issue)

  For now we have gone back to backup path but I haven't been able to 
find similar problems online, and my own Cisco tunnel experience leaves 
me empty so far.

  Has anyone experienced a similar issue? A working tunnel suddenly 
having major performance issues?

Thanks,
Joe