[AusNOG] Strange broadcast behaviour on linux 2.6.32 - riddle me this

[Shannon Gernyi]

Hey all,

I have a few boxen running CentOS 6.5 in a very vanilla install.

I ran into some weird behaviour that I couldn't get to the bottom of, and whilst a reboot "fixed" it, I'm not going to be able to sleep at night without asking the question, so riddle me this:

Box A is on a private network:

eth0 172.27.3.111/24

I'm have a requirement to broadcast on eth0, however, the box isn't sending said broadcast packets to the ethernet broadcast mac - it's sending out ARP requests for 172.27.3.255 instead.

12:49:44.129501 ARP, Request who-has 172.27.3.255 tell 172.27.3.111, length 28
12:49:45.129494 ARP, Request who-has 172.27.3.255 tell 172.27.3.111, length 28
12:49:46.129494 ARP, Request who-has 172.27.3.255 tell 172.27.3.111, length 28
12:49:48.132491 ARP, Request who-has 172.27.3.255 tell 172.27.3.111, length 28
12:49:49.132488 ARP, Request who-has 172.27.3.255 tell 172.27.3.111, length 28
12:49:50.132491 ARP, Request who-has 172.27.3.255 tell 172.27.3.111, length 28
12:49:52.135503 ARP, Request who-has 172.27.3.255 tell 172.27.3.111, length 28
12:49:53.135486 ARP, Request who-has 172.27.3.255 tell 172.27.3.111, length 28
12:49:54.135513 ARP, Request who-has 172.27.3.255 tell 172.27.3.111, length 28
12:49:56.139497 ARP, Request who-has 172.27.3.255 tell 172.27.3.111, length 28
12:49:57.139491 ARP, Request who-has 172.27.3.255 tell 172.27.3.111, length 28
12:49:58.139494 ARP, Request who-has 172.27.3.255 tell 172.27.3.111, length 28
12:50:00.131503 ARP, Request who-has 172.27.3.255 tell 172.27.3.111, length 28
12:50:01.131490 ARP, Request who-has 172.27.3.255 tell 172.27.3.111, length 28
12:50:02.131495 ARP, Request who-has 172.27.3.255 tell 172.27.3.111, length 28
12:50:04.134495 ARP, Request who-has 172.27.3.255 tell 172.27.3.111, length 28
12:50:05.134487 ARP, Request who-has 172.27.3.255 tell 172.27.3.111, length 28
12:50:06.134490 ARP, Request who-has 172.27.3.255 tell 172.27.3.111, length 28
12:50:08.137499 ARP, Request who-has 172.27.3.255 tell 172.27.3.111, length 28
12:50:09.137488 ARP, Request who-has 172.27.3.255 tell 172.27.3.111, length 28

eth0 is definitely configured correctly regarding network/mask: 

eth0      Link encap:Ethernet  HWaddr REMOVED
          inet addr:172.27.3.111  Bcast:172.27.3.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:227898706 errors:0 dropped:0 overruns:0 frame:0
          TX packets:219816363 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:59578359175 (55.4 GiB)  TX bytes:145429518480 (135.4 GiB)
          Interrupt:245

arp output confirms the above dump:

arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
172.27.3.255                   (incomplete)                              eth0


And finally, manually adding the IP to the arp table with a destination of ethernet broadcast (ie, where the frames should be directed anyway) made things work:

arp -s 172.27.3.255 ff:ff:ff:ff:ff:ff


12:56:30.232505 ARP, Request who-has 172.27.3.255 tell 172.27.3.111, length 28
12:56:31.232506 ARP, Request who-has 172.27.3.255 tell 172.27.3.111, length 28
12:56:32.232496 ARP, Request who-has 172.27.3.255 tell 172.27.3.111, length 28
12:56:34.235510 ARP, Request who-has 172.27.3.255 tell 172.27.3.111, length 28
12:56:35.235505 ARP, Request who-has 172.27.3.255 tell 172.27.3.111, length 28
12:56:36.042743 IP 172.27.3.111.53957 > 172.27.3.255.ha-cluster: UDP, length 225
12:56:36.236321 IP 172.27.3.111.53957 > 172.27.3.255.ha-cluster: UDP, length 246
12:56:36.236353 IP 172.27.3.111.53957 > 172.27.3.255.ha-cluster: UDP, length 225
12:56:38.238098 IP 172.27.3.111.53957 > 172.27.3.255.ha-cluster: UDP, length 225


Now, I can't go back and debug further, but if anyone has seen this or similar behaviour before - please share possible explanations. It's doing my head in!