[AusNOG] DNS Craziness
Joshua D'Alton
joshua at railgun.com.au
Tue Feb 18 23:14:27 EST 2014
8.8.8.8 is reporting 2 different addresses because you are hitting 2
different clusters. When you lookup over MP, 8.8.8.8A then looks up from
route53 over route A, when you look up via iprimus you hit 8.8.8.8B, which
looks up over route B. It might seem quite coincidental that google has the
same/similar upstreams and connectivity to AWS that you do, and therefore
exhibiting the same weirdness, but given the lack of connectivity options
in australia, not totally surprising.
I'd be interested in your traces to route53, but ultimately i'd be making
an AWS support request and telling them one of their clusters is out of
sync (and serving the wrong record).
On Tue, Feb 18, 2014 at 11:04 PM, Skeeve Stevens <
skeeve+ausnog at eintellegonetworks.com> wrote:
> Hey Joshua,
>
> I was thinking the same... but why would 8.8.8.8 report two different
> addresses based on the source?
>
> Yes, the wrong address is an old address.
>
> The DNS changed over a week ago, so should be fine. I do need to check
> the dates... but even so, why would google report differently from
> different sources?
>
>
> ...Skeeve
>
> *Skeeve Stevens - *eintellego Networks Pty Ltd
> skeeve at eintellegonetworks.com ; www.eintellegonetworks.com
>
> Phone: 1300 239 038; Cell +61 (0)414 753 383 ; skype://skeeve
>
> facebook.com/eintellegonetworks ; <http://twitter.com/networkceoau>
> linkedin.com/in/skeeve
>
> twitter.com/theispguy ; blog: www.theispguy.com
>
>
> The Experts Who The Experts Call
> Juniper - Cisco - Cloud - Consulting - IPv4 Brokering
>
>
> On Tue, Feb 18, 2014 at 10:59 PM, Joshua D'Alton <joshua at railgun.com.au>wrote:
>
>> The fact you see the same behaviour directly from AWS shows it is nothing
>> to do with google, they are merely doing the recursive lookup for you.
>>
>> So I'd be checking the AWS route53 settings. Also, is the wrong IP a
>> previous IP, if so was it changed recently and therefore maybe some
>> TTL/caching issue, or if it isn't the correct IP, does the IP
>> resolve/display some sort of domain squatting site, if so I'd be checking
>> nameservers and your domain panel incase you've been hijacked (deliberately
>> or not).
>>
>>
>> On Tue, Feb 18, 2014 at 10:53 PM, Skeeve Stevens <
>> skeeve+ausnog at eintellegonetworks.com> wrote:
>>
>>> That would have to be some seriously specific GeoDNS as they are both
>>> hitting Google Sydney (I assume).
>>>
>>> Respond with blah over MP and blah of iPrimus... I wouldn't think so?
>>>
>>>
>>> ...Skeeve
>>>
>>> *Skeeve Stevens - *eintellego Networks Pty Ltd
>>> skeeve at eintellegonetworks.com ; www.eintellegonetworks.com
>>>
>>> Phone: 1300 239 038; Cell +61 (0)414 753 383 ; skype://skeeve
>>>
>>> facebook.com/eintellegonetworks ; <http://twitter.com/networkceoau>
>>> linkedin.com/in/skeeve
>>>
>>> twitter.com/theispguy ; blog: www.theispguy.com
>>>
>>>
>>> The Experts Who The Experts Call
>>> Juniper - Cisco - Cloud - Consulting - IPv4 Brokering
>>>
>>>
>>> On Tue, Feb 18, 2014 at 10:49 PM, Nick @ Deltaband <nick at deltaband.com>wrote:
>>>
>>>> Without seeing the hosts it's hard to know, but Route 53 supports a
>>>> GeoDNS type service that can provide different answers based on a bunch of
>>>> different policies inc latency. Could it be that?
>>>>
>>>>
>>>> On 18 February 2014 22:41, Skeeve Stevens <
>>>> skeeve+ausnog at eintellegonetworks.com> wrote:
>>>>
>>>>> Hey all,
>>>>>
>>>>> I hope to hell I haven't missed something here, but I have spent a lot
>>>>> of hours trying to figure out this problem and it is proving a little hard
>>>>> to nail down.
>>>>>
>>>>> Simply, googles public DNS 8.8.8.8 is giving me two different
>>>>> responses from different peering connections.
>>>>>
>>>>> A response from 8.8.8.8 over Megaport MLPA is giving me a different
>>>>> response to 8.8.8.8 over direct iPrimus -> Google peering.
>>>>>
>>>>> The DNS is being sourced from AWS Route53 - so there shouldn't be any
>>>>> conflicting information.
>>>>>
>>>>> There is no round-robin involved because both sides are consistently
>>>>> returning the same information.
>>>>>
>>>>> The correct answer is actually via the iPrimus connection.
>>>>>
>>>>> The mind blowing thing is that I get the same 'different' response if
>>>>> I query the AWS DNS server directly via each connection (as below).
>>>>>
>>>>> I hope this makes sense... and that I haven't missed something
>>>>> obvious. It almost feels like there is some split-view DNS going on here,
>>>>> but 8.8.8.8 wouldn't do that.
>>>>>
>>>>> Alternatively, one of the connections has some sort of transparent
>>>>> DNS... i.e. the Megaport one.. but it is just going through a Juniper
>>>>> SRX550 cluster doing NAT into a Juniper MX border going into Megaport -
>>>>> none of which is doing DNS magic.
>>>>>
>>>>> The iPrimus connection is going into Checkpoints as the edge straight
>>>>> into iPrimus... but they are reporting the correct answer.
>>>>>
>>>>> My head hurts :(
>>>>>
>>>>>
>>>>> *VIA MEGAPORT*
>>>>>
>>>>> [root at auth01n ~]# traceroute 8.8.8.8
>>>>> traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
>>>>> 1 10.x.0.3 (10.x.0.3) 0.382 ms 0.353 ms 0.256 ms
>>>>> 2 10.x.0.62 (10.x.0.62) 3.370 ms 3.325 ms 3.320 ms
>>>>> 3 10.x.0.25 (10.x.0.25) 3.314 ms 3.297 ms 3.269 ms
>>>>> *NAT HERE*
>>>>> 4 as15169.sydney.megaport.com (103.26.68.56) 3.180 ms 3.142 ms
>>>>> 3.142 ms
>>>>> 5 72.14.237.21 (72.14.237.21) 3.125 ms 3.070 ms 3.077 ms
>>>>> 6 google-public-dns-a.google.com (8.8.8.8) 3.042 ms 2.331 ms
>>>>> 2.236 ms
>>>>>
>>>>>
>>>>> [root at auth01n ~]# dig a vpn.x.com @8.8.8.8
>>>>>
>>>>> ; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.23.rc1.el6_5.1 <<>> a vpn.x.com @
>>>>> 8.8.8.8
>>>>> ;; global options: +cmd
>>>>> ;; Got answer:
>>>>> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24261
>>>>> ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
>>>>>
>>>>> ;; QUESTION SECTION:
>>>>> ;vpn.x.com. IN A
>>>>>
>>>>> ;; ANSWER SECTION:
>>>>> vpn.x.com. 299 IN A *x.x.171.124*
>>>>>
>>>>> ;; Query time: 166 msec
>>>>> ;; SERVER: 8.8.8.8#53(8.8.8.8)
>>>>> ;; WHEN: Tue Feb 18 22:28:24 2014
>>>>> ;; MSG SIZE rcvd: 48
>>>>>
>>>>> =============
>>>>>
>>>>> *VIA IPRIMUS/EQUINIX*
>>>>>
>>>>> [root at auth01 ~]# traceroute 8.8.8.8
>>>>> traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
>>>>> 1 10.x.65.2 (10.x.65.2) 0.476 ms 1.003 ms 1.227 ms
>>>>> 2 10.x.126.5 (10.x.126.5) 0.471 ms 0.608 ms 0.410 ms
>>>>> *NAT HERE*
>>>>> 3 x.x.static.syd.iprimus.net.au (203.134.x.x) 2.253 ms 2.064 ms
>>>>> 2.075 ms
>>>>> 4 atm3-2.ac02.syd.iprimus.net.au (203.134.2.249) 1.149 ms 0.768
>>>>> ms at-1-0-2.ic02.pth.iprimus.net.au (203.134.2.225) 1.107 ms
>>>>> 5 xe-0-3-0.bsr01.equ.iprimus.net.au (203.134.2.234) 1.313 ms
>>>>> 0.646 ms xe-1-0-0.bsr01.equ.iprimus.net.au (203.134.2.254) 1.249 ms
>>>>> 6 google-gw.syd.iprimus.net.au (203.134.2.98) 1.406 ms 1.260 ms
>>>>> 1.209 ms
>>>>> 7 72.14.237.21 (72.14.237.21) 1.602 ms 1.871 ms 2.017 ms
>>>>> 8 google-public-dns-a.google.com (8.8.8.8) 1.333 ms 1.393 ms
>>>>> 1.403 ms
>>>>>
>>>>>
>>>>> [root at auth01 ~]# dig a vpn.x.com @8.8.8.8
>>>>>
>>>>> ; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.23.rc1.el6_5.1 <<>> a vpn.x.com @
>>>>> 8.8.8.8
>>>>> ;; global options: +cmd
>>>>> ;; Got answer:
>>>>> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43084
>>>>> ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
>>>>>
>>>>> ;; QUESTION SECTION:
>>>>> ;vpn.x.com. IN A
>>>>>
>>>>> ;; ANSWER SECTION:
>>>>> vpn.x.com. 174 IN A *x.x.102.124*
>>>>>
>>>>> ;; Query time: 143 msec
>>>>> ;; SERVER: 8.8.8.8#53(8.8.8.8)
>>>>> ;; WHEN: Tue Feb 18 22:30:29 2014
>>>>> ;; MSG SIZE rcvd: 48
>>>>>
>>>>>
>>>>> ...Skeeve
>>>>>
>>>>> *Skeeve Stevens - *eintellego Networks Pty Ltd
>>>>> skeeve at eintellegonetworks.com ; www.eintellegonetworks.com
>>>>>
>>>>> Phone: 1300 239 038; Cell +61 (0)414 753 383 ; skype://skeeve
>>>>>
>>>>> facebook.com/eintellegonetworks ; <http://twitter.com/networkceoau>
>>>>> linkedin.com/in/skeeve
>>>>>
>>>>> twitter.com/theispguy ; blog: www.theispguy.com
>>>>>
>>>>>
>>>>> The Experts Who The Experts Call
>>>>> Juniper - Cisco - Cloud - Consulting - IPv4 Brokering
>>>>>
>>>>> _______________________________________________
>>>>> AusNOG mailing list
>>>>> AusNOG at lists.ausnog.net
>>>>> http://lists.ausnog.net/mailman/listinfo/ausnog
>>>>>
>>>>>
>>>>
>>>
>>> _______________________________________________
>>> AusNOG mailing list
>>> AusNOG at lists.ausnog.net
>>> http://lists.ausnog.net/mailman/listinfo/ausnog
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20140218/184d53fa/attachment-0001.html>
More information about the AusNOG
mailing list