[AusNOG] Optus downtime chat + affecting SMS verification toTelstra?

[Christopher O'Shea]

I wouldn't be so quick to blame it on a single thing. We have all been
there, An incident always comes down to many things not going the way you
think.

Reading between the lines, I see that a peer's network creates larger than
"normal" routes, and seeing they called out IPv6 in their submission to
Senate [1]
Lack of filtering of v6 for that peer due to an oversight or
misunderstanding of the template/group between v4 and v6.

Then, when it was shared with their PE routers (Which seem to be Cisco) On
the ASK9K (Not sure what they use), the default limit of  524288 [2] for v6
could lead to the session's termination by default.

We should read these reports and understand if the same thing could happen
to your network, what protection you have to stop this, and your device's
default behaviour.

I would like to know more about their out-of-band and why it had issues.
(Could it be that DNS broke, issue getting to internal documentation or was
the password vault access broken, or the IP limit of the OOB device was too
tight).

Chris O'Shea

[1]
https://www.aph.gov.au/DocumentStore.ashx?id=2ed95079-023d-49d5-87fd-d9029740629b&subId=750333
 reports of the Optus outage
[2]
https://www.cisco.com/c/en/us/td/docs/routers/asr9000/software/routing/command/reference/b-routing-cr-asr9000/bgp-commands.html#wp3192417938



On Fri, Nov 17, 2023 at 2:02 AM Tony Wicks <tony at wicks.co.nz> wrote:

> To be fair, Assuming there were config issues (i.e. the lack of
> maximum-prefixes and the lack of filtering preventing large route tables
> hitting devices that can not carry full tables) the behaviour of a network
> device when its RIB/FIB or memory is exceeded also significantly comes into
> play. Dropping BGP is fine, crashing the router so it requires a hard reset
> is another case entirely. In my experience (I have not used Cisco's in a
> telco environment for many years however) Cisco devices have been much more
> pre-disposed to crash catastrophically than over vendor devices like Nokia
> or Juniper.
>
>
>
> -----Original Message-----
> From: AusNOG <ausnog-bounces at lists.ausnog.net> On Behalf Of DaZZa
> Sent: Friday, November 17, 2023 2:38 PM
> To: Andrew Oakeley <andrew at oakeley.com.au>
> Cc: michael.bethune at australiaonline.au; Luke Thompson <
> luke.t at tncrew.com.au>; ausnog at lists.ausnog.net
> Subject: Re: [AusNOG] Optus downtime chat + affecting SMS verification
> toTelstra?
>
> What a load of crap.
>
> The root cause was they're morons, and configured the routers incorrectly.
>
> Cisco had nothing to do with it. I'll bet the routers behaved exactly as
> they were intended to behave.
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20231117/103b20fd/attachment.htm>