[AusNOG] Optus downtime chat + affecting SMS verification toTelstra?
Michael Bethune
mike at ozonline.com.au
Tue Nov 14 13:02:35 AEDT 2023
"Optus network received changes to routing information from an
international peering network following a software upgrade"
I note they are very careful to avoid nominating whose software upgrade.
I also note that when they say they received routing updates,
don't they limit the number of prefixes accepted by their BGP from
any given peer?
Sounds like a carefully crafted statement to enable them to point fingers
elsewhere, not unexpected.
- Michael.
Quoting francisfides at mailup.net:
> Looks like it was a software upgrade:
> https://www.abc.net.au/news/2023-11-13/optus-identifies-cause-of-nationwide-outage-software-upgrade/103099902
>
> Nothing in their media centre, just appears as a new box on their
> outage response page: https://www.optus.com.au/notices/outage-response
>
> Cheers
>
> ----
> Text:
>
> "We have been working to understand what caused the outage on
> Wednesday, and we now know what the cause was and have taken steps
> to ensure it will not happen again. We apologise sincerely for
> letting our customers down and the inconvenience it caused.
>
> At around 4.05am Wednesday morning, the Optus network received
> changes to routing information from an international peering network
> following a software upgrade. These routing information changes
> propagated through multiple layers in our network and exceeded
> preset safety levels on key routers. This resulted in those routers
> disconnecting from the Optus IP Core network to protect themselves.
>
> The restoration required a large-scale effort of the team and in
> some cases required Optus to reconnect or reboot routers physically,
> requiring the dispatch of people across a number of sites in
> Australia. This is why restoration was progressive over the afternoon.
>
> Given the widespread impact of the outage, our investigations into
> the issue took longer than we would have liked as we examined
> several different paths to restoration. The restoration of the
> network was at all times our priority and we subsequently
> established the cause working together with our partners. We have
> made changes to the network to address this issue so that it cannot
> occur again.
>
> We are committed to learning from what has occurred and continuing
> to work with our international vendors and partners to increase the
> resilience of our network. We will also support and fully cooperate
> with the reviews being undertaken by the Government and the Senate.
>
> We continue to invest heavily to improve the resiliency of our
> network and services."
>
> --
>
> francisfides at mailup.net
>
> On Thu, Nov 9, 2023, at 07:15, DaZZa wrote:
>> I have all three you're asking about.
>>
>> But I'm very small potatoes compared to most of the members of this
>> list, and my required remote footprint is correspondingly small, so
>> it's easy to maintain.
>>
>> D
>>
>> On Thu, 9 Nov 2023 at 06:18, Phillip Grasso
>> <phillip.grasso at gmail.com> wrote:
>>>>
>>>> I mean come on, it's nearly 2024 and a [major] telco does not
>>>> have remote console access?
>>>
>>>
>>> If we send a poll out to this community, how many would be able to
>>> genuinely honestly answer:
>>>
>>> Do you have a console or appropriate control plane access into all
>>> your critical infrastructure?
>>> Do you have independant out of band that does not share any
>>> infrastructure with your current system(s) - with exemption for
>>> physical location and power.
>>> Do you have the ability to remote power control your devices?
>>>
>>> We know from the facebook outage in 2021 that they probably didn't
>>> have the above, so its not entirely uncommon for folks to have
>>> *proper independant* console and remote access.
>>>
>>>
>>> I empathize with the Optus team and their customers who have been
>>> negatively impacted by this incident. I sincerely hope that some
>>> positive outcomes can emerge from this situation, including:
>>>
>>> - Attention to critical infrastructure resilience
>>> - BGP clue increases
>>> - Incident management improves
>>> (I'm sure there's more).
>>>
>>> Network is a black box to most people and I think a large chunk of
>>> Australia now knows what it feels like to not have it.
>>>
>>>
>>> On Wed, 8 Nov 2023 at 11:06, Ben Buxton <bb.ausnog at bb.cactii.net> wrote:
>>>>
>>>>
>>>>
>>>> On Wed, 8 Nov 2023 at 10:14, DaZZa <dazzagibbs at gmail.com> wrote:
>>>>>
>>>>> Yeah, I'd be willing to bet that it's a change which wasn't thoroughly
>>>>> tested before being rolled out, and which had an inadequate backout
>>>>> plan.
>>>>
>>>>
>>>> Also, "Our on-site technician is actively prioritising
>>>> establishing a console connection.".
>>>>
>>>> I mean come on, it's nearly 2024 and a [major] telco does not
>>>> have remote console access? Whilst I'm
>>>> looking forward to enthusiastically reading the PM, I'll have to
>>>> book a physio appointment in advance due to
>>>> neck strain from all the head shaking it'll likely induce.
>>>>
>>>> BB
>>>>
>>>>
>>>>>
>>>>>
>>>>> Interestingly, my Optus mobile actually had a valid connection for a
>>>>> short time - wasn't able to actually DO anything, but was connected to
>>>>> the OPtus network - but it's now gone to "SOS" mode.
>>>>>
>>>>> D
>>>>>
>>>>> On Wed, 8 Nov 2023 at 10:01, John Edwards <jaedwards at gmail.com> wrote:
>>>>> >
>>>>> > The 4am Wednesday morning outage start looks suspiciously like
>>>>> a firmware upgrade window.
>>>>> >
>>>>> > I note that Optus devices where I am are showing "SoS" which
>>>>> indicates the tower is unable to reach the location register,
>>>>> which presumably is on a private network and indicative of a
>>>>> pretty major fault rather than just IP.
>>>>> >
>>>>> > John
>>>>> >
>>>>> >
>>>>> > On Wed, 8 Nov 2023 at 09:10, DaZZa <dazzagibbs at gmail.com> wrote:
>>>>> >>
>>>>> >> The Optus hamster finally died of old age.
>>>>> >>
>>>>> >> I would suggest your SMS issues would be caused by whoever is issuing
>>>>> >> the SMS using Optus - not so much by the Telstra end receiving it.
>>>>> >>
>>>>> >> Anecdotally, Optus enterprise/wholesale appears to be still functional
>>>>> >> - at least my link appears to be working fine - and my BGP
>>>>> >> advertisements are still being seen overseas - seems to be only NBN
>>>>> >> and mobile based services which are busted
>>>>> >>
>>>>> >> D
>>>>> >>
>>>>> >> On Wed, 8 Nov 2023 at 09:27, <francisfides at mailup.net> wrote:
>>>>> >> >
>>>>> >> > Morning all,
>>>>> >> > Hope the chaos isn't too hard on your work/family.
>>>>> >> > I have had trouble with a couple of SMS verifications
>>>>> coming through to me, my Telstra number. Is this related?
>>>>> >> >
>>>>> >> > Any general banter around the downtime would be fine too -
>>>>> looks like it all began at 4.07am AEDT?
>>>>> >> >
>>>>> >> > Cheers
>>>>> >> >
>>>>> >> > --
>>>>> >> >
>>>>> >> > francisfides at mailup.net
>>>>> >> > _______________________________________________
>>>>> >> > AusNOG mailing list
>>>>> >> > AusNOG at lists.ausnog.net
>>>>> >> > https://lists.ausnog.net/mailman/listinfo/ausnog
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> veg·e·tar·i·an:
>>>>> >> Ancient tribal slang for the village idiot who can't hunt,
>>>>> fish or ride
>>>>> >> _______________________________________________
>>>>> >> AusNOG mailing list
>>>>> >> AusNOG at lists.ausnog.net
>>>>> >> https://lists.ausnog.net/mailman/listinfo/ausnog
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> veg·e·tar·i·an:
>>>>> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>>>>> _______________________________________________
>>>>> AusNOG mailing list
>>>>> AusNOG at lists.ausnog.net
>>>>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>>>
>>>> _______________________________________________
>>>> AusNOG mailing list
>>>> AusNOG at lists.ausnog.net
>>>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>
>>
>>
>> --
>> veg·e·tar·i·an:
>> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
More information about the AusNOG
mailing list