[AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[Phillip Grasso]

>
> I mean come on, it's nearly 2024 and a [major] telco does not have remote
> console access?
>

If we send a poll out to this community, how many would be able to
genuinely honestly answer:

Do you have a console or appropriate control plane access into all your
critical infrastructure?
Do you have independant out of band that does not share any infrastructure
with your current system(s) - with exemption for physical location and
power.
Do you have the ability to remote power control your devices?

We know from the facebook outage in 2021 that they probably didn't have the
above, so its not entirely uncommon for folks to have *proper independant*
console and remote access.


I empathize with the Optus team and their customers who have been
negatively impacted by this incident. I sincerely hope that some positive
outcomes can emerge from this situation, including:

- Attention to critical infrastructure resilience
- BGP clue increases
- Incident management improves
(I'm sure there's more).

Network is a black box to most people and I think a large chunk of
Australia now knows what it feels like to not have it.


On Wed, 8 Nov 2023 at 11:06, Ben Buxton <bb.ausnog at bb.cactii.net> wrote:

>
>
> On Wed, 8 Nov 2023 at 10:14, DaZZa <dazzagibbs at gmail.com> wrote:
>
>> Yeah, I'd be willing to bet that it's a change which wasn't thoroughly
>> tested before being rolled out, and which had an inadequate backout
>> plan.
>>
>
> Also, "Our on-site technician is actively prioritising establishing a
> console connection.".
>
> I mean come on, it's nearly 2024 and a [major] telco does not have remote
> console access? Whilst I'm
> looking forward to enthusiastically reading the PM, I'll have to book a
> physio appointment in advance due to
> neck strain from all the head shaking it'll likely induce.
>
> BB
>
>
>
>>
>> Interestingly, my Optus mobile actually had a valid connection for a
>> short time - wasn't able to actually DO anything, but was connected to
>> the OPtus network - but it's now gone to "SOS" mode.
>>
>> D
>>
>> On Wed, 8 Nov 2023 at 10:01, John Edwards <jaedwards at gmail.com> wrote:
>> >
>> > The 4am Wednesday morning outage start looks suspiciously like a
>> firmware upgrade window.
>> >
>> > I note that Optus devices where I am are showing "SoS" which indicates
>> the tower is unable to reach the location register, which presumably is on
>> a private network and indicative of a pretty major fault rather than just
>> IP.
>> >
>> > John
>> >
>> >
>> > On Wed, 8 Nov 2023 at 09:10, DaZZa <dazzagibbs at gmail.com> wrote:
>> >>
>> >> The Optus hamster finally died of old age.
>> >>
>> >> I would suggest your SMS issues would be caused by whoever is issuing
>> >> the SMS using Optus - not so much by the Telstra end receiving it.
>> >>
>> >> Anecdotally, Optus enterprise/wholesale appears to be still functional
>> >> - at least my link appears to be working fine - and my BGP
>> >> advertisements are still being seen overseas - seems to be only NBN
>> >> and mobile based services which are busted
>> >>
>> >> D
>> >>
>> >> On Wed, 8 Nov 2023 at 09:27, <francisfides at mailup.net> wrote:
>> >> >
>> >> > Morning all,
>> >> > Hope the chaos isn't too hard on your work/family.
>> >> > I have had trouble with a couple of SMS verifications coming through
>> to me, my Telstra number. Is this related?
>> >> >
>> >> > Any general banter around the downtime would be fine too - looks
>> like it all began at 4.07am AEDT?
>> >> >
>> >> > Cheers
>> >> >
>> >> > --
>> >> >
>> >> >   francisfides at mailup.net
>> >> > _______________________________________________
>> >> > AusNOG mailing list
>> >> > AusNOG at lists.ausnog.net
>> >> > https://lists.ausnog.net/mailman/listinfo/ausnog
>> >>
>> >>
>> >>
>> >> --
>> >> veg·e·tar·i·an:
>> >> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>> >> _______________________________________________
>> >> AusNOG mailing list
>> >> AusNOG at lists.ausnog.net
>> >> https://lists.ausnog.net/mailman/listinfo/ausnog
>>
>>
>>
>> --
>> veg·e·tar·i·an:
>> Ancient tribal slang for the village idiot who can't hunt, fish or ride
>> _______________________________________________
>> AusNOG mailing list
>> AusNOG at lists.ausnog.net
>> https://lists.ausnog.net/mailman/listinfo/ausnog
>>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20231109/9b663a31/attachment.htm>