[AusNOG] Optus downtime chat + affecting SMS verification to Telstra?

[John Edwards]

Those who believe it's a BGP issue might want to review
https://www.cidr-report.org/cgi-bin/plota?file=%2fvar%2fdata%2fbgp%2fas2.0%2fbgp%2dactive%2etxt&descr=Active%20BGP%20entries%20%28FIB%29&ylabel=Active%20BGP%20entries%20%28FIB%29&with=step

This shows a global table getting dangerously close to a 1024K boundary
found in some router FIBs.

John


On Wed, 8 Nov 2023 at 11:57, Christopher Hawker <chris at thesysadmin.dev>
wrote:

> They weren't appearing earlier so I'd say it's new. Smells like BGP has
> shat itself.
>
> - ​CH
> ------------------------------
> *From:* AusNOG <ausnog-bounces at lists.ausnog.net> on behalf of Tim Malone <
> tdmalone at gmail.com>
> *Sent:* Wednesday, November 8, 2023 12:16 PM
> *To:* Nick Jenkin <njenkin at gmail.com>
> *Cc:* ausnog at lists.ausnog.net <ausnog at lists.ausnog.net>
> *Subject:* Re: [AusNOG] Optus downtime chat + affecting SMS verification
> to Telstra?
>
> Cloudflare is also reporting BGP origin hijack warnings:
> https://radar.cloudflare.com/as7474?dateRange=2d
> (unsure if related though)
>
>
> On Wed, 8 Nov 2023 at 11:10, Nick Jenkin <njenkin at gmail.com> wrote:
>
> Just a few announcements...
> https://radar.cloudflare.com/traffic/as4804?dateRange=2d
>
> On Wed, 8 Nov 2023 at 11:06, Ben Buxton <bb.ausnog at bb.cactii.net> wrote:
>
>
>
> On Wed, 8 Nov 2023 at 10:14, DaZZa <dazzagibbs at gmail.com> wrote:
>
> Yeah, I'd be willing to bet that it's a change which wasn't thoroughly
> tested before being rolled out, and which had an inadequate backout
> plan.
>
>
> Also, "Our on-site technician is actively prioritising establishing a
> console connection.".
>
> I mean come on, it's nearly 2024 and a [major] telco does not have remote
> console access? Whilst I'm
> looking forward to enthusiastically reading the PM, I'll have to book a
> physio appointment in advance due to
> neck strain from all the head shaking it'll likely induce.
>
> BB
>
>
>
>
> Interestingly, my Optus mobile actually had a valid connection for a
> short time - wasn't able to actually DO anything, but was connected to
> the OPtus network - but it's now gone to "SOS" mode.
>
> D
>
> On Wed, 8 Nov 2023 at 10:01, John Edwards <jaedwards at gmail.com> wrote:
> >
> > The 4am Wednesday morning outage start looks suspiciously like a
> firmware upgrade window.
> >
> > I note that Optus devices where I am are showing "SoS" which indicates
> the tower is unable to reach the location register, which presumably is on
> a private network and indicative of a pretty major fault rather than just
> IP.
> >
> > John
> >
> >
> > On Wed, 8 Nov 2023 at 09:10, DaZZa <dazzagibbs at gmail.com> wrote:
> >>
> >> The Optus hamster finally died of old age.
> >>
> >> I would suggest your SMS issues would be caused by whoever is issuing
> >> the SMS using Optus - not so much by the Telstra end receiving it.
> >>
> >> Anecdotally, Optus enterprise/wholesale appears to be still functional
> >> - at least my link appears to be working fine - and my BGP
> >> advertisements are still being seen overseas - seems to be only NBN
> >> and mobile based services which are busted
> >>
> >> D
> >>
> >> On Wed, 8 Nov 2023 at 09:27, <francisfides at mailup.net> wrote:
> >> >
> >> > Morning all,
> >> > Hope the chaos isn't too hard on your work/family.
> >> > I have had trouble with a couple of SMS verifications coming through
> to me, my Telstra number. Is this related?
> >> >
> >> > Any general banter around the downtime would be fine too - looks like
> it all began at 4.07am AEDT?
> >> >
> >> > Cheers
> >> >
> >> > --
> >> >
> >> >   francisfides at mailup.net
> >> > _______________________________________________
> >> > AusNOG mailing list
> >> > AusNOG at lists.ausnog.net
> >> > https://lists.ausnog.net/mailman/listinfo/ausnog
> >>
> >>
> >>
> >> --
> >> veg·e·tar·i·an:
> >> Ancient tribal slang for the village idiot who can't hunt, fish or ride
> >> _______________________________________________
> >> AusNOG mailing list
> >> AusNOG at lists.ausnog.net
> >> https://lists.ausnog.net/mailman/listinfo/ausnog
>
>
>
> --
> veg·e·tar·i·an:
> Ancient tribal slang for the village idiot who can't hunt, fish or ride
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> https://lists.ausnog.net/mailman/listinfo/ausnog
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ausnog.net/pipermail/ausnog/attachments/20231108/e19e47ff/attachment.htm>