From Luke.Taucher at riotinto.com Thu Nov 2 16:24:41 2023 From: Luke.Taucher at riotinto.com (Taucher, Luke (IST)) Date: Thu, 2 Nov 2023 05:24:41 +0000 Subject: [AusNOG] Contact Request - Swoop Networks Message-ID: Hi team, If there?s anybody from Swoop Networks here, could you contact me off list? Cheers, Luke Taucher Specialist, Cyber Threat Intelligence | Information Systems and Technology 155 Charlotte Street, Brisbane, Queensland, 4000, Australia | M +61 (0) 410 256 465 [Logo Description automatically generated] Our operations are located on land and waters that have belonged to Indigenous peoples for thousands of years. I pay my respects to Elders, both past and present, and further acknowledge the important role that Indigenous peoples continue to play within communities and our business.? Rio Tinto Limited ACN 004 458 404 Registered office Level 43, 120 Collins Street, Melbourne Vic 3000, Australia. This email is confidential and contains personal data. It may also be privileged. If you are not the intended recipient, please notify the sender immediately and delete this message from your system without first printing or copying it. Rio Tinto?s Data Privacy Standard applies to all personal data contained in this email (including any attachments).? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.png Type: image/png Size: 2696 bytes Desc: image003.png URL: From mrp at mrp.net Tue Nov 7 14:53:40 2023 From: mrp at mrp.net (Mark Prior) Date: Tue, 7 Nov 2023 14:23:40 +1030 Subject: [AusNOG] IPv6 status in ANZ Message-ID: It's been about seven years since I last nagged at AusNOG about the poor state of IPv6 adoption in Oz and it hasn't got any better since :-( And I don't mean the plumbing, I mean the services, I have recently rewritten the script I use to look for IPv6 services (web, mail, dns, ntp, xmpp and sip) and as part of that process I leveraged PeeringDB to find all the domains that claim to connect to an IX in either Australia or New Zealand. As a result I now have two sections on the web page that collect them all together. As an added bonus I also look for DNSSEC, and its deployment makes IPv6 look like a roaring success! The Australian IX member section is and the New Zealand IX member section is Of course there will be organisations in both lists but I only probe them once (a day). Start from the top of the page if you want a better understanding of what is being checked (and see some graphs) and check out the link under the domain name to find "diagnostics" about the tests. Regards, Mark. From francisfides at mailup.net Wed Nov 8 09:26:26 2023 From: francisfides at mailup.net (francisfides at mailup.net) Date: Wed, 08 Nov 2023 08:26:26 +1000 Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? Message-ID: Morning all, Hope the chaos isn't too hard on your work/family. I have had trouble with a couple of SMS verifications coming through to me, my Telstra number. Is this related? Any general banter around the downtime would be fine too - looks like it all began at 4.07am AEDT? Cheers -- francisfides at mailup.net From joseph at goldman.id.au Wed Nov 8 09:29:14 2023 From: joseph at goldman.id.au (Joseph Goldman) Date: Tue, 07 Nov 2023 22:29:14 +0000 Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? In-Reply-To: References: Message-ID: Was actually wondering why there wasn't a thread. My guess would be for 2FA SMS's not coming through, that the SMS provider the software/site is using relies on Optus to send the message. I would also have to imagine the outage is a bad config pushed out - surely a hardware failure or fibre cut wouldn't cause such a big outage. Bad day to have your name on the last push :/. ------ Original Message ------ From: francisfides at mailup.net To: AusNOG at lists.ausnog.net Sent: 8/11/2023 9:26:26 AM Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? >Morning all, >Hope the chaos isn't too hard on your work/family. >I have had trouble with a couple of SMS verifications coming through to me, my Telstra number. Is this related? > >Any general banter around the downtime would be fine too - looks like it all began at 4.07am AEDT? > >Cheers > >-- > >francisfides at mailup.net >_______________________________________________ >AusNOG mailing list >AusNOG at lists.ausnog.net >https://lists.ausnog.net/mailman/listinfo/ausnog > From shaun at peeringdb.com Wed Nov 8 09:36:26 2023 From: shaun at peeringdb.com (Shaun Coffey) Date: Wed, 8 Nov 2023 09:36:26 +1100 Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? In-Reply-To: References: Message-ID: This was the update a bit after 7:30: The suspected root cause of the issue lies with the route reflectors, which are currently handling an excessive number of routes, leading to session shutdown and a complete traffic halt. Our on-site technician is actively prioritising establishing a console connection. Rest assured that said technician is also being provided additional technical support remotely. Kindly expect next update to be provided by 9AM AEDT if not sooner. On Wed, 8 Nov 2023 at 9:29?am, Joseph Goldman wrote: > Was actually wondering why there wasn't a thread. > > My guess would be for 2FA SMS's not coming through, that the SMS > provider the software/site is using relies on Optus to send the message. > > I would also have to imagine the outage is a bad config pushed out - > surely a hardware failure or fibre cut wouldn't cause such a big outage. > Bad day to have your name on the last push :/. > > ------ Original Message ------ > From: francisfides at mailup.net > To: AusNOG at lists.ausnog.net > Sent: 8/11/2023 9:26:26 AM > Subject: [AusNOG] Optus downtime chat + affecting SMS verification to > Telstra? > > >Morning all, > >Hope the chaos isn't too hard on your work/family. > >I have had trouble with a couple of SMS verifications coming through to > me, my Telstra number. Is this related? > > > >Any general banter around the downtime would be fine too - looks like it > all began at 4.07am AEDT? > > > >Cheers > > > >-- > > > >francisfides at mailup.net > >_______________________________________________ > >AusNOG mailing list > >AusNOG at lists.ausnog.net > >https://lists.ausnog.net/mailman/listinfo/ausnog > > > > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dazzagibbs at gmail.com Wed Nov 8 09:39:22 2023 From: dazzagibbs at gmail.com (DaZZa) Date: Wed, 8 Nov 2023 09:39:22 +1100 Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? In-Reply-To: References: Message-ID: The Optus hamster finally died of old age. I would suggest your SMS issues would be caused by whoever is issuing the SMS using Optus - not so much by the Telstra end receiving it. Anecdotally, Optus enterprise/wholesale appears to be still functional - at least my link appears to be working fine - and my BGP advertisements are still being seen overseas - seems to be only NBN and mobile based services which are busted D On Wed, 8 Nov 2023 at 09:27, wrote: > > Morning all, > Hope the chaos isn't too hard on your work/family. > I have had trouble with a couple of SMS verifications coming through to me, my Telstra number. Is this related? > > Any general banter around the downtime would be fine too - looks like it all began at 4.07am AEDT? > > Cheers > > -- > > francisfides at mailup.net > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog -- veg?e?tar?i?an: Ancient tribal slang for the village idiot who can't hunt, fish or ride From jaedwards at gmail.com Wed Nov 8 10:01:08 2023 From: jaedwards at gmail.com (John Edwards) Date: Wed, 8 Nov 2023 09:31:08 +1030 Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? In-Reply-To: References: Message-ID: The 4am Wednesday morning outage start looks suspiciously like a firmware upgrade window. I note that Optus devices where I am are showing "SoS" which indicates the tower is unable to reach the location register, which presumably is on a private network and indicative of a pretty major fault rather than just IP. John On Wed, 8 Nov 2023 at 09:10, DaZZa wrote: > The Optus hamster finally died of old age. > > I would suggest your SMS issues would be caused by whoever is issuing > the SMS using Optus - not so much by the Telstra end receiving it. > > Anecdotally, Optus enterprise/wholesale appears to be still functional > - at least my link appears to be working fine - and my BGP > advertisements are still being seen overseas - seems to be only NBN > and mobile based services which are busted > > D > > On Wed, 8 Nov 2023 at 09:27, wrote: > > > > Morning all, > > Hope the chaos isn't too hard on your work/family. > > I have had trouble with a couple of SMS verifications coming through to > me, my Telstra number. Is this related? > > > > Any general banter around the downtime would be fine too - looks like it > all began at 4.07am AEDT? > > > > Cheers > > > > -- > > > > francisfides at mailup.net > > _______________________________________________ > > AusNOG mailing list > > AusNOG at lists.ausnog.net > > https://lists.ausnog.net/mailman/listinfo/ausnog > > > > -- > veg?e?tar?i?an: > Ancient tribal slang for the village idiot who can't hunt, fish or ride > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dazzagibbs at gmail.com Wed Nov 8 10:14:38 2023 From: dazzagibbs at gmail.com (DaZZa) Date: Wed, 8 Nov 2023 10:14:38 +1100 Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? In-Reply-To: References: Message-ID: Yeah, I'd be willing to bet that it's a change which wasn't thoroughly tested before being rolled out, and which had an inadequate backout plan. Interestingly, my Optus mobile actually had a valid connection for a short time - wasn't able to actually DO anything, but was connected to the OPtus network - but it's now gone to "SOS" mode. D On Wed, 8 Nov 2023 at 10:01, John Edwards wrote: > > The 4am Wednesday morning outage start looks suspiciously like a firmware upgrade window. > > I note that Optus devices where I am are showing "SoS" which indicates the tower is unable to reach the location register, which presumably is on a private network and indicative of a pretty major fault rather than just IP. > > John > > > On Wed, 8 Nov 2023 at 09:10, DaZZa wrote: >> >> The Optus hamster finally died of old age. >> >> I would suggest your SMS issues would be caused by whoever is issuing >> the SMS using Optus - not so much by the Telstra end receiving it. >> >> Anecdotally, Optus enterprise/wholesale appears to be still functional >> - at least my link appears to be working fine - and my BGP >> advertisements are still being seen overseas - seems to be only NBN >> and mobile based services which are busted >> >> D >> >> On Wed, 8 Nov 2023 at 09:27, wrote: >> > >> > Morning all, >> > Hope the chaos isn't too hard on your work/family. >> > I have had trouble with a couple of SMS verifications coming through to me, my Telstra number. Is this related? >> > >> > Any general banter around the downtime would be fine too - looks like it all began at 4.07am AEDT? >> > >> > Cheers >> > >> > -- >> > >> > francisfides at mailup.net >> > _______________________________________________ >> > AusNOG mailing list >> > AusNOG at lists.ausnog.net >> > https://lists.ausnog.net/mailman/listinfo/ausnog >> >> >> >> -- >> veg?e?tar?i?an: >> Ancient tribal slang for the village idiot who can't hunt, fish or ride >> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog -- veg?e?tar?i?an: Ancient tribal slang for the village idiot who can't hunt, fish or ride From bb.ausnog at bb.cactii.net Wed Nov 8 10:27:05 2023 From: bb.ausnog at bb.cactii.net (Ben Buxton) Date: Wed, 8 Nov 2023 10:27:05 +1100 Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? In-Reply-To: References: Message-ID: On Wed, 8 Nov 2023 at 09:29, Joseph Goldman wrote: > > I would also have to imagine the outage is a bad config pushed out - > surely a hardware failure or fibre cut wouldn't cause such a big outage. > Bad day to have your name on the last push :/. > In a blameless culture, I'l probably consider it a badge of honour to have my name on the push, but this is Optus so some engineer will be thrown under the bus. (In an even better culture though, no human would be manually performing any push...but again this is Optus...) BB > > ------ Original Message ------ > From: francisfides at mailup.net > To: AusNOG at lists.ausnog.net > Sent: 8/11/2023 9:26:26 AM > Subject: [AusNOG] Optus downtime chat + affecting SMS verification to > Telstra? > > >Morning all, > >Hope the chaos isn't too hard on your work/family. > >I have had trouble with a couple of SMS verifications coming through to > me, my Telstra number. Is this related? > > > >Any general banter around the downtime would be fine too - looks like it > all began at 4.07am AEDT? > > > >Cheers > > > >-- > > > >francisfides at mailup.net > >_______________________________________________ > >AusNOG mailing list > >AusNOG at lists.ausnog.net > >https://lists.ausnog.net/mailman/listinfo/ausnog > > > > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brad at bradleyamm.com Wed Nov 8 10:33:14 2023 From: brad at bradleyamm.com (Bradley Amm) Date: Tue, 7 Nov 2023 23:33:14 +0000 Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? In-Reply-To: References: Message-ID: https://www.abc.net.au/news/2023-11-08/optus-outage-live-blog/103076996 Get Outlook for iOS ________________________________ From: AusNOG on behalf of DaZZa Sent: Wednesday, November 8, 2023 6:39:22 AM To: francisfides at mailup.net Cc: AusNOG at lists.ausnog.net Subject: Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? The Optus hamster finally died of old age. I would suggest your SMS issues would be caused by whoever is issuing the SMS using Optus - not so much by the Telstra end receiving it. Anecdotally, Optus enterprise/wholesale appears to be still functional - at least my link appears to be working fine - and my BGP advertisements are still being seen overseas - seems to be only NBN and mobile based services which are busted D On Wed, 8 Nov 2023 at 09:27, wrote: > > Morning all, > Hope the chaos isn't too hard on your work/family. > I have had trouble with a couple of SMS verifications coming through to me, my Telstra number. Is this related? > > Any general banter around the downtime would be fine too - looks like it all began at 4.07am AEDT? > > Cheers > > -- > > francisfides at mailup.net > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog -- veg?e?tar?i?an: Ancient tribal slang for the village idiot who can't hunt, fish or ride _______________________________________________ AusNOG mailing list AusNOG at lists.ausnog.net https://lists.ausnog.net/mailman/listinfo/ausnog -------------- next part -------------- An HTML attachment was scrubbed... URL: From brad at bradleyamm.com Wed Nov 8 10:42:27 2023 From: brad at bradleyamm.com (Bradley Amm) Date: Tue, 7 Nov 2023 23:42:27 +0000 Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? In-Reply-To: References: Message-ID: Maybe they were annoyed at glen maxwell getting 200 Get Outlook for iOS ________________________________ From: AusNOG on behalf of francisfides at mailup.net Sent: Wednesday, November 8, 2023 6:26:26 AM To: AusNOG at lists.ausnog.net Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? Morning all, Hope the chaos isn't too hard on your work/family. I have had trouble with a couple of SMS verifications coming through to me, my Telstra number. Is this related? Any general banter around the downtime would be fine too - looks like it all began at 4.07am AEDT? Cheers -- francisfides at mailup.net _______________________________________________ AusNOG mailing list AusNOG at lists.ausnog.net https://lists.ausnog.net/mailman/listinfo/ausnog -------------- next part -------------- An HTML attachment was scrubbed... URL: From yahoo at vapourforge.com Wed Nov 8 10:49:10 2023 From: yahoo at vapourforge.com (yahoo) Date: Wed, 08 Nov 2023 10:49:10 +1100 Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? In-Reply-To: Message-ID: I happened to be looking at my phone when it happened. It said words to the effect of 'the carrier has turned off mobile data for a short time" with a little triangle symbol that I've never seen before.I guess it wasn't wrong depending on how you define shortSent from my Galaxy -------- Original message --------From: francisfides at mailup.net Date: 8/11/23 9:27 am (GMT+10:00) To: AusNOG at lists.ausnog.net Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? Morning all,Hope the chaos isn't too hard on your work/family. I have had trouble with a couple of SMS verifications coming through to me, my Telstra number. Is this related?Any general banter around the downtime would be fine too - looks like it all began at 4.07am AEDT?Cheers-- ? ? francisfides at mailup.net_______________________________________________AusNOG mailing listAusNOG at lists.ausnog.nethttps://lists.ausnog.net/mailman/listinfo/ausnog -------------- next part -------------- An HTML attachment was scrubbed... URL: From hudrob at gmail.com Wed Nov 8 10:54:54 2023 From: hudrob at gmail.com (Robert Hudson) Date: Wed, 8 Nov 2023 10:54:54 +1100 Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? In-Reply-To: References: Message-ID: Gotta say, that interview didn't give me any confidence at all. It makes me wonder if the CEO even knows what Optus does as its core business, or has had any sort of media training on how to communicate with customers and the public. "We'd like the media to help by directing people to our messaging" - that they can't access... On Wed, 8 Nov 2023 at 10:33, Bradley Amm wrote: > https://www.abc.net.au/news/2023-11-08/optus-outage-live-blog/103076996 > > Get Outlook for iOS > ------------------------------ > *From:* AusNOG on behalf of DaZZa < > dazzagibbs at gmail.com> > *Sent:* Wednesday, November 8, 2023 6:39:22 AM > *To:* francisfides at mailup.net > *Cc:* AusNOG at lists.ausnog.net > *Subject:* Re: [AusNOG] Optus downtime chat + affecting SMS verification > to Telstra? > > The Optus hamster finally died of old age. > > I would suggest your SMS issues would be caused by whoever is issuing > the SMS using Optus - not so much by the Telstra end receiving it. > > Anecdotally, Optus enterprise/wholesale appears to be still functional > - at least my link appears to be working fine - and my BGP > advertisements are still being seen overseas - seems to be only NBN > and mobile based services which are busted > > D > > On Wed, 8 Nov 2023 at 09:27, wrote: > > > > Morning all, > > Hope the chaos isn't too hard on your work/family. > > I have had trouble with a couple of SMS verifications coming through to > me, my Telstra number. Is this related? > > > > Any general banter around the downtime would be fine too - looks like it > all began at 4.07am AEDT? > > > > Cheers > > > > -- > > > > francisfides at mailup.net > > _______________________________________________ > > AusNOG mailing list > > AusNOG at lists.ausnog.net > > https://lists.ausnog.net/mailman/listinfo/ausnog > > > > -- > veg?e?tar?i?an: > Ancient tribal slang for the village idiot who can't hunt, fish or ride > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dazzagibbs at gmail.com Wed Nov 8 11:03:11 2023 From: dazzagibbs at gmail.com (DaZZa) Date: Wed, 8 Nov 2023 11:03:11 +1100 Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? In-Reply-To: References: Message-ID: I loved the scramble to avoid saying "We f***** up". And the dodge when asked "How many staff are on-shore and how manby overseas". I laughed She's a talking head. I didn't expect any information from her - what I took out of it is that 7 hours after this started, they *still* don't know what caused it, or how to fix it. D On Wed, 8 Nov 2023 at 10:55, Robert Hudson wrote: > > Gotta say, that interview didn't give me any confidence at all. It makes me wonder if the CEO even knows what Optus does as its core business, or has had any sort of media training on how to communicate with customers and the public. > > "We'd like the media to help by directing people to our messaging" - that they can't access... > > On Wed, 8 Nov 2023 at 10:33, Bradley Amm wrote: >> >> https://www.abc.net.au/news/2023-11-08/optus-outage-live-blog/103076996 >> >> Get Outlook for iOS >> ________________________________ >> From: AusNOG on behalf of DaZZa >> Sent: Wednesday, November 8, 2023 6:39:22 AM >> To: francisfides at mailup.net >> Cc: AusNOG at lists.ausnog.net >> Subject: Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? >> >> The Optus hamster finally died of old age. >> >> I would suggest your SMS issues would be caused by whoever is issuing >> the SMS using Optus - not so much by the Telstra end receiving it. >> >> Anecdotally, Optus enterprise/wholesale appears to be still functional >> - at least my link appears to be working fine - and my BGP >> advertisements are still being seen overseas - seems to be only NBN >> and mobile based services which are busted >> >> D >> >> On Wed, 8 Nov 2023 at 09:27, wrote: >> > >> > Morning all, >> > Hope the chaos isn't too hard on your work/family. >> > I have had trouble with a couple of SMS verifications coming through to me, my Telstra number. Is this related? >> > >> > Any general banter around the downtime would be fine too - looks like it all began at 4.07am AEDT? >> > >> > Cheers >> > >> > -- >> > >> > francisfides at mailup.net >> > _______________________________________________ >> > AusNOG mailing list >> > AusNOG at lists.ausnog.net >> > https://lists.ausnog.net/mailman/listinfo/ausnog >> >> >> >> -- >> veg?e?tar?i?an: >> Ancient tribal slang for the village idiot who can't hunt, fish or ride >> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog >> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog -- veg?e?tar?i?an: Ancient tribal slang for the village idiot who can't hunt, fish or ride From bb.ausnog at bb.cactii.net Wed Nov 8 11:06:09 2023 From: bb.ausnog at bb.cactii.net (Ben Buxton) Date: Wed, 8 Nov 2023 11:06:09 +1100 Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? In-Reply-To: References: Message-ID: On Wed, 8 Nov 2023 at 10:14, DaZZa wrote: > Yeah, I'd be willing to bet that it's a change which wasn't thoroughly > tested before being rolled out, and which had an inadequate backout > plan. > Also, "Our on-site technician is actively prioritising establishing a console connection.". I mean come on, it's nearly 2024 and a [major] telco does not have remote console access? Whilst I'm looking forward to enthusiastically reading the PM, I'll have to book a physio appointment in advance due to neck strain from all the head shaking it'll likely induce. BB > > Interestingly, my Optus mobile actually had a valid connection for a > short time - wasn't able to actually DO anything, but was connected to > the OPtus network - but it's now gone to "SOS" mode. > > D > > On Wed, 8 Nov 2023 at 10:01, John Edwards wrote: > > > > The 4am Wednesday morning outage start looks suspiciously like a > firmware upgrade window. > > > > I note that Optus devices where I am are showing "SoS" which indicates > the tower is unable to reach the location register, which presumably is on > a private network and indicative of a pretty major fault rather than just > IP. > > > > John > > > > > > On Wed, 8 Nov 2023 at 09:10, DaZZa wrote: > >> > >> The Optus hamster finally died of old age. > >> > >> I would suggest your SMS issues would be caused by whoever is issuing > >> the SMS using Optus - not so much by the Telstra end receiving it. > >> > >> Anecdotally, Optus enterprise/wholesale appears to be still functional > >> - at least my link appears to be working fine - and my BGP > >> advertisements are still being seen overseas - seems to be only NBN > >> and mobile based services which are busted > >> > >> D > >> > >> On Wed, 8 Nov 2023 at 09:27, wrote: > >> > > >> > Morning all, > >> > Hope the chaos isn't too hard on your work/family. > >> > I have had trouble with a couple of SMS verifications coming through > to me, my Telstra number. Is this related? > >> > > >> > Any general banter around the downtime would be fine too - looks like > it all began at 4.07am AEDT? > >> > > >> > Cheers > >> > > >> > -- > >> > > >> > francisfides at mailup.net > >> > _______________________________________________ > >> > AusNOG mailing list > >> > AusNOG at lists.ausnog.net > >> > https://lists.ausnog.net/mailman/listinfo/ausnog > >> > >> > >> > >> -- > >> veg?e?tar?i?an: > >> Ancient tribal slang for the village idiot who can't hunt, fish or ride > >> _______________________________________________ > >> AusNOG mailing list > >> AusNOG at lists.ausnog.net > >> https://lists.ausnog.net/mailman/listinfo/ausnog > > > > -- > veg?e?tar?i?an: > Ancient tribal slang for the village idiot who can't hunt, fish or ride > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njenkin at gmail.com Wed Nov 8 11:10:26 2023 From: njenkin at gmail.com (Nick Jenkin) Date: Wed, 8 Nov 2023 11:10:26 +1100 Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? In-Reply-To: References: Message-ID: Just a few announcements... https://radar.cloudflare.com/traffic/as4804?dateRange=2d On Wed, 8 Nov 2023 at 11:06, Ben Buxton wrote: > > > On Wed, 8 Nov 2023 at 10:14, DaZZa wrote: > >> Yeah, I'd be willing to bet that it's a change which wasn't thoroughly >> tested before being rolled out, and which had an inadequate backout >> plan. >> > > Also, "Our on-site technician is actively prioritising establishing a > console connection.". > > I mean come on, it's nearly 2024 and a [major] telco does not have remote > console access? Whilst I'm > looking forward to enthusiastically reading the PM, I'll have to book a > physio appointment in advance due to > neck strain from all the head shaking it'll likely induce. > > BB > > > >> >> Interestingly, my Optus mobile actually had a valid connection for a >> short time - wasn't able to actually DO anything, but was connected to >> the OPtus network - but it's now gone to "SOS" mode. >> >> D >> >> On Wed, 8 Nov 2023 at 10:01, John Edwards wrote: >> > >> > The 4am Wednesday morning outage start looks suspiciously like a >> firmware upgrade window. >> > >> > I note that Optus devices where I am are showing "SoS" which indicates >> the tower is unable to reach the location register, which presumably is on >> a private network and indicative of a pretty major fault rather than just >> IP. >> > >> > John >> > >> > >> > On Wed, 8 Nov 2023 at 09:10, DaZZa wrote: >> >> >> >> The Optus hamster finally died of old age. >> >> >> >> I would suggest your SMS issues would be caused by whoever is issuing >> >> the SMS using Optus - not so much by the Telstra end receiving it. >> >> >> >> Anecdotally, Optus enterprise/wholesale appears to be still functional >> >> - at least my link appears to be working fine - and my BGP >> >> advertisements are still being seen overseas - seems to be only NBN >> >> and mobile based services which are busted >> >> >> >> D >> >> >> >> On Wed, 8 Nov 2023 at 09:27, wrote: >> >> > >> >> > Morning all, >> >> > Hope the chaos isn't too hard on your work/family. >> >> > I have had trouble with a couple of SMS verifications coming through >> to me, my Telstra number. Is this related? >> >> > >> >> > Any general banter around the downtime would be fine too - looks >> like it all began at 4.07am AEDT? >> >> > >> >> > Cheers >> >> > >> >> > -- >> >> > >> >> > francisfides at mailup.net >> >> > _______________________________________________ >> >> > AusNOG mailing list >> >> > AusNOG at lists.ausnog.net >> >> > https://lists.ausnog.net/mailman/listinfo/ausnog >> >> >> >> >> >> >> >> -- >> >> veg?e?tar?i?an: >> >> Ancient tribal slang for the village idiot who can't hunt, fish or ride >> >> _______________________________________________ >> >> AusNOG mailing list >> >> AusNOG at lists.ausnog.net >> >> https://lists.ausnog.net/mailman/listinfo/ausnog >> >> >> >> -- >> veg?e?tar?i?an: >> Ancient tribal slang for the village idiot who can't hunt, fish or ride >> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog >> > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marka at isc.org Wed Nov 8 12:13:28 2023 From: marka at isc.org (Mark Andrews) Date: Wed, 8 Nov 2023 02:13:28 +0100 Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? In-Reply-To: References: Message-ID: I?m wondering if it is leaking over to Vodafone as International Roaming has failed for me here in Prague at about the same time. All I?m getting is ?no serv?ce? even after forcing the carrier to Vodafone CZ which I?ve been using the last 8 days. -- Mark Andrews > On 8 Nov 2023, at 01:10, Nick Jenkin wrote: > > ? > Just a few announcements... https://radar.cloudflare.com/traffic/as4804?dateRange=2d > >> On Wed, 8 Nov 2023 at 11:06, Ben Buxton wrote: >> >> >>> On Wed, 8 Nov 2023 at 10:14, DaZZa wrote: >>> Yeah, I'd be willing to bet that it's a change which wasn't thoroughly >>> tested before being rolled out, and which had an inadequate backout >>> plan. >> >> Also, "Our on-site technician is actively prioritising establishing a console connection.". >> >> I mean come on, it's nearly 2024 and a [major] telco does not have remote console access? Whilst I'm >> looking forward to enthusiastically reading the PM, I'll have to book a physio appointment in advance due to >> neck strain from all the head shaking it'll likely induce. >> >> BB >> >> >>> >>> Interestingly, my Optus mobile actually had a valid connection for a >>> short time - wasn't able to actually DO anything, but was connected to >>> the OPtus network - but it's now gone to "SOS" mode. >>> >>> D >>> >>> On Wed, 8 Nov 2023 at 10:01, John Edwards wrote: >>> > >>> > The 4am Wednesday morning outage start looks suspiciously like a firmware upgrade window. >>> > >>> > I note that Optus devices where I am are showing "SoS" which indicates the tower is unable to reach the location register, which presumably is on a private network and indicative of a pretty major fault rather than just IP. >>> > >>> > John >>> > >>> > >>> > On Wed, 8 Nov 2023 at 09:10, DaZZa wrote: >>> >> >>> >> The Optus hamster finally died of old age. >>> >> >>> >> I would suggest your SMS issues would be caused by whoever is issuing >>> >> the SMS using Optus - not so much by the Telstra end receiving it. >>> >> >>> >> Anecdotally, Optus enterprise/wholesale appears to be still functional >>> >> - at least my link appears to be working fine - and my BGP >>> >> advertisements are still being seen overseas - seems to be only NBN >>> >> and mobile based services which are busted >>> >> >>> >> D >>> >> >>> >> On Wed, 8 Nov 2023 at 09:27, wrote: >>> >> > >>> >> > Morning all, >>> >> > Hope the chaos isn't too hard on your work/family. >>> >> > I have had trouble with a couple of SMS verifications coming through to me, my Telstra number. Is this related? >>> >> > >>> >> > Any general banter around the downtime would be fine too - looks like it all began at 4.07am AEDT? >>> >> > >>> >> > Cheers >>> >> > >>> >> > -- >>> >> > >>> >> > francisfides at mailup.net >>> >> > _______________________________________________ >>> >> > AusNOG mailing list >>> >> > AusNOG at lists.ausnog.net >>> >> > https://lists.ausnog.net/mailman/listinfo/ausnog >>> >> >>> >> >>> >> >>> >> -- >>> >> veg?e?tar?i?an: >>> >> Ancient tribal slang for the village idiot who can't hunt, fish or ride >>> >> _______________________________________________ >>> >> AusNOG mailing list >>> >> AusNOG at lists.ausnog.net >>> >> https://lists.ausnog.net/mailman/listinfo/ausnog >>> >>> >>> >>> -- >>> veg?e?tar?i?an: >>> Ancient tribal slang for the village idiot who can't hunt, fish or ride >>> _______________________________________________ >>> AusNOG mailing list >>> AusNOG at lists.ausnog.net >>> https://lists.ausnog.net/mailman/listinfo/ausnog >> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog -------------- next part -------------- An HTML attachment was scrubbed... URL: From yahoo at vapourforge.com Wed Nov 8 12:14:52 2023 From: yahoo at vapourforge.com (Jake Anderson) Date: Wed, 8 Nov 2023 12:14:52 +1100 Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? In-Reply-To: References: Message-ID: <5caeccdd-7c69-4312-88b2-e1dd465d5cc6@vapourforge.com> Looks like they may getting some services back https://radar.cloudflare.com/as4804?dateStart=2023-11-07&dateEnd=2023-11-07 On 8/11/2023 9:26 am, francisfides at mailup.net wrote: > Morning all, > Hope the chaos isn't too hard on your work/family. > I have had trouble with a couple of SMS verifications coming through to me, my Telstra number. Is this related? > > Any general banter around the downtime would be fine too - looks like it all began at 4.07am AEDT? > > Cheers > From tdmalone at gmail.com Wed Nov 8 12:16:13 2023 From: tdmalone at gmail.com (Tim Malone) Date: Wed, 8 Nov 2023 12:16:13 +1100 Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? In-Reply-To: References: Message-ID: Cloudflare is also reporting BGP origin hijack warnings: https://radar.cloudflare.com/as7474?dateRange=2d (unsure if related though) On Wed, 8 Nov 2023 at 11:10, Nick Jenkin wrote: > Just a few announcements... > https://radar.cloudflare.com/traffic/as4804?dateRange=2d > > On Wed, 8 Nov 2023 at 11:06, Ben Buxton wrote: > >> >> >> On Wed, 8 Nov 2023 at 10:14, DaZZa wrote: >> >>> Yeah, I'd be willing to bet that it's a change which wasn't thoroughly >>> tested before being rolled out, and which had an inadequate backout >>> plan. >>> >> >> Also, "Our on-site technician is actively prioritising establishing a >> console connection.". >> >> I mean come on, it's nearly 2024 and a [major] telco does not have remote >> console access? Whilst I'm >> looking forward to enthusiastically reading the PM, I'll have to book a >> physio appointment in advance due to >> neck strain from all the head shaking it'll likely induce. >> >> BB >> >> >> >>> >>> Interestingly, my Optus mobile actually had a valid connection for a >>> short time - wasn't able to actually DO anything, but was connected to >>> the OPtus network - but it's now gone to "SOS" mode. >>> >>> D >>> >>> On Wed, 8 Nov 2023 at 10:01, John Edwards wrote: >>> > >>> > The 4am Wednesday morning outage start looks suspiciously like a >>> firmware upgrade window. >>> > >>> > I note that Optus devices where I am are showing "SoS" which indicates >>> the tower is unable to reach the location register, which presumably is on >>> a private network and indicative of a pretty major fault rather than just >>> IP. >>> > >>> > John >>> > >>> > >>> > On Wed, 8 Nov 2023 at 09:10, DaZZa wrote: >>> >> >>> >> The Optus hamster finally died of old age. >>> >> >>> >> I would suggest your SMS issues would be caused by whoever is issuing >>> >> the SMS using Optus - not so much by the Telstra end receiving it. >>> >> >>> >> Anecdotally, Optus enterprise/wholesale appears to be still functional >>> >> - at least my link appears to be working fine - and my BGP >>> >> advertisements are still being seen overseas - seems to be only NBN >>> >> and mobile based services which are busted >>> >> >>> >> D >>> >> >>> >> On Wed, 8 Nov 2023 at 09:27, wrote: >>> >> > >>> >> > Morning all, >>> >> > Hope the chaos isn't too hard on your work/family. >>> >> > I have had trouble with a couple of SMS verifications coming >>> through to me, my Telstra number. Is this related? >>> >> > >>> >> > Any general banter around the downtime would be fine too - looks >>> like it all began at 4.07am AEDT? >>> >> > >>> >> > Cheers >>> >> > >>> >> > -- >>> >> > >>> >> > francisfides at mailup.net >>> >> > _______________________________________________ >>> >> > AusNOG mailing list >>> >> > AusNOG at lists.ausnog.net >>> >> > https://lists.ausnog.net/mailman/listinfo/ausnog >>> >> >>> >> >>> >> >>> >> -- >>> >> veg?e?tar?i?an: >>> >> Ancient tribal slang for the village idiot who can't hunt, fish or >>> ride >>> >> _______________________________________________ >>> >> AusNOG mailing list >>> >> AusNOG at lists.ausnog.net >>> >> https://lists.ausnog.net/mailman/listinfo/ausnog >>> >>> >>> >>> -- >>> veg?e?tar?i?an: >>> Ancient tribal slang for the village idiot who can't hunt, fish or ride >>> _______________________________________________ >>> AusNOG mailing list >>> AusNOG at lists.ausnog.net >>> https://lists.ausnog.net/mailman/listinfo/ausnog >>> >> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog >> > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at thesysadmin.dev Wed Nov 8 12:27:40 2023 From: chris at thesysadmin.dev (Christopher Hawker) Date: Wed, 8 Nov 2023 01:27:40 +0000 Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? In-Reply-To: References: Message-ID: They weren't appearing earlier so I'd say it's new. Smells like BGP has shat itself. - ?CH ________________________________ From: AusNOG on behalf of Tim Malone Sent: Wednesday, November 8, 2023 12:16 PM To: Nick Jenkin Cc: ausnog at lists.ausnog.net Subject: Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? Cloudflare is also reporting BGP origin hijack warnings: https://radar.cloudflare.com/as7474?dateRange=2d (unsure if related though) On Wed, 8 Nov 2023 at 11:10, Nick Jenkin > wrote: Just a few announcements... https://radar.cloudflare.com/traffic/as4804?dateRange=2d On Wed, 8 Nov 2023 at 11:06, Ben Buxton > wrote: On Wed, 8 Nov 2023 at 10:14, DaZZa > wrote: Yeah, I'd be willing to bet that it's a change which wasn't thoroughly tested before being rolled out, and which had an inadequate backout plan. Also, "Our on-site technician is actively prioritising establishing a console connection.". I mean come on, it's nearly 2024 and a [major] telco does not have remote console access? Whilst I'm looking forward to enthusiastically reading the PM, I'll have to book a physio appointment in advance due to neck strain from all the head shaking it'll likely induce. BB Interestingly, my Optus mobile actually had a valid connection for a short time - wasn't able to actually DO anything, but was connected to the OPtus network - but it's now gone to "SOS" mode. D On Wed, 8 Nov 2023 at 10:01, John Edwards > wrote: > > The 4am Wednesday morning outage start looks suspiciously like a firmware upgrade window. > > I note that Optus devices where I am are showing "SoS" which indicates the tower is unable to reach the location register, which presumably is on a private network and indicative of a pretty major fault rather than just IP. > > John > > > On Wed, 8 Nov 2023 at 09:10, DaZZa > wrote: >> >> The Optus hamster finally died of old age. >> >> I would suggest your SMS issues would be caused by whoever is issuing >> the SMS using Optus - not so much by the Telstra end receiving it. >> >> Anecdotally, Optus enterprise/wholesale appears to be still functional >> - at least my link appears to be working fine - and my BGP >> advertisements are still being seen overseas - seems to be only NBN >> and mobile based services which are busted >> >> D >> >> On Wed, 8 Nov 2023 at 09:27, > wrote: >> > >> > Morning all, >> > Hope the chaos isn't too hard on your work/family. >> > I have had trouble with a couple of SMS verifications coming through to me, my Telstra number. Is this related? >> > >> > Any general banter around the downtime would be fine too - looks like it all began at 4.07am AEDT? >> > >> > Cheers >> > >> > -- >> > >> > francisfides at mailup.net >> > _______________________________________________ >> > AusNOG mailing list >> > AusNOG at lists.ausnog.net >> > https://lists.ausnog.net/mailman/listinfo/ausnog >> >> >> >> -- >> veg?e?tar?i?an: >> Ancient tribal slang for the village idiot who can't hunt, fish or ride >> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog -- veg?e?tar?i?an: Ancient tribal slang for the village idiot who can't hunt, fish or ride _______________________________________________ AusNOG mailing list AusNOG at lists.ausnog.net https://lists.ausnog.net/mailman/listinfo/ausnog _______________________________________________ AusNOG mailing list AusNOG at lists.ausnog.net https://lists.ausnog.net/mailman/listinfo/ausnog _______________________________________________ AusNOG mailing list AusNOG at lists.ausnog.net https://lists.ausnog.net/mailman/listinfo/ausnog -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaedwards at gmail.com Wed Nov 8 12:44:30 2023 From: jaedwards at gmail.com (John Edwards) Date: Wed, 8 Nov 2023 12:14:30 +1030 Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? In-Reply-To: References: Message-ID: Those who believe it's a BGP issue might want to review https://www.cidr-report.org/cgi-bin/plota?file=%2fvar%2fdata%2fbgp%2fas2.0%2fbgp%2dactive%2etxt&descr=Active%20BGP%20entries%20%28FIB%29&ylabel=Active%20BGP%20entries%20%28FIB%29&with=step This shows a global table getting dangerously close to a 1024K boundary found in some router FIBs. John On Wed, 8 Nov 2023 at 11:57, Christopher Hawker wrote: > They weren't appearing earlier so I'd say it's new. Smells like BGP has > shat itself. > > - ?CH > ------------------------------ > *From:* AusNOG on behalf of Tim Malone < > tdmalone at gmail.com> > *Sent:* Wednesday, November 8, 2023 12:16 PM > *To:* Nick Jenkin > *Cc:* ausnog at lists.ausnog.net > *Subject:* Re: [AusNOG] Optus downtime chat + affecting SMS verification > to Telstra? > > Cloudflare is also reporting BGP origin hijack warnings: > https://radar.cloudflare.com/as7474?dateRange=2d > (unsure if related though) > > > On Wed, 8 Nov 2023 at 11:10, Nick Jenkin wrote: > > Just a few announcements... > https://radar.cloudflare.com/traffic/as4804?dateRange=2d > > On Wed, 8 Nov 2023 at 11:06, Ben Buxton wrote: > > > > On Wed, 8 Nov 2023 at 10:14, DaZZa wrote: > > Yeah, I'd be willing to bet that it's a change which wasn't thoroughly > tested before being rolled out, and which had an inadequate backout > plan. > > > Also, "Our on-site technician is actively prioritising establishing a > console connection.". > > I mean come on, it's nearly 2024 and a [major] telco does not have remote > console access? Whilst I'm > looking forward to enthusiastically reading the PM, I'll have to book a > physio appointment in advance due to > neck strain from all the head shaking it'll likely induce. > > BB > > > > > Interestingly, my Optus mobile actually had a valid connection for a > short time - wasn't able to actually DO anything, but was connected to > the OPtus network - but it's now gone to "SOS" mode. > > D > > On Wed, 8 Nov 2023 at 10:01, John Edwards wrote: > > > > The 4am Wednesday morning outage start looks suspiciously like a > firmware upgrade window. > > > > I note that Optus devices where I am are showing "SoS" which indicates > the tower is unable to reach the location register, which presumably is on > a private network and indicative of a pretty major fault rather than just > IP. > > > > John > > > > > > On Wed, 8 Nov 2023 at 09:10, DaZZa wrote: > >> > >> The Optus hamster finally died of old age. > >> > >> I would suggest your SMS issues would be caused by whoever is issuing > >> the SMS using Optus - not so much by the Telstra end receiving it. > >> > >> Anecdotally, Optus enterprise/wholesale appears to be still functional > >> - at least my link appears to be working fine - and my BGP > >> advertisements are still being seen overseas - seems to be only NBN > >> and mobile based services which are busted > >> > >> D > >> > >> On Wed, 8 Nov 2023 at 09:27, wrote: > >> > > >> > Morning all, > >> > Hope the chaos isn't too hard on your work/family. > >> > I have had trouble with a couple of SMS verifications coming through > to me, my Telstra number. Is this related? > >> > > >> > Any general banter around the downtime would be fine too - looks like > it all began at 4.07am AEDT? > >> > > >> > Cheers > >> > > >> > -- > >> > > >> > francisfides at mailup.net > >> > _______________________________________________ > >> > AusNOG mailing list > >> > AusNOG at lists.ausnog.net > >> > https://lists.ausnog.net/mailman/listinfo/ausnog > >> > >> > >> > >> -- > >> veg?e?tar?i?an: > >> Ancient tribal slang for the village idiot who can't hunt, fish or ride > >> _______________________________________________ > >> AusNOG mailing list > >> AusNOG at lists.ausnog.net > >> https://lists.ausnog.net/mailman/listinfo/ausnog > > > > -- > veg?e?tar?i?an: > Ancient tribal slang for the village idiot who can't hunt, fish or ride > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dazzagibbs at gmail.com Wed Nov 8 13:08:31 2023 From: dazzagibbs at gmail.com (DaZZa) Date: Wed, 8 Nov 2023 13:08:31 +1100 Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? In-Reply-To: <5caeccdd-7c69-4312-88b2-e1dd465d5cc6@vapourforge.com> References: <5caeccdd-7c69-4312-88b2-e1dd465d5cc6@vapourforge.com> Message-ID: The ABC's rolling coverage reports "Some services across fixed and mobile are now gradually being restored.", but it may take a "few hours" for everything to recover. Now we just get to wait for the report on the outage. it oughta be a doozy! D On Wed, 8 Nov 2023 at 12:15, Jake Anderson wrote: > > Looks like they may getting some services back > https://radar.cloudflare.com/as4804?dateStart=2023-11-07&dateEnd=2023-11-07 > > On 8/11/2023 9:26 am, francisfides at mailup.net wrote: > > Morning all, > > Hope the chaos isn't too hard on your work/family. > > I have had trouble with a couple of SMS verifications coming through to me, my Telstra number. Is this related? > > > > Any general banter around the downtime would be fine too - looks like it all began at 4.07am AEDT? > > > > Cheers > > > > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog -- veg?e?tar?i?an: Ancient tribal slang for the village idiot who can't hunt, fish or ride From jonathon at networkdynamics.com.au Wed Nov 8 16:59:21 2023 From: jonathon at networkdynamics.com.au (Jonathon Musters) Date: Wed, 8 Nov 2023 05:59:21 +0000 Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? In-Reply-To: References: <5caeccdd-7c69-4312-88b2-e1dd465d5cc6@vapourforge.com> Message-ID: Hey All, This sounds much like what happened in Canada with Rogers ? https://en.wikipedia.org/wiki/2022_Rogers_Communications_outage The same words seem to be coming from the top ? From: AusNOG on behalf of DaZZa Date: Wednesday, November 8, 2023 at 1:10?PM To: Jake Anderson Cc: ausnog at lists.ausnog.net Subject: Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? The ABC's rolling coverage reports "Some services across fixed and mobile are now gradually being restored.", but it may take a "few hours" for everything to recover. Now we just get to wait for the report on the outage. it oughta be a doozy! D On Wed, 8 Nov 2023 at 12:15, Jake Anderson wrote: > > Looks like they may getting some services back > https://radar.cloudflare.com/as4804?dateStart=2023-11-07&dateEnd=2023-11-07 > > On 8/11/2023 9:26 am, francisfides at mailup.net wrote: > > Morning all, > > Hope the chaos isn't too hard on your work/family. > > I have had trouble with a couple of SMS verifications coming through to me, my Telstra number. Is this related? > > > > Any general banter around the downtime would be fine too - looks like it all began at 4.07am AEDT? > > > > Cheers > > > > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog -- veg?e?tar?i?an: Ancient tribal slang for the village idiot who can't hunt, fish or ride _______________________________________________ AusNOG mailing list AusNOG at lists.ausnog.net https://lists.ausnog.net/mailman/listinfo/ausnog -------------- next part -------------- An HTML attachment was scrubbed... URL: From Steven.Waite at comtel.com.au Wed Nov 8 19:22:13 2023 From: Steven.Waite at comtel.com.au (Steven Waite) Date: Wed, 8 Nov 2023 08:22:13 +0000 Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? In-Reply-To: References: <5caeccdd-7c69-4312-88b2-e1dd465d5cc6@vapourforge.com> Message-ID: <427ecbdf3ad1439aa5cc5617fccf88fe@comtel.com.au> I am sure the outage report would be good reading. Was hearing a lot about overloaded route reflectors. Thanks Steve From: AusNOG On Behalf Of Jonathon Musters Sent: Wednesday, November 8, 2023 3:59 PM To: ausnog at lists.ausnog.net Cc: ausnog at lists.ausnog.net Subject: Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? Hey All, This sounds much like what happened in Canada with Rogers ? https://en.wikipedia.org/wiki/2022_Rogers_Communications_outage The same words seem to be coming from the top ? [https://d36urhup7zbd7q.cloudfront.net/f77068df-cfa4-420e-8b83-7db24e8ff55d/Network_Dynamics_Logo_09_102620.format_png.resize_200x.png] Jonathon Musters Network Dynamics Pty Ltd Phone 1300 768 249 Website networkdynamics.com Email jonathon at networkdynamics.com.au This email and any attachments are intended only for the use of the recipient and may be confidential and/or legally privileged. Network Dynamics disclaims liability for any errors, omissions, viruses, loss and/or damage arising from using, opening or transmitting this email. If you are not the intended recipient you must not use, interfere with, disclose, copy or retain this email and you should notify the sender immediately by return email or by contacting Network Dynamics by telephone on 1300 768 249. All views and opinions expressed in this email message are the personal opinions of the author and do not represent those of the company. No liability can be held for any damages, however caused, to any recipients of this message. From: AusNOG > on behalf of DaZZa > Date: Wednesday, November 8, 2023 at 1:10?PM To: Jake Anderson > Cc: ausnog at lists.ausnog.net > Subject: Re: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? The ABC's rolling coverage reports "Some services across fixed and mobile are now gradually being restored.", but it may take a "few hours" for everything to recover. Now we just get to wait for the report on the outage. it oughta be a doozy! D On Wed, 8 Nov 2023 at 12:15, Jake Anderson > wrote: > > Looks like they may getting some services back > https://radar.cloudflare.com/as4804?dateStart=2023-11-07&dateEnd=2023-11-07 > > On 8/11/2023 9:26 am, francisfides at mailup.net wrote: > > Morning all, > > Hope the chaos isn't too hard on your work/family. > > I have had trouble with a couple of SMS verifications coming through to me, my Telstra number. Is this related? > > > > Any general banter around the downtime would be fine too - looks like it all began at 4.07am AEDT? > > > > Cheers > > > > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog -- veg?e?tar?i?an: Ancient tribal slang for the village idiot who can't hunt, fish or ride _______________________________________________ AusNOG mailing list AusNOG at lists.ausnog.net https://lists.ausnog.net/mailman/listinfo/ausnog -------------- next part -------------- An HTML attachment was scrubbed... URL: From tom at snnap.net Wed Nov 8 21:29:07 2023 From: tom at snnap.net (Tom Storey) Date: Wed, 8 Nov 2023 10:29:07 +0000 Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? In-Reply-To: References: Message-ID: If their OOB phone lines or mobile broadband services were on-net they might as well not have had any to begin with. At a guess they probably were otherwise they wouldn't have had to send someone to physically stand in front of a device. On Wed, 8 Nov 2023, 00:06 Ben Buxton, wrote: > > > On Wed, 8 Nov 2023 at 10:14, DaZZa wrote: > >> Yeah, I'd be willing to bet that it's a change which wasn't thoroughly >> tested before being rolled out, and which had an inadequate backout >> plan. >> > > Also, "Our on-site technician is actively prioritising establishing a > console connection.". > > I mean come on, it's nearly 2024 and a [major] telco does not have remote > console access? Whilst I'm > looking forward to enthusiastically reading the PM, I'll have to book a > physio appointment in advance due to > neck strain from all the head shaking it'll likely induce. > > BB > > > >> >> Interestingly, my Optus mobile actually had a valid connection for a >> short time - wasn't able to actually DO anything, but was connected to >> the OPtus network - but it's now gone to "SOS" mode. >> >> D >> >> On Wed, 8 Nov 2023 at 10:01, John Edwards wrote: >> > >> > The 4am Wednesday morning outage start looks suspiciously like a >> firmware upgrade window. >> > >> > I note that Optus devices where I am are showing "SoS" which indicates >> the tower is unable to reach the location register, which presumably is on >> a private network and indicative of a pretty major fault rather than just >> IP. >> > >> > John >> > >> > >> > On Wed, 8 Nov 2023 at 09:10, DaZZa wrote: >> >> >> >> The Optus hamster finally died of old age. >> >> >> >> I would suggest your SMS issues would be caused by whoever is issuing >> >> the SMS using Optus - not so much by the Telstra end receiving it. >> >> >> >> Anecdotally, Optus enterprise/wholesale appears to be still functional >> >> - at least my link appears to be working fine - and my BGP >> >> advertisements are still being seen overseas - seems to be only NBN >> >> and mobile based services which are busted >> >> >> >> D >> >> >> >> On Wed, 8 Nov 2023 at 09:27, wrote: >> >> > >> >> > Morning all, >> >> > Hope the chaos isn't too hard on your work/family. >> >> > I have had trouble with a couple of SMS verifications coming through >> to me, my Telstra number. Is this related? >> >> > >> >> > Any general banter around the downtime would be fine too - looks >> like it all began at 4.07am AEDT? >> >> > >> >> > Cheers >> >> > >> >> > -- >> >> > >> >> > francisfides at mailup.net >> >> > _______________________________________________ >> >> > AusNOG mailing list >> >> > AusNOG at lists.ausnog.net >> >> > https://lists.ausnog.net/mailman/listinfo/ausnog >> >> >> >> >> >> >> >> -- >> >> veg?e?tar?i?an: >> >> Ancient tribal slang for the village idiot who can't hunt, fish or ride >> >> _______________________________________________ >> >> AusNOG mailing list >> >> AusNOG at lists.ausnog.net >> >> https://lists.ausnog.net/mailman/listinfo/ausnog >> >> >> >> -- >> veg?e?tar?i?an: >> Ancient tribal slang for the village idiot who can't hunt, fish or ride >> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog >> > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From the.damo at gmail.com Wed Nov 8 22:02:51 2023 From: the.damo at gmail.com (Damian Guppy) Date: Wed, 8 Nov 2023 19:02:51 +0800 Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? In-Reply-To: References: Message-ID: Any money on all the on call engineers having Optus mobiles and nbn so couldn?t be contacted? ?Damian On Wed, Nov 8, 2023 at 6:29?PM Tom Storey wrote: > If their OOB phone lines or mobile broadband services were on-net they > might as well not have had any to begin with. At a guess they probably were > otherwise they wouldn't have had to send someone to physically stand in > front of a device. > > On Wed, 8 Nov 2023, 00:06 Ben Buxton, wrote: > >> >> >> On Wed, 8 Nov 2023 at 10:14, DaZZa wrote: >> >>> Yeah, I'd be willing to bet that it's a change which wasn't thoroughly >>> tested before being rolled out, and which had an inadequate backout >>> plan. >>> >> >> Also, "Our on-site technician is actively prioritising establishing a >> console connection.". >> >> I mean come on, it's nearly 2024 and a [major] telco does not have remote >> console access? Whilst I'm >> looking forward to enthusiastically reading the PM, I'll have to book a >> physio appointment in advance due to >> neck strain from all the head shaking it'll likely induce. >> >> BB >> >> >> >>> >>> Interestingly, my Optus mobile actually had a valid connection for a >>> short time - wasn't able to actually DO anything, but was connected to >>> the OPtus network - but it's now gone to "SOS" mode. >>> >>> D >>> >>> On Wed, 8 Nov 2023 at 10:01, John Edwards wrote: >>> > >>> > The 4am Wednesday morning outage start looks suspiciously like a >>> firmware upgrade window. >>> > >>> > I note that Optus devices where I am are showing "SoS" which indicates >>> the tower is unable to reach the location register, which presumably is on >>> a private network and indicative of a pretty major fault rather than just >>> IP. >>> > >>> > John >>> > >>> > >>> > On Wed, 8 Nov 2023 at 09:10, DaZZa wrote: >>> >> >>> >> The Optus hamster finally died of old age. >>> >> >>> >> I would suggest your SMS issues would be caused by whoever is issuing >>> >> the SMS using Optus - not so much by the Telstra end receiving it. >>> >> >>> >> Anecdotally, Optus enterprise/wholesale appears to be still functional >>> >> - at least my link appears to be working fine - and my BGP >>> >> advertisements are still being seen overseas - seems to be only NBN >>> >> and mobile based services which are busted >>> >> >>> >> D >>> >> >>> >> On Wed, 8 Nov 2023 at 09:27, wrote: >>> >> > >>> >> > Morning all, >>> >> > Hope the chaos isn't too hard on your work/family. >>> >> > I have had trouble with a couple of SMS verifications coming >>> through to me, my Telstra number. Is this related? >>> >> > >>> >> > Any general banter around the downtime would be fine too - looks >>> like it all began at 4.07am AEDT? >>> >> > >>> >> > Cheers >>> >> > >>> >> > -- >>> >> > >>> >> > francisfides at mailup.net >>> >> > _______________________________________________ >>> >> > AusNOG mailing list >>> >> > AusNOG at lists.ausnog.net >>> >> > https://lists.ausnog.net/mailman/listinfo/ausnog >>> >> >>> >> >>> >> >>> >> -- >>> >> veg?e?tar?i?an: >>> >> Ancient tribal slang for the village idiot who can't hunt, fish or >>> ride >>> >> _______________________________________________ >>> >> AusNOG mailing list >>> >> AusNOG at lists.ausnog.net >>> >> https://lists.ausnog.net/mailman/listinfo/ausnog >>> >>> >>> >>> -- >>> veg?e?tar?i?an: >>> Ancient tribal slang for the village idiot who can't hunt, fish or ride >>> _______________________________________________ >>> AusNOG mailing list >>> AusNOG at lists.ausnog.net >>> https://lists.ausnog.net/mailman/listinfo/ausnog >>> >> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog >> > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nathan.Brookfield at iperium.com.au Wed Nov 8 22:14:46 2023 From: Nathan.Brookfield at iperium.com.au (Nathan Brookfield) Date: Wed, 8 Nov 2023 11:14:46 +0000 Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? In-Reply-To: References: Message-ID: <46BFF89F-F5A5-4FC3-8B6A-E7BE34B8765E@iperium.com.au> My money is the on-call aren?t in Australia and couldn?t access the network or OOB because it was all on net haha! On 8 Nov 2023, at 21:03, Damian Guppy wrote: ? Any money on all the on call engineers having Optus mobiles and nbn so couldn?t be contacted? ?Damian On Wed, Nov 8, 2023 at 6:29?PM Tom Storey > wrote: If their OOB phone lines or mobile broadband services were on-net they might as well not have had any to begin with. At a guess they probably were otherwise they wouldn't have had to send someone to physically stand in front of a device. On Wed, 8 Nov 2023, 00:06 Ben Buxton, > wrote: On Wed, 8 Nov 2023 at 10:14, DaZZa > wrote: Yeah, I'd be willing to bet that it's a change which wasn't thoroughly tested before being rolled out, and which had an inadequate backout plan. Also, "Our on-site technician is actively prioritising establishing a console connection.". I mean come on, it's nearly 2024 and a [major] telco does not have remote console access? Whilst I'm looking forward to enthusiastically reading the PM, I'll have to book a physio appointment in advance due to neck strain from all the head shaking it'll likely induce. BB Interestingly, my Optus mobile actually had a valid connection for a short time - wasn't able to actually DO anything, but was connected to the OPtus network - but it's now gone to "SOS" mode. D On Wed, 8 Nov 2023 at 10:01, John Edwards > wrote: > > The 4am Wednesday morning outage start looks suspiciously like a firmware upgrade window. > > I note that Optus devices where I am are showing "SoS" which indicates the tower is unable to reach the location register, which presumably is on a private network and indicative of a pretty major fault rather than just IP. > > John > > > On Wed, 8 Nov 2023 at 09:10, DaZZa > wrote: >> >> The Optus hamster finally died of old age. >> >> I would suggest your SMS issues would be caused by whoever is issuing >> the SMS using Optus - not so much by the Telstra end receiving it. >> >> Anecdotally, Optus enterprise/wholesale appears to be still functional >> - at least my link appears to be working fine - and my BGP >> advertisements are still being seen overseas - seems to be only NBN >> and mobile based services which are busted >> >> D >> >> On Wed, 8 Nov 2023 at 09:27, > wrote: >> > >> > Morning all, >> > Hope the chaos isn't too hard on your work/family. >> > I have had trouble with a couple of SMS verifications coming through to me, my Telstra number. Is this related? >> > >> > Any general banter around the downtime would be fine too - looks like it all began at 4.07am AEDT? >> > >> > Cheers >> > >> > -- >> > >> > francisfides at mailup.net >> > _______________________________________________ >> > AusNOG mailing list >> > AusNOG at lists.ausnog.net >> > https://lists.ausnog.net/mailman/listinfo/ausnog >> >> >> >> -- >> veg?e?tar?i?an: >> Ancient tribal slang for the village idiot who can't hunt, fish or ride >> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog -- veg?e?tar?i?an: Ancient tribal slang for the village idiot who can't hunt, fish or ride _______________________________________________ AusNOG mailing list AusNOG at lists.ausnog.net https://lists.ausnog.net/mailman/listinfo/ausnog _______________________________________________ AusNOG mailing list AusNOG at lists.ausnog.net https://lists.ausnog.net/mailman/listinfo/ausnog _______________________________________________ AusNOG mailing list AusNOG at lists.ausnog.net https://lists.ausnog.net/mailman/listinfo/ausnog _______________________________________________ AusNOG mailing list AusNOG at lists.ausnog.net https://lists.ausnog.net/mailman/listinfo/ausnog -------------- next part -------------- An HTML attachment was scrubbed... URL: From brad at bradleyamm.com Wed Nov 8 23:06:41 2023 From: brad at bradleyamm.com (Bradley Amm) Date: Wed, 8 Nov 2023 20:06:41 +0800 Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? In-Reply-To: <46BFF89F-F5A5-4FC3-8B6A-E7BE34B8765E@iperium.com.au> References: <46BFF89F-F5A5-4FC3-8B6A-E7BE34B8765E@iperium.com.au> Message-ID: <9046DAFB-6470-440B-8663-49215EEE6431@bradleyamm.com> An HTML attachment was scrubbed... URL: From spoofer-info at caida.org Thu Nov 9 05:00:15 2023 From: spoofer-info at caida.org (CAIDA Spoofer Project) Date: Wed, 8 Nov 2023 10:00:15 -0800 Subject: [AusNOG] Spoofer Report for AusNOG for Oct 2023 Message-ID: <1699466415.299245.26004.nullmailer@caida.org> In response to feedback from operational security communities, CAIDA's source address validation measurement project (https://spoofer.caida.org) is automatically generating monthly reports of ASes originating prefixes in BGP for systems from which we received packets with a spoofed source address. We are publishing these reports to network and security operations lists in order to ensure this information reaches operational contacts in these ASes. This report summarises tests conducted within aus. Inferred improvements during Oct 2023: ASN Name Fixed-By 4826 VOCUS-BACKBONE 2023-10-02 Further information for the inferred remediation is available at: https://spoofer.caida.org/remedy.php Source Address Validation issues inferred during Oct 2023: ASN Name First-Spoofed Last-Spoofed 10214 PENTANET 2023-05-30 2023-10-31 Further information for these tests where we received spoofed packets is available at: https://spoofer.caida.org/recent_tests.php?country_include=aus&no_block=1 Please send any feedback or suggestions to spoofer-info at caida.org From phillip.grasso at gmail.com Thu Nov 9 06:18:11 2023 From: phillip.grasso at gmail.com (Phillip Grasso) Date: Thu, 9 Nov 2023 06:18:11 +1100 Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? In-Reply-To: References: Message-ID: > > I mean come on, it's nearly 2024 and a [major] telco does not have remote > console access? > If we send a poll out to this community, how many would be able to genuinely honestly answer: Do you have a console or appropriate control plane access into all your critical infrastructure? Do you have independant out of band that does not share any infrastructure with your current system(s) - with exemption for physical location and power. Do you have the ability to remote power control your devices? We know from the facebook outage in 2021 that they probably didn't have the above, so its not entirely uncommon for folks to have *proper independant* console and remote access. I empathize with the Optus team and their customers who have been negatively impacted by this incident. I sincerely hope that some positive outcomes can emerge from this situation, including: - Attention to critical infrastructure resilience - BGP clue increases - Incident management improves (I'm sure there's more). Network is a black box to most people and I think a large chunk of Australia now knows what it feels like to not have it. On Wed, 8 Nov 2023 at 11:06, Ben Buxton wrote: > > > On Wed, 8 Nov 2023 at 10:14, DaZZa wrote: > >> Yeah, I'd be willing to bet that it's a change which wasn't thoroughly >> tested before being rolled out, and which had an inadequate backout >> plan. >> > > Also, "Our on-site technician is actively prioritising establishing a > console connection.". > > I mean come on, it's nearly 2024 and a [major] telco does not have remote > console access? Whilst I'm > looking forward to enthusiastically reading the PM, I'll have to book a > physio appointment in advance due to > neck strain from all the head shaking it'll likely induce. > > BB > > > >> >> Interestingly, my Optus mobile actually had a valid connection for a >> short time - wasn't able to actually DO anything, but was connected to >> the OPtus network - but it's now gone to "SOS" mode. >> >> D >> >> On Wed, 8 Nov 2023 at 10:01, John Edwards wrote: >> > >> > The 4am Wednesday morning outage start looks suspiciously like a >> firmware upgrade window. >> > >> > I note that Optus devices where I am are showing "SoS" which indicates >> the tower is unable to reach the location register, which presumably is on >> a private network and indicative of a pretty major fault rather than just >> IP. >> > >> > John >> > >> > >> > On Wed, 8 Nov 2023 at 09:10, DaZZa wrote: >> >> >> >> The Optus hamster finally died of old age. >> >> >> >> I would suggest your SMS issues would be caused by whoever is issuing >> >> the SMS using Optus - not so much by the Telstra end receiving it. >> >> >> >> Anecdotally, Optus enterprise/wholesale appears to be still functional >> >> - at least my link appears to be working fine - and my BGP >> >> advertisements are still being seen overseas - seems to be only NBN >> >> and mobile based services which are busted >> >> >> >> D >> >> >> >> On Wed, 8 Nov 2023 at 09:27, wrote: >> >> > >> >> > Morning all, >> >> > Hope the chaos isn't too hard on your work/family. >> >> > I have had trouble with a couple of SMS verifications coming through >> to me, my Telstra number. Is this related? >> >> > >> >> > Any general banter around the downtime would be fine too - looks >> like it all began at 4.07am AEDT? >> >> > >> >> > Cheers >> >> > >> >> > -- >> >> > >> >> > francisfides at mailup.net >> >> > _______________________________________________ >> >> > AusNOG mailing list >> >> > AusNOG at lists.ausnog.net >> >> > https://lists.ausnog.net/mailman/listinfo/ausnog >> >> >> >> >> >> >> >> -- >> >> veg?e?tar?i?an: >> >> Ancient tribal slang for the village idiot who can't hunt, fish or ride >> >> _______________________________________________ >> >> AusNOG mailing list >> >> AusNOG at lists.ausnog.net >> >> https://lists.ausnog.net/mailman/listinfo/ausnog >> >> >> >> -- >> veg?e?tar?i?an: >> Ancient tribal slang for the village idiot who can't hunt, fish or ride >> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog >> > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dazzagibbs at gmail.com Thu Nov 9 08:15:39 2023 From: dazzagibbs at gmail.com (DaZZa) Date: Thu, 9 Nov 2023 08:15:39 +1100 Subject: [AusNOG] Optus downtime chat + affecting SMS verification to Telstra? In-Reply-To: References: Message-ID: I have all three you're asking about. But I'm very small potatoes compared to most of the members of this list, and my required remote footprint is correspondingly small, so it's easy to maintain. D On Thu, 9 Nov 2023 at 06:18, Phillip Grasso wrote: >> >> I mean come on, it's nearly 2024 and a [major] telco does not have remote console access? > > > If we send a poll out to this community, how many would be able to genuinely honestly answer: > > Do you have a console or appropriate control plane access into all your critical infrastructure? > Do you have independant out of band that does not share any infrastructure with your current system(s) - with exemption for physical location and power. > Do you have the ability to remote power control your devices? > > We know from the facebook outage in 2021 that they probably didn't have the above, so its not entirely uncommon for folks to have *proper independant* console and remote access. > > > I empathize with the Optus team and their customers who have been negatively impacted by this incident. I sincerely hope that some positive outcomes can emerge from this situation, including: > > - Attention to critical infrastructure resilience > - BGP clue increases > - Incident management improves > (I'm sure there's more). > > Network is a black box to most people and I think a large chunk of Australia now knows what it feels like to not have it. > > > On Wed, 8 Nov 2023 at 11:06, Ben Buxton wrote: >> >> >> >> On Wed, 8 Nov 2023 at 10:14, DaZZa wrote: >>> >>> Yeah, I'd be willing to bet that it's a change which wasn't thoroughly >>> tested before being rolled out, and which had an inadequate backout >>> plan. >> >> >> Also, "Our on-site technician is actively prioritising establishing a console connection.". >> >> I mean come on, it's nearly 2024 and a [major] telco does not have remote console access? Whilst I'm >> looking forward to enthusiastically reading the PM, I'll have to book a physio appointment in advance due to >> neck strain from all the head shaking it'll likely induce. >> >> BB >> >> >>> >>> >>> Interestingly, my Optus mobile actually had a valid connection for a >>> short time - wasn't able to actually DO anything, but was connected to >>> the OPtus network - but it's now gone to "SOS" mode. >>> >>> D >>> >>> On Wed, 8 Nov 2023 at 10:01, John Edwards wrote: >>> > >>> > The 4am Wednesday morning outage start looks suspiciously like a firmware upgrade window. >>> > >>> > I note that Optus devices where I am are showing "SoS" which indicates the tower is unable to reach the location register, which presumably is on a private network and indicative of a pretty major fault rather than just IP. >>> > >>> > John >>> > >>> > >>> > On Wed, 8 Nov 2023 at 09:10, DaZZa wrote: >>> >> >>> >> The Optus hamster finally died of old age. >>> >> >>> >> I would suggest your SMS issues would be caused by whoever is issuing >>> >> the SMS using Optus - not so much by the Telstra end receiving it. >>> >> >>> >> Anecdotally, Optus enterprise/wholesale appears to be still functional >>> >> - at least my link appears to be working fine - and my BGP >>> >> advertisements are still being seen overseas - seems to be only NBN >>> >> and mobile based services which are busted >>> >> >>> >> D >>> >> >>> >> On Wed, 8 Nov 2023 at 09:27, wrote: >>> >> > >>> >> > Morning all, >>> >> > Hope the chaos isn't too hard on your work/family. >>> >> > I have had trouble with a couple of SMS verifications coming through to me, my Telstra number. Is this related? >>> >> > >>> >> > Any general banter around the downtime would be fine too - looks like it all began at 4.07am AEDT? >>> >> > >>> >> > Cheers >>> >> > >>> >> > -- >>> >> > >>> >> > francisfides at mailup.net >>> >> > _______________________________________________ >>> >> > AusNOG mailing list >>> >> > AusNOG at lists.ausnog.net >>> >> > https://lists.ausnog.net/mailman/listinfo/ausnog >>> >> >>> >> >>> >> >>> >> -- >>> >> veg?e?tar?i?an: >>> >> Ancient tribal slang for the village idiot who can't hunt, fish or ride >>> >> _______________________________________________ >>> >> AusNOG mailing list >>> >> AusNOG at lists.ausnog.net >>> >> https://lists.ausnog.net/mailman/listinfo/ausnog >>> >>> >>> >>> -- >>> veg?e?tar?i?an: >>> Ancient tribal slang for the village idiot who can't hunt, fish or ride >>> _______________________________________________ >>> AusNOG mailing list >>> AusNOG at lists.ausnog.net >>> https://lists.ausnog.net/mailman/listinfo/ausnog >> >> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog -- veg?e?tar?i?an: Ancient tribal slang for the village idiot who can't hunt, fish or ride From Andres.Miedzowicz at au.logicalis.com Thu Nov 9 11:14:25 2023 From: Andres.Miedzowicz at au.logicalis.com (Andres Miedzowicz) Date: Thu, 9 Nov 2023 00:14:25 +0000 Subject: [AusNOG] DCB in non-converged networks Message-ID: Hello everyone, I wanted to get some opinions on the use of DCB and its associated protocols in a storage-only (iSCSI), non-converged network. Any thoughts about the pros and cons of enabling DCB in a scenario where 100% of the traffic on a switch is bi-directional iSCSI storage (virtual machines and backups)? Thanks in advance. Andres -------------- next part -------------- An HTML attachment was scrubbed... URL: From luke at iggleden.com Thu Nov 9 12:19:43 2023 From: luke at iggleden.com (Luke Iggleden) Date: Thu, 9 Nov 2023 12:19:43 +1100 Subject: [AusNOG] DCB in non-converged networks In-Reply-To: References: Message-ID: <89502812-dbd2-4de2-8e5f-6c5a0bb296ac@iggleden.com> Hi Andres, Unless you are running other services on the switch it's not useful. Typically these are the only useful changes: Jumbo Frames (YMMV), depends on vendor. Flow Control on (so hosts can issue back off - hopefully without dropping frames) Depending on the switch, buffer tuning. Don't use control plane things, like MLAG, Stacking, STP, etc etc. Flat fabric. Cheers, Luke Iggleden On 9/11/2023 11:14 am, Andres Miedzowicz wrote: > > Hello everyone, > > I wanted to get some opinions on the use of DCB and its associated > protocols in a storage-only (iSCSI), non-converged network. Any > thoughts about the pros and cons of enabling DCB in a scenario where > 100% of the traffic on a switch is bi-directional iSCSI storage > (virtual machines and backups)? > > Thanks in advance. > > Andres > > > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog -------------- next part -------------- An HTML attachment was scrubbed... URL: From hudrob at gmail.com Thu Nov 9 13:18:43 2023 From: hudrob at gmail.com (Robert Hudson) Date: Thu, 9 Nov 2023 13:18:43 +1100 Subject: [AusNOG] DCB in non-converged networks In-Reply-To: <89502812-dbd2-4de2-8e5f-6c5a0bb296ac@iggleden.com> References: <89502812-dbd2-4de2-8e5f-6c5a0bb296ac@iggleden.com> Message-ID: I largely agree with Luke. Given you're on a dedicated iSCSI network, keep it simple. DCB and other services will only add things that you'll later need to troubleshoot and eliminate as the root cause of network issues on your iSCSI network when they invariably happen (it's rare that I've come across a well and consistently configured iSCSI network, and I've been playing in that space since the mid 2000s). Chances are your OS/hypervisor vendor of choice publishes best practices for how to configure DCB - but as noted, DCB is specifically there to deal with converged networks (where your iSCSI traffic is sharing an ethernet fabric with other traffic types), and you don't seem to have that situation. Jumbo frames help in busy iSCSI networks by increasing throughput - but you need to make sure every device from one end of the communications to the other fully supports it. Again, follow vendor advice here. Getting this wrong can cause all sorts of "fun". Flow control, buffer tuning (large buffers tend to help with iSCSI traffic), etc, can all help to eke out a few more small percentage points of performance, but again, the further you drift from the KISS principle, the more fun you're likely to have troubleshooting later. Above all - set and document policy in all things, audit against that policy both at initial setup and for drift during the lifecycle of the environment. On Thu, 9 Nov 2023 at 12:20, Luke Iggleden wrote: > Hi Andres, > > Unless you are running other services on the switch it's not useful. > > Typically these are the only useful changes: > > Jumbo Frames (YMMV), depends on vendor. > > Flow Control on (so hosts can issue back off - hopefully without dropping > frames) > > Depending on the switch, buffer tuning. > > Don't use control plane things, like MLAG, Stacking, STP, etc etc. Flat > fabric. > > > Cheers, > > Luke Iggleden > > > On 9/11/2023 11:14 am, Andres Miedzowicz wrote: > > Hello everyone, > > > > I wanted to get some opinions on the use of DCB and its associated > protocols in a storage-only (iSCSI), non-converged network. Any thoughts > about the pros and cons of enabling DCB in a scenario where 100% of the > traffic on a switch is bi-directional iSCSI storage (virtual machines and > backups)? > > > > Thanks in advance. > > > > Andres > > _______________________________________________ > AusNOG mailing listAusNOG at lists.ausnog.nethttps://lists.ausnog.net/mailman/listinfo/ausnog > > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ptiggerdine at gmail.com Thu Nov 9 15:57:43 2023 From: ptiggerdine at gmail.com (Peter Tiggerdine) Date: Thu, 9 Nov 2023 14:57:43 +1000 Subject: [AusNOG] DCB in non-converged networks In-Reply-To: References: <89502812-dbd2-4de2-8e5f-6c5a0bb296ac@iggleden.com> Message-ID: I am going to buck the trend. DCB shouldn't be enabled and configured on a whim, however, if this is for backup, how long before these backups need to be rehydrated/booted in another DC or moved to a second DC for business availability/continuity purposes? As for tuning (jumbo-frames, QoS, Flow Control) the network to the requirements, this is still best practice (as we would for voice) - particularly since iSCSI overhead is heavy (in comparison to FCoE). I agree with the KISS principle, but DCBs aren't the place for that because of the complexity that they are (the same would be said for MPLS, VXLAN, etc) it's about business/regulatory requirements that drives a competitive edge. Think "If you're going to do it, do it properly" is more applicable. Regards, Peter Tiggerdine GPG Fingerprint: 2A3F EA19 F6C2 93C1 411D 5AB2 D5A8 E8A8 0E74 6127 On Thu, Nov 9, 2023 at 12:19?PM Robert Hudson wrote: > I largely agree with Luke. Given you're on a dedicated iSCSI network, > keep it simple. DCB and other services will only add things that you'll > later need to troubleshoot and eliminate as the root cause of network > issues on your iSCSI network when they invariably happen (it's rare that > I've come across a well and consistently configured iSCSI network, and I've > been playing in that space since the mid 2000s). Chances are your > OS/hypervisor vendor of choice publishes best practices for how to > configure DCB - but as noted, DCB is specifically there to deal with > converged networks (where your iSCSI traffic is sharing an ethernet fabric > with other traffic types), and you don't seem to have that situation. > > Jumbo frames help in busy iSCSI networks by increasing throughput - but > you need to make sure every device from one end of the communications to > the other fully supports it. Again, follow vendor advice here. Getting > this wrong can cause all sorts of "fun". > > Flow control, buffer tuning (large buffers tend to help with iSCSI > traffic), etc, can all help to eke out a few more small percentage points > of performance, but again, the further you drift from the KISS principle, > the more fun you're likely to have troubleshooting later. > > Above all - set and document policy in all things, audit against that > policy both at initial setup and for drift during the lifecycle of the > environment. > > On Thu, 9 Nov 2023 at 12:20, Luke Iggleden wrote: > >> Hi Andres, >> >> Unless you are running other services on the switch it's not useful. >> >> Typically these are the only useful changes: >> >> Jumbo Frames (YMMV), depends on vendor. >> >> Flow Control on (so hosts can issue back off - hopefully without dropping >> frames) >> >> Depending on the switch, buffer tuning. >> >> Don't use control plane things, like MLAG, Stacking, STP, etc etc. Flat >> fabric. >> >> >> Cheers, >> >> Luke Iggleden >> >> >> On 9/11/2023 11:14 am, Andres Miedzowicz wrote: >> >> Hello everyone, >> >> >> >> I wanted to get some opinions on the use of DCB and its associated >> protocols in a storage-only (iSCSI), non-converged network. Any thoughts >> about the pros and cons of enabling DCB in a scenario where 100% of the >> traffic on a switch is bi-directional iSCSI storage (virtual machines and >> backups)? >> >> >> >> Thanks in advance. >> >> >> >> Andres >> >> _______________________________________________ >> AusNOG mailing listAusNOG at lists.ausnog.nethttps://lists.ausnog.net/mailman/listinfo/ausnog >> >> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog >> > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From maxs at intellectit.com.au Thu Nov 9 22:59:51 2023 From: maxs at intellectit.com.au (Max Soukhomlinov) Date: Thu, 9 Nov 2023 11:59:51 +0000 Subject: [AusNOG] Do Optus layer 2 services - services support LACP? References: <139dfd5e-76ad-4ef7-9112-f0ff8670f76d.d97b3e74-c9b5-494f-a9d6-f22e57dc808a.2a4c6e16-352c-46a4-87db-5bffd237eaa1@emailsignatures365.codetwo.com> <139dfd5e-76ad-4ef7-9112-f0ff8670f76d.fae46e48-2398-4d39-8140-fdd09dc88f93.830bb954-c1fb-40bf-9a64-1d81933ccb5c@emailsignatures365.codetwo.com> <139dfd5e-76ad-4ef7-9112-f0ff8670f76d.cd820413-6279-4554-9f68-561cb6d76a78.81d1ecd0-35b2-47a5-bbba-070beac9f4b2@emailsignatures365.codetwo.com> Message-ID: Hi All, would anyone know if Optus' SD Ethernet (and NBN EE as a layer2) supports Layer 2 Control Protocol Transparency, specifically LACP? We've been trying to get an answer from the Optus account manager for some time now without much success. Cheers, Max [cid:iit-animated-logo-v2.3-230px_43b1166c-e2e8-492a-ad31-bcdde7b8426a.gif] [cid:orange-triangle_d4d9c8da-926b-4dfc-a7f6-23120f39e276.png] Max Soukhomlinov | Director 1300 799 165, 201 | +61 411 596 249 | maxs at intellectit.com.au Suite 214, Level 2, 343 Little Collins St, Melbourne 3000 www.intellectit.com.au Celebrating 20 Years of supporting Australian business. Disclaimer: This message may contain information which is confidential or privileged. If you are not the named addressee of this email, you must not disclose, disseminate or copy this email (and any attachments). The integrity and security of this email cannot be guaranteed as it may have been corrupted, intercepted or altered in transmission, or contain viruses. Liability is not accepted for loss or damage caused by any virus, errors or omissions arising from transmission by the Internet. If this email has been sent to you in error, please notify the sender by reply email and destroy the original -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: iit-animated-logo-v2.3-230px_43b1166c-e2e8-492a-ad31-bcdde7b8426a.gif Type: image/gif Size: 4029 bytes Desc: iit-animated-logo-v2.3-230px_43b1166c-e2e8-492a-ad31-bcdde7b8426a.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: orange-triangle_d4d9c8da-926b-4dfc-a7f6-23120f39e276.png Type: image/png Size: 1173 bytes Desc: orange-triangle_d4d9c8da-926b-4dfc-a7f6-23120f39e276.png URL: From markzzzsmith at gmail.com Fri Nov 10 09:22:58 2023 From: markzzzsmith at gmail.com (Mark Smith) Date: Fri, 10 Nov 2023 09:22:58 +1100 Subject: [AusNOG] DCB in non-converged networks In-Reply-To: References: <89502812-dbd2-4de2-8e5f-6c5a0bb296ac@iggleden.com> Message-ID: Hi All, On Thu, 9 Nov 2023, 13:19 Robert Hudson, wrote: > I largely agree with Luke. Given you're on a dedicated iSCSI network, > keep it simple. DCB and other services will only add things that you'll > later need to troubleshoot and eliminate as the root cause of network > issues on your iSCSI network when they invariably happen (it's rare that > I've come across a well and consistently configured iSCSI network, and I've > been playing in that space since the mid 2000s). Chances are your > OS/hypervisor vendor of choice publishes best practices for how to > configure DCB - but as noted, DCB is specifically there to deal with > converged networks (where your iSCSI traffic is sharing an ethernet fabric > with other traffic types), and you don't seem to have that situation. > > Jumbo frames help in busy iSCSI networks by increasing throughput - but > you need to make sure every device from one end of the communications to > the other fully supports it. Again, follow vendor advice here. Getting > this wrong can cause all sorts of "fun". > Related to MTUs, etc., people here might be interested in a presentation on the topic I did at NZNOG in March. MTUs, MRUs, PMTUs https://www.youtube.com/live/_D03z8LrauU?si=Mr5hadabWXePhCjk&t=1971 > Flow control, buffer tuning (large buffers tend to help with iSCSI > traffic), etc, can all help to eke out a few more small percentage points > of performance, but again, the further you drift from the KISS principle, > the more fun you're likely to have troubleshooting later. > Strongly agree with KISS (foreshadowing the advice at the end of the presentation.) > Above all - set and document policy in all things, audit against that > policy both at initial setup and for drift during the lifecycle of the > environment. > > On Thu, 9 Nov 2023 at 12:20, Luke Iggleden wrote: > >> Hi Andres, >> >> Unless you are running other services on the switch it's not useful. >> >> Typically these are the only useful changes: >> >> Jumbo Frames (YMMV), depends on vendor. >> >> Flow Control on (so hosts can issue back off - hopefully without dropping >> frames) >> >> Depending on the switch, buffer tuning. >> >> Don't use control plane things, like MLAG, Stacking, STP, etc etc. Flat >> fabric. >> >> >> Cheers, >> >> Luke Iggleden >> >> >> On 9/11/2023 11:14 am, Andres Miedzowicz wrote: >> >> Hello everyone, >> >> >> >> I wanted to get some opinions on the use of DCB and its associated >> protocols in a storage-only (iSCSI), non-converged network. Any thoughts >> about the pros and cons of enabling DCB in a scenario where 100% of the >> traffic on a switch is bi-directional iSCSI storage (virtual machines and >> backups)? >> >> >> >> Thanks in advance. >> >> >> >> Andres >> >> _______________________________________________ >> AusNOG mailing listAusNOG at lists.ausnog.nethttps://lists.ausnog.net/mailman/listinfo/ausnog >> >> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog >> > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tmiles42 at gmail.com Fri Nov 10 15:13:29 2023 From: tmiles42 at gmail.com (Tony Miles) Date: Fri, 10 Nov 2023 14:13:29 +1000 Subject: [AusNOG] Do Optus layer 2 services - services support LACP? In-Reply-To: References: <139dfd5e-76ad-4ef7-9112-f0ff8670f76d.d97b3e74-c9b5-494f-a9d6-f22e57dc808a.2a4c6e16-352c-46a4-87db-5bffd237eaa1@emailsignatures365.codetwo.com> <139dfd5e-76ad-4ef7-9112-f0ff8670f76d.fae46e48-2398-4d39-8140-fdd09dc88f93.830bb954-c1fb-40bf-9a64-1d81933ccb5c@emailsignatures365.codetwo.com> <139dfd5e-76ad-4ef7-9112-f0ff8670f76d.cd820413-6279-4554-9f68-561cb6d76a78.81d1ecd0-35b2-47a5-bbba-070beac9f4b2@emailsignatures365.codetwo.com> Message-ID: Generically I would suggest that trying to run LACP over a carrier link is not the best idea. LACP is intended as link aggregation for the two devices that are directly connected together and so LACP packets are usually interpreted as "local" only. Most gear will give you the option to tunnel L2 control packets, but only on a P2P type service. Optus might work depending on product and configuration, I suspect nbn will never work. What are you trying to achieve and are there other options you should consider instead ? On Thu, 9 Nov 2023, 22:00 Max Soukhomlinov, wrote: > Hi All, would anyone know if Optus? SD Ethernet (and NBN EE as a layer2) > supports Layer 2 Control Protocol Transparency, specifically LACP? > > > > We?ve been trying to get an answer from the Optus account manager for some > time now without much success. > > > > Cheers, > > Max > > > > > [image: Intellect IT animated logo] *Max Soukhomlinov* | Director > 1300 799 165, 201 | +61 411 596 249 <+61%20411%20596%20249> | > maxs at intellectit.com.au > Suite 214, Level 2, 343 Little Collins St, Melbourne 3000 > www.intellectit.com.au > Celebrating 20 Years of supporting Australian business. > Disclaimer: This message may contain information which is confidential or > privileged. If you are not the named addressee of this email, you must not > disclose, disseminate or copy this email (and any attachments). The > integrity and security of this email cannot be guaranteed as it may have > been corrupted, intercepted or altered in transmission, or contain viruses. > Liability is not accepted for loss or damage caused by any virus, errors or > omissions arising from transmission by the Internet. If this email has been > sent to you in error, please notify the sender by reply email and destroy > the original > > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: iit-animated-logo-v2.3-230px_43b1166c-e2e8-492a-ad31-bcdde7b8426a.gif Type: image/gif Size: 4029 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: orange-triangle_d4d9c8da-926b-4dfc-a7f6-23120f39e276.png Type: image/png Size: 1173 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: orange-triangle_d4d9c8da-926b-4dfc-a7f6-23120f39e276.png Type: image/png Size: 1173 bytes Desc: not available URL: From matthew at kobayashi.au Fri Nov 10 15:29:38 2023 From: matthew at kobayashi.au (Matthew Kobayashi) Date: Fri, 10 Nov 2023 14:29:38 +1000 Subject: [AusNOG] Do Optus layer 2 services - services support LACP? In-Reply-To: References: <139dfd5e-76ad-4ef7-9112-f0ff8670f76d.d97b3e74-c9b5-494f-a9d6-f22e57dc808a.2a4c6e16-352c-46a4-87db-5bffd237eaa1@emailsignatures365.codetwo.com> <139dfd5e-76ad-4ef7-9112-f0ff8670f76d.fae46e48-2398-4d39-8140-fdd09dc88f93.830bb954-c1fb-40bf-9a64-1d81933ccb5c@emailsignatures365.codetwo.com> <139dfd5e-76ad-4ef7-9112-f0ff8670f76d.cd820413-6279-4554-9f68-561cb6d76a78.81d1ecd0-35b2-47a5-bbba-070beac9f4b2@emailsignatures365.codetwo.com> Message-ID: Hi Max, I can't speak specifically to Optus' Ethernet product, but the general rule of thumb is that EPL services (point-to-point, UNI at each end) are transparent to Layer 2 Control Protocols such as LACP, whereas EVPL services (point-to-multipoint, E-NNI aggregation with services delimited by VLAN IDs at one end) are not L2CP transparent. Your best bet for a definitive answer is going to be talking to your account manager. If they don't have the answer, ask them to check with their pre-sales engineers. It's not an uncommon question, so it shouldn't be difficult to get an answer. On Thu, 9 Nov 2023 at 22:00, Max Soukhomlinov wrote: > Hi All, would anyone know if Optus? SD Ethernet (and NBN EE as a layer2) > supports Layer 2 Control Protocol Transparency, specifically LACP? > > > > We?ve been trying to get an answer from the Optus account manager for some > time now without much success. > > > > Cheers, > > Max > > > > > [image: Intellect IT animated logo] *Max Soukhomlinov* | Director > 1300 799 165, 201 | +61 411 596 249 <+61%20411%20596%20249> | > maxs at intellectit.com.au > Suite 214, Level 2, 343 Little Collins St, Melbourne 3000 > www.intellectit.com.au > Celebrating 20 Years of supporting Australian business. > Disclaimer: This message may contain information which is confidential or > privileged. If you are not the named addressee of this email, you must not > disclose, disseminate or copy this email (and any attachments). The > integrity and security of this email cannot be guaranteed as it may have > been corrupted, intercepted or altered in transmission, or contain viruses. > Liability is not accepted for loss or damage caused by any virus, errors or > omissions arising from transmission by the Internet. If this email has been > sent to you in error, please notify the sender by reply email and destroy > the original > > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: iit-animated-logo-v2.3-230px_43b1166c-e2e8-492a-ad31-bcdde7b8426a.gif Type: image/gif Size: 4029 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: orange-triangle_d4d9c8da-926b-4dfc-a7f6-23120f39e276.png Type: image/png Size: 1173 bytes Desc: not available URL: From raphael.timothy at gmail.com Fri Nov 10 15:44:14 2023 From: raphael.timothy at gmail.com (Tim Raphael) Date: Fri, 10 Nov 2023 15:44:14 +1100 Subject: [AusNOG] Do Optus layer 2 services - services support LACP? In-Reply-To: References: Message-ID: <1906D59D-CED7-42FF-AFC4-6F210E3212B6@gmail.com> An HTML attachment was scrubbed... URL: From markzzzsmith at gmail.com Fri Nov 10 16:37:46 2023 From: markzzzsmith at gmail.com (Mark Smith) Date: Fri, 10 Nov 2023 16:37:46 +1100 Subject: [AusNOG] Do Optus layer 2 services - services support LACP? In-Reply-To: References: <139dfd5e-76ad-4ef7-9112-f0ff8670f76d.d97b3e74-c9b5-494f-a9d6-f22e57dc808a.2a4c6e16-352c-46a4-87db-5bffd237eaa1@emailsignatures365.codetwo.com> <139dfd5e-76ad-4ef7-9112-f0ff8670f76d.fae46e48-2398-4d39-8140-fdd09dc88f93.830bb954-c1fb-40bf-9a64-1d81933ccb5c@emailsignatures365.codetwo.com> <139dfd5e-76ad-4ef7-9112-f0ff8670f76d.cd820413-6279-4554-9f68-561cb6d76a78.81d1ecd0-35b2-47a5-bbba-070beac9f4b2@emailsignatures365.codetwo.com> Message-ID: On Fri, 10 Nov 2023 at 15:14, Tony Miles wrote: > Generically I would suggest that trying to run LACP over a carrier link is > not the best idea. LACP is intended as link aggregation for the two devices > that are directly connected together and so LACP packets are usually > interpreted as "local" only. Most gear will give you the option to tunnel > L2 control packets, but only on a P2P type service. Optus might work > depending on product and configuration, I suspect nbn will never work. > Agree. More broadly it can end in tears if customers rely on and are allowed to rely on undocumented service feature capabilities, because once you allow a customer to try it and use an undocumented service capability, and it works, and then they rely on it, and it can be hard or impossible to take it away if you need to. (It's a long story, however in the distant past I've had to work on taking away customers' ability to run STP over their VPLSes, if they happened to be, because we needed to use it internally to prevent forwarding loops while LACP was negotiating on a specific vendor's equipment that had followed the IEEE's specs to the letter (after that experience, you realise most vendors aren't entirely implementing LAG/LACP fully IEEE compliantly). Running STP/RSTP wasn't a supported service capability. It would work on a VPLS that was only using our access circuits, however if the customer's VPLS had at least one 3rd party access circuit, then we knew they weren't running STP. The lesson of the story is don't allow customers to actively "suck it and see" for unsupported service capabilities. The technical product spec provided to customers should be the only things that work, so you definitely know what customers are and are not running over your service.) Regards, Mark. > > What are you trying to achieve and are there other options you should > consider instead ? > > On Thu, 9 Nov 2023, 22:00 Max Soukhomlinov, > wrote: > >> Hi All, would anyone know if Optus? SD Ethernet (and NBN EE as a layer2) >> supports Layer 2 Control Protocol Transparency, specifically LACP? >> >> >> >> We?ve been trying to get an answer from the Optus account manager for >> some time now without much success. >> >> >> >> Cheers, >> >> Max >> >> >> >> >> [image: Intellect IT animated logo] *Max Soukhomlinov* | Director >> 1300 799 165, 201 | +61 411 596 249 <+61%20411%20596%20249> | >> maxs at intellectit.com.au >> Suite 214, Level 2, 343 Little Collins St, Melbourne 3000 >> www.intellectit.com.au >> Celebrating 20 Years of supporting Australian business. >> Disclaimer: This message may contain information which is confidential or >> privileged. If you are not the named addressee of this email, you must not >> disclose, disseminate or copy this email (and any attachments). The >> integrity and security of this email cannot be guaranteed as it may have >> been corrupted, intercepted or altered in transmission, or contain viruses. >> Liability is not accepted for loss or damage caused by any virus, errors or >> omissions arising from transmission by the Internet. If this email has been >> sent to you in error, please notify the sender by reply email and destroy >> the original >> >> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog >> > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From markzzzsmith at gmail.com Fri Nov 10 16:39:15 2023 From: markzzzsmith at gmail.com (Mark Smith) Date: Fri, 10 Nov 2023 16:39:15 +1100 Subject: [AusNOG] Do Optus layer 2 services - services support LACP? In-Reply-To: References: <139dfd5e-76ad-4ef7-9112-f0ff8670f76d.d97b3e74-c9b5-494f-a9d6-f22e57dc808a.2a4c6e16-352c-46a4-87db-5bffd237eaa1@emailsignatures365.codetwo.com> <139dfd5e-76ad-4ef7-9112-f0ff8670f76d.fae46e48-2398-4d39-8140-fdd09dc88f93.830bb954-c1fb-40bf-9a64-1d81933ccb5c@emailsignatures365.codetwo.com> <139dfd5e-76ad-4ef7-9112-f0ff8670f76d.cd820413-6279-4554-9f68-561cb6d76a78.81d1ecd0-35b2-47a5-bbba-070beac9f4b2@emailsignatures365.codetwo.com> Message-ID: Forgot to add, this is the best LAG/LACP troubleshooting page on the Internet ever, or at least it was back in 2014. https://networkingbodges.blogspot.com/2012/12/all-sorts-of-things-about-lacp-and-lags.html On Fri, 10 Nov 2023 at 16:37, Mark Smith wrote: > > > On Fri, 10 Nov 2023 at 15:14, Tony Miles wrote: > >> Generically I would suggest that trying to run LACP over a carrier link >> is not the best idea. LACP is intended as link aggregation for the two >> devices that are directly connected together and so LACP packets are >> usually interpreted as "local" only. Most gear will give you the option to >> tunnel L2 control packets, but only on a P2P type service. Optus might work >> depending on product and configuration, I suspect nbn will never work. >> > > Agree. More broadly it can end in tears if customers rely on and are > allowed to rely on undocumented service feature capabilities, because once > you allow a customer to try it and use an undocumented service capability, > and it works, and then they rely on it, and it can be hard or impossible to > take it away if you need to. > > (It's a long story, however in the distant past I've had to work on taking > away customers' ability to run STP over their VPLSes, if they happened to > be, because we needed to use it internally to prevent forwarding loops > while LACP was negotiating on a specific vendor's equipment that had > followed the IEEE's specs to the letter (after that experience, you realise > most vendors aren't entirely implementing LAG/LACP fully IEEE compliantly). > > Running STP/RSTP wasn't a supported service capability. It would work on a > VPLS that was only using our access circuits, however if the customer's > VPLS had at least one 3rd party access circuit, then we knew they weren't > running STP. > > The lesson of the story is don't allow customers to actively "suck it and > see" for unsupported service capabilities. The technical product spec > provided to customers should be the only things that work, so you > definitely know what customers are and are not running over your service.) > > Regards, > Mark. > > > > > > > > > > >> >> What are you trying to achieve and are there other options you should >> consider instead ? >> >> On Thu, 9 Nov 2023, 22:00 Max Soukhomlinov, >> wrote: >> >>> Hi All, would anyone know if Optus? SD Ethernet (and NBN EE as a layer2) >>> supports Layer 2 Control Protocol Transparency, specifically LACP? >>> >>> >>> >>> We?ve been trying to get an answer from the Optus account manager for >>> some time now without much success. >>> >>> >>> >>> Cheers, >>> >>> Max >>> >>> >>> >>> >>> [image: Intellect IT animated logo] *Max Soukhomlinov* | Director >>> 1300 799 165, 201 | +61 411 596 249 <+61%20411%20596%20249> | >>> maxs at intellectit.com.au >>> Suite 214, Level 2, 343 Little Collins St, Melbourne 3000 >>> www.intellectit.com.au >>> Celebrating 20 Years of supporting Australian business. >>> Disclaimer: This message may contain information which is confidential >>> or privileged. If you are not the named addressee of this email, you must >>> not disclose, disseminate or copy this email (and any attachments). The >>> integrity and security of this email cannot be guaranteed as it may have >>> been corrupted, intercepted or altered in transmission, or contain viruses. >>> Liability is not accepted for loss or damage caused by any virus, errors or >>> omissions arising from transmission by the Internet. If this email has been >>> sent to you in error, please notify the sender by reply email and destroy >>> the original >>> >>> _______________________________________________ >>> AusNOG mailing list >>> AusNOG at lists.ausnog.net >>> https://lists.ausnog.net/mailman/listinfo/ausnog >>> >> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at telcom.net.au Fri Nov 10 17:28:03 2023 From: dave at telcom.net.au (David Treacy) Date: Fri, 10 Nov 2023 06:28:03 +0000 Subject: [AusNOG] Do Optus layer 2 services - services support LACP? In-Reply-To: References: <139dfd5e-76ad-4ef7-9112-f0ff8670f76d.d97b3e74-c9b5-494f-a9d6-f22e57dc808a.2a4c6e16-352c-46a4-87db-5bffd237eaa1@emailsignatures365.codetwo.com> <139dfd5e-76ad-4ef7-9112-f0ff8670f76d.fae46e48-2398-4d39-8140-fdd09dc88f93.830bb954-c1fb-40bf-9a64-1d81933ccb5c@emailsignatures365.codetwo.com> <139dfd5e-76ad-4ef7-9112-f0ff8670f76d.cd820413-6279-4554-9f68-561cb6d76a78.81d1ecd0-35b2-47a5-bbba-070beac9f4b2@emailsignatures365.codetwo.com> Message-ID: Depends on the service you have. We supply some of their business grade (RAD) L2 NTU and they will definitely transport LACP and most other L2 control protocols. BUT 100% depends on the service you ordered. Ahh SD is SDN... probably unlikely -- -- David Treacy TELCOM Networks M: 0408 775 999 E: dave at telcom.net.au W: www.telcom.net.au From: AusNOG On Behalf Of Max Soukhomlinov Sent: Thursday, 9 November 2023 11:00 PM To: AusNOG at lists.ausnog.net Subject: [AusNOG] Do Optus layer 2 services - services support LACP? Hi All, would anyone know if Optus' SD Ethernet (and NBN EE as a layer2) supports Layer 2 Control Protocol Transparency, specifically LACP? We've been trying to get an answer from the Optus account manager for some time now without much success. Cheers, Max [Intellect IT animated logo] [cid:image002.png at 01DA13FA.A140EF30] Max Soukhomlinov | Director 1300 799 165, 201 | +61 411 596 249 | maxs at intellectit.com.au Suite 214, Level 2, 343 Little Collins St, Melbourne 3000 www.intellectit.com.au Celebrating 20 Years of supporting Australian business. Disclaimer: This message may contain information which is confidential or privileged. If you are not the named addressee of this email, you must not disclose, disseminate or copy this email (and any attachments). The integrity and security of this email cannot be guaranteed as it may have been corrupted, intercepted or altered in transmission, or contain viruses. Liability is not accepted for loss or damage caused by any virus, errors or omissions arising from transmission by the Internet. If this email has been sent to you in error, please notify the sender by reply email and destroy the original -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 4029 bytes Desc: image001.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.png Type: image/png Size: 1173 bytes Desc: image002.png URL: From francisfides at mailup.net Mon Nov 13 22:32:56 2023 From: francisfides at mailup.net (francisfides at mailup.net) Date: Mon, 13 Nov 2023 21:32:56 +1000 Subject: [AusNOG] Optus downtime chat + affecting SMS verification toTelstra? In-Reply-To: References: Message-ID: <4f8931e0-fd97-41fc-8489-c4ec5b7dec90@app.fastmail.com> Looks like it was a software upgrade: https://www.abc.net.au/news/2023-11-13/optus-identifies-cause-of-nationwide-outage-software-upgrade/103099902 Nothing in their media centre, just appears as a new box on their outage response page: https://www.optus.com.au/notices/outage-response Cheers ---- Text: "We have been working to understand what caused the outage on Wednesday, and we now know what the cause was and have taken steps to ensure it will not happen again. We apologise sincerely for letting our customers down and the inconvenience it caused. At around 4.05am Wednesday morning, the Optus network received changes to routing information from an international peering network following a software upgrade. These routing information changes propagated through multiple layers in our network and exceeded preset safety levels on key routers. This resulted in those routers disconnecting from the Optus IP Core network to protect themselves. The restoration required a large-scale effort of the team and in some cases required Optus to reconnect or reboot routers physically, requiring the dispatch of people across a number of sites in Australia. This is why restoration was progressive over the afternoon. Given the widespread impact of the outage, our investigations into the issue took longer than we would have liked as we examined several different paths to restoration. The restoration of the network was at all times our priority and we subsequently established the cause working together with our partners. We have made changes to the network to address this issue so that it cannot occur again. We are committed to learning from what has occurred and continuing to work with our international vendors and partners to increase the resilience of our network. We will also support and fully cooperate with the reviews being undertaken by the Government and the Senate. We continue to invest heavily to improve the resiliency of our network and services." -- francisfides at mailup.net On Thu, Nov 9, 2023, at 07:15, DaZZa wrote: > I have all three you're asking about. > > But I'm very small potatoes compared to most of the members of this > list, and my required remote footprint is correspondingly small, so > it's easy to maintain. > > D > > On Thu, 9 Nov 2023 at 06:18, Phillip Grasso wrote: >>> >>> I mean come on, it's nearly 2024 and a [major] telco does not have remote console access? >> >> >> If we send a poll out to this community, how many would be able to genuinely honestly answer: >> >> Do you have a console or appropriate control plane access into all your critical infrastructure? >> Do you have independant out of band that does not share any infrastructure with your current system(s) - with exemption for physical location and power. >> Do you have the ability to remote power control your devices? >> >> We know from the facebook outage in 2021 that they probably didn't have the above, so its not entirely uncommon for folks to have *proper independant* console and remote access. >> >> >> I empathize with the Optus team and their customers who have been negatively impacted by this incident. I sincerely hope that some positive outcomes can emerge from this situation, including: >> >> - Attention to critical infrastructure resilience >> - BGP clue increases >> - Incident management improves >> (I'm sure there's more). >> >> Network is a black box to most people and I think a large chunk of Australia now knows what it feels like to not have it. >> >> >> On Wed, 8 Nov 2023 at 11:06, Ben Buxton wrote: >>> >>> >>> >>> On Wed, 8 Nov 2023 at 10:14, DaZZa wrote: >>>> >>>> Yeah, I'd be willing to bet that it's a change which wasn't thoroughly >>>> tested before being rolled out, and which had an inadequate backout >>>> plan. >>> >>> >>> Also, "Our on-site technician is actively prioritising establishing a console connection.". >>> >>> I mean come on, it's nearly 2024 and a [major] telco does not have remote console access? Whilst I'm >>> looking forward to enthusiastically reading the PM, I'll have to book a physio appointment in advance due to >>> neck strain from all the head shaking it'll likely induce. >>> >>> BB >>> >>> >>>> >>>> >>>> Interestingly, my Optus mobile actually had a valid connection for a >>>> short time - wasn't able to actually DO anything, but was connected to >>>> the OPtus network - but it's now gone to "SOS" mode. >>>> >>>> D >>>> >>>> On Wed, 8 Nov 2023 at 10:01, John Edwards wrote: >>>> > >>>> > The 4am Wednesday morning outage start looks suspiciously like a firmware upgrade window. >>>> > >>>> > I note that Optus devices where I am are showing "SoS" which indicates the tower is unable to reach the location register, which presumably is on a private network and indicative of a pretty major fault rather than just IP. >>>> > >>>> > John >>>> > >>>> > >>>> > On Wed, 8 Nov 2023 at 09:10, DaZZa wrote: >>>> >> >>>> >> The Optus hamster finally died of old age. >>>> >> >>>> >> I would suggest your SMS issues would be caused by whoever is issuing >>>> >> the SMS using Optus - not so much by the Telstra end receiving it. >>>> >> >>>> >> Anecdotally, Optus enterprise/wholesale appears to be still functional >>>> >> - at least my link appears to be working fine - and my BGP >>>> >> advertisements are still being seen overseas - seems to be only NBN >>>> >> and mobile based services which are busted >>>> >> >>>> >> D >>>> >> >>>> >> On Wed, 8 Nov 2023 at 09:27, wrote: >>>> >> > >>>> >> > Morning all, >>>> >> > Hope the chaos isn't too hard on your work/family. >>>> >> > I have had trouble with a couple of SMS verifications coming through to me, my Telstra number. Is this related? >>>> >> > >>>> >> > Any general banter around the downtime would be fine too - looks like it all began at 4.07am AEDT? >>>> >> > >>>> >> > Cheers >>>> >> > >>>> >> > -- >>>> >> > >>>> >> > francisfides at mailup.net >>>> >> > _______________________________________________ >>>> >> > AusNOG mailing list >>>> >> > AusNOG at lists.ausnog.net >>>> >> > https://lists.ausnog.net/mailman/listinfo/ausnog >>>> >> >>>> >> >>>> >> >>>> >> -- >>>> >> veg?e?tar?i?an: >>>> >> Ancient tribal slang for the village idiot who can't hunt, fish or ride >>>> >> _______________________________________________ >>>> >> AusNOG mailing list >>>> >> AusNOG at lists.ausnog.net >>>> >> https://lists.ausnog.net/mailman/listinfo/ausnog >>>> >>>> >>>> >>>> -- >>>> veg?e?tar?i?an: >>>> Ancient tribal slang for the village idiot who can't hunt, fish or ride >>>> _______________________________________________ >>>> AusNOG mailing list >>>> AusNOG at lists.ausnog.net >>>> https://lists.ausnog.net/mailman/listinfo/ausnog >>> >>> _______________________________________________ >>> AusNOG mailing list >>> AusNOG at lists.ausnog.net >>> https://lists.ausnog.net/mailman/listinfo/ausnog > > > > -- > veg?e?tar?i?an: > Ancient tribal slang for the village idiot who can't hunt, fish or ride > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog From mike at ozonline.com.au Tue Nov 14 13:02:35 2023 From: mike at ozonline.com.au (Michael Bethune) Date: Tue, 14 Nov 2023 13:02:35 +1100 Subject: [AusNOG] Optus downtime chat + affecting SMS verification toTelstra? In-Reply-To: <4f8931e0-fd97-41fc-8489-c4ec5b7dec90@app.fastmail.com> References: <4f8931e0-fd97-41fc-8489-c4ec5b7dec90@app.fastmail.com> Message-ID: <20231114130235.01014a8se84okgc4@horde-2.ozonline.com.au> "Optus network received changes to routing information from an international peering network following a software upgrade" I note they are very careful to avoid nominating whose software upgrade. I also note that when they say they received routing updates, don't they limit the number of prefixes accepted by their BGP from any given peer? Sounds like a carefully crafted statement to enable them to point fingers elsewhere, not unexpected. - Michael. Quoting francisfides at mailup.net: > Looks like it was a software upgrade: > https://www.abc.net.au/news/2023-11-13/optus-identifies-cause-of-nationwide-outage-software-upgrade/103099902 > > Nothing in their media centre, just appears as a new box on their > outage response page: https://www.optus.com.au/notices/outage-response > > Cheers > > ---- > Text: > > "We have been working to understand what caused the outage on > Wednesday, and we now know what the cause was and have taken steps > to ensure it will not happen again. We apologise sincerely for > letting our customers down and the inconvenience it caused. > > At around 4.05am Wednesday morning, the Optus network received > changes to routing information from an international peering network > following a software upgrade. These routing information changes > propagated through multiple layers in our network and exceeded > preset safety levels on key routers. This resulted in those routers > disconnecting from the Optus IP Core network to protect themselves. > > The restoration required a large-scale effort of the team and in > some cases required Optus to reconnect or reboot routers physically, > requiring the dispatch of people across a number of sites in > Australia. This is why restoration was progressive over the afternoon. > > Given the widespread impact of the outage, our investigations into > the issue took longer than we would have liked as we examined > several different paths to restoration. The restoration of the > network was at all times our priority and we subsequently > established the cause working together with our partners. We have > made changes to the network to address this issue so that it cannot > occur again. > > We are committed to learning from what has occurred and continuing > to work with our international vendors and partners to increase the > resilience of our network. We will also support and fully cooperate > with the reviews being undertaken by the Government and the Senate. > > We continue to invest heavily to improve the resiliency of our > network and services." > > -- > > francisfides at mailup.net > > On Thu, Nov 9, 2023, at 07:15, DaZZa wrote: >> I have all three you're asking about. >> >> But I'm very small potatoes compared to most of the members of this >> list, and my required remote footprint is correspondingly small, so >> it's easy to maintain. >> >> D >> >> On Thu, 9 Nov 2023 at 06:18, Phillip Grasso >> wrote: >>>> >>>> I mean come on, it's nearly 2024 and a [major] telco does not >>>> have remote console access? >>> >>> >>> If we send a poll out to this community, how many would be able to >>> genuinely honestly answer: >>> >>> Do you have a console or appropriate control plane access into all >>> your critical infrastructure? >>> Do you have independant out of band that does not share any >>> infrastructure with your current system(s) - with exemption for >>> physical location and power. >>> Do you have the ability to remote power control your devices? >>> >>> We know from the facebook outage in 2021 that they probably didn't >>> have the above, so its not entirely uncommon for folks to have >>> *proper independant* console and remote access. >>> >>> >>> I empathize with the Optus team and their customers who have been >>> negatively impacted by this incident. I sincerely hope that some >>> positive outcomes can emerge from this situation, including: >>> >>> - Attention to critical infrastructure resilience >>> - BGP clue increases >>> - Incident management improves >>> (I'm sure there's more). >>> >>> Network is a black box to most people and I think a large chunk of >>> Australia now knows what it feels like to not have it. >>> >>> >>> On Wed, 8 Nov 2023 at 11:06, Ben Buxton wrote: >>>> >>>> >>>> >>>> On Wed, 8 Nov 2023 at 10:14, DaZZa wrote: >>>>> >>>>> Yeah, I'd be willing to bet that it's a change which wasn't thoroughly >>>>> tested before being rolled out, and which had an inadequate backout >>>>> plan. >>>> >>>> >>>> Also, "Our on-site technician is actively prioritising >>>> establishing a console connection.". >>>> >>>> I mean come on, it's nearly 2024 and a [major] telco does not >>>> have remote console access? Whilst I'm >>>> looking forward to enthusiastically reading the PM, I'll have to >>>> book a physio appointment in advance due to >>>> neck strain from all the head shaking it'll likely induce. >>>> >>>> BB >>>> >>>> >>>>> >>>>> >>>>> Interestingly, my Optus mobile actually had a valid connection for a >>>>> short time - wasn't able to actually DO anything, but was connected to >>>>> the OPtus network - but it's now gone to "SOS" mode. >>>>> >>>>> D >>>>> >>>>> On Wed, 8 Nov 2023 at 10:01, John Edwards wrote: >>>>> > >>>>> > The 4am Wednesday morning outage start looks suspiciously like >>>>> a firmware upgrade window. >>>>> > >>>>> > I note that Optus devices where I am are showing "SoS" which >>>>> indicates the tower is unable to reach the location register, >>>>> which presumably is on a private network and indicative of a >>>>> pretty major fault rather than just IP. >>>>> > >>>>> > John >>>>> > >>>>> > >>>>> > On Wed, 8 Nov 2023 at 09:10, DaZZa wrote: >>>>> >> >>>>> >> The Optus hamster finally died of old age. >>>>> >> >>>>> >> I would suggest your SMS issues would be caused by whoever is issuing >>>>> >> the SMS using Optus - not so much by the Telstra end receiving it. >>>>> >> >>>>> >> Anecdotally, Optus enterprise/wholesale appears to be still functional >>>>> >> - at least my link appears to be working fine - and my BGP >>>>> >> advertisements are still being seen overseas - seems to be only NBN >>>>> >> and mobile based services which are busted >>>>> >> >>>>> >> D >>>>> >> >>>>> >> On Wed, 8 Nov 2023 at 09:27, wrote: >>>>> >> > >>>>> >> > Morning all, >>>>> >> > Hope the chaos isn't too hard on your work/family. >>>>> >> > I have had trouble with a couple of SMS verifications >>>>> coming through to me, my Telstra number. Is this related? >>>>> >> > >>>>> >> > Any general banter around the downtime would be fine too - >>>>> looks like it all began at 4.07am AEDT? >>>>> >> > >>>>> >> > Cheers >>>>> >> > >>>>> >> > -- >>>>> >> > >>>>> >> > francisfides at mailup.net >>>>> >> > _______________________________________________ >>>>> >> > AusNOG mailing list >>>>> >> > AusNOG at lists.ausnog.net >>>>> >> > https://lists.ausnog.net/mailman/listinfo/ausnog >>>>> >> >>>>> >> >>>>> >> >>>>> >> -- >>>>> >> veg?e?tar?i?an: >>>>> >> Ancient tribal slang for the village idiot who can't hunt, >>>>> fish or ride >>>>> >> _______________________________________________ >>>>> >> AusNOG mailing list >>>>> >> AusNOG at lists.ausnog.net >>>>> >> https://lists.ausnog.net/mailman/listinfo/ausnog >>>>> >>>>> >>>>> >>>>> -- >>>>> veg?e?tar?i?an: >>>>> Ancient tribal slang for the village idiot who can't hunt, fish or ride >>>>> _______________________________________________ >>>>> AusNOG mailing list >>>>> AusNOG at lists.ausnog.net >>>>> https://lists.ausnog.net/mailman/listinfo/ausnog >>>> >>>> _______________________________________________ >>>> AusNOG mailing list >>>> AusNOG at lists.ausnog.net >>>> https://lists.ausnog.net/mailman/listinfo/ausnog >> >> >> >> -- >> veg?e?tar?i?an: >> Ancient tribal slang for the village idiot who can't hunt, fish or ride >> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > From jaedwards at gmail.com Tue Nov 14 13:27:13 2023 From: jaedwards at gmail.com (John Edwards) Date: Tue, 14 Nov 2023 12:57:13 +1030 Subject: [AusNOG] Optus downtime chat + affecting SMS verification toTelstra? In-Reply-To: <20231114130235.01014a8se84okgc4@horde-2.ozonline.com.au> References: <4f8931e0-fd97-41fc-8489-c4ec5b7dec90@app.fastmail.com> <20231114130235.01014a8se84okgc4@horde-2.ozonline.com.au> Message-ID: The default behaviour of the "maximum prefix" BGP feature is to bring down the BGP session with the peer. The alternate behaviour is to log a warning and accept a prefix. I am not aware of an implementation that just allows "Accept up to X routes and then don't accept any more". That sounds logical but in reality would lead to inconsistent behaviour that is more readily addressed with existing routing policy tools. It appears that a failure of routing policy was a major contributor to an Optus outage, where that policy had an assumption of trusting internal peers and the fault was exacerbated by some mechanism where a policy failure was able to impact other logical networks on the same device (assuming there is/was more than 1 logical network). Or maybe someone just leaked full routes into OSPF ? John -------------- next part -------------- An HTML attachment was scrubbed... URL: From reuben-ausnog at reub.net Tue Nov 14 13:35:43 2023 From: reuben-ausnog at reub.net (Reuben Farrelly) Date: Tue, 14 Nov 2023 13:35:43 +1100 Subject: [AusNOG] Optus downtime chat + affecting SMS verification toTelstra? In-Reply-To: References: <4f8931e0-fd97-41fc-8489-c4ec5b7dec90@app.fastmail.com> <20231114130235.01014a8se84okgc4@horde-2.ozonline.com.au> Message-ID: <8279d710-2359-4ee5-b2a1-8858ccb6d1d3@reub.net> There is sometimes the option of a session reset and session restart at a specified interval after the event has triggered. Vendor dependent of course, but the option exists in IOS XE at least and most likely other vendors too. This allows for a recovery once too many prefixes have been received. That probably would have saved a lot of site visits for Optus once the root cause of the prefixes was fixed at the edge. Then there is the unanswered question of where an Out Of Band management network fitted into this picture which also likely would have provided a get-out-of-jail-free card much earlier in the day. Reuben On 14/11/2023 1:27 pm, John Edwards wrote: > The default behaviour of the "maximum prefix" BGP feature is to bring > down the BGP session with the peer. > > The alternate behaviour is to log a warning and accept a prefix. > > I am not aware of an implementation that just allows "Accept up to X > routes and then don't accept any more". > > That sounds logical but in?reality would lead to inconsistent behaviour > that is more readily addressed with existing routing policy tools. > > It appears that a failure of routing policy was a major contributor to > an Optus outage, where that policy had an assumption of > trusting?internal peers and the fault was exacerbated by some mechanism > where a policy failure was able to impact other logical networks on the > same device (assuming there is/was more than 1 logical network). > > Or maybe someone just leaked full routes into OSPF?? > > John > > > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog From bb.ausnog at bb.cactii.net Tue Nov 14 13:36:46 2023 From: bb.ausnog at bb.cactii.net (Ben Buxton) Date: Tue, 14 Nov 2023 13:36:46 +1100 Subject: [AusNOG] Optus downtime chat + affecting SMS verification toTelstra? In-Reply-To: <20231114130235.01014a8se84okgc4@horde-2.ozonline.com.au> References: <4f8931e0-fd97-41fc-8489-c4ec5b7dec90@app.fastmail.com> <20231114130235.01014a8se84okgc4@horde-2.ozonline.com.au> Message-ID: Blaming routing updates from peers is a scapegoat and never is the cause of an outage - public BGP is the wild west and you're always getting broken information - it's your responsibility to filter those updates and (unless it's a zero-day poison packet bug) you only have yourself to blame if you fall over from them. If I were an optus business customer, reading that outage page would just make me even more determined to move elsewhere. They vaguely categorised the "what" of the outage into a big bucket (software upgrade related), but gave absolutely no useful information or explain the "why" which would regain my confidence. Why did this upgrade trigger an outage? - Was there a behaviour/feature change they neglected to take into account? - Did the upgrade require a config change that broke? - Were they neglectful in following config best practices? (filtering, prefix limits, restarts, etc?) - Did the new software have an unidentified bug? - Why did testing not catch this problem (they do test changes...right?) - How did progressive rollout still lead to this impact? (they do progressive rollouts over N days/weeks...right?) Why did mitigation take so long? - What detection/telemetry measures led them to realise the scope of the outage? (news reports dont count) - Were they dependent on the downed network for oncall paging & comms? - Why did their rollback plan fail? (they had a rollback plan...right?) - Why was remote console/power access not working? (they have both...right?) - Were they dependent on the downed network for said access? - Were their playbooks/credential access dependent on the downed network? "We have made changes to the network to address this issue so that it cannot occur again." ... this smells like "whoops forgot to set max-prefix (with restart!)". Bugs, config stuff-ups, etc happen, and they will continue to happen - it is a lie to state that outages will never happen again. This is the culmination of monumental failures in the trigger, prevention and mitigation measures which cannot be fixed in a couple of days, it sounds like much deeper architectural and organisational issues need addressing. Many of the above failures are things that a young network will experience and learn from, but for Optus these should all be well planned for already. I suspect any government investigation will simply add more bureaucracy and boxes to tick rather than effect meaningful change, but one can always be hopeful... BB On Tue, 14 Nov 2023 at 13:02, Michael Bethune wrote: > "Optus network received changes to routing information from an > international peering network following a software upgrade" > > I note they are very careful to avoid nominating whose software upgrade. > > I also note that when they say they received routing updates, > don't they limit the number of prefixes accepted by their BGP from > any given peer? > > Sounds like a carefully crafted statement to enable them to point fingers > elsewhere, not unexpected. > > - Michael. > > Quoting francisfides at mailup.net: > > > Looks like it was a software upgrade: > > > https://www.abc.net.au/news/2023-11-13/optus-identifies-cause-of-nationwide-outage-software-upgrade/103099902 > > > > Nothing in their media centre, just appears as a new box on their > > outage response page: https://www.optus.com.au/notices/outage-response > > > > Cheers > > > > ---- > > Text: > > > > "We have been working to understand what caused the outage on > > Wednesday, and we now know what the cause was and have taken steps > > to ensure it will not happen again. We apologise sincerely for > > letting our customers down and the inconvenience it caused. > > > > At around 4.05am Wednesday morning, the Optus network received > > changes to routing information from an international peering network > > following a software upgrade. These routing information changes > > propagated through multiple layers in our network and exceeded > > preset safety levels on key routers. This resulted in those routers > > disconnecting from the Optus IP Core network to protect themselves. > > > > The restoration required a large-scale effort of the team and in > > some cases required Optus to reconnect or reboot routers physically, > > requiring the dispatch of people across a number of sites in > > Australia. This is why restoration was progressive over the afternoon. > > > > Given the widespread impact of the outage, our investigations into > > the issue took longer than we would have liked as we examined > > several different paths to restoration. The restoration of the > > network was at all times our priority and we subsequently > > established the cause working together with our partners. We have > > made changes to the network to address this issue so that it cannot > > occur again. > > > > We are committed to learning from what has occurred and continuing > > to work with our international vendors and partners to increase the > > resilience of our network. We will also support and fully cooperate > > with the reviews being undertaken by the Government and the Senate. > > > > We continue to invest heavily to improve the resiliency of our > > network and services." > > > > -- > > > > francisfides at mailup.net > > > > On Thu, Nov 9, 2023, at 07:15, DaZZa wrote: > >> I have all three you're asking about. > >> > >> But I'm very small potatoes compared to most of the members of this > >> list, and my required remote footprint is correspondingly small, so > >> it's easy to maintain. > >> > >> D > >> > >> On Thu, 9 Nov 2023 at 06:18, Phillip Grasso > >> wrote: > >>>> > >>>> I mean come on, it's nearly 2024 and a [major] telco does not > >>>> have remote console access? > >>> > >>> > >>> If we send a poll out to this community, how many would be able to > >>> genuinely honestly answer: > >>> > >>> Do you have a console or appropriate control plane access into all > >>> your critical infrastructure? > >>> Do you have independant out of band that does not share any > >>> infrastructure with your current system(s) - with exemption for > >>> physical location and power. > >>> Do you have the ability to remote power control your devices? > >>> > >>> We know from the facebook outage in 2021 that they probably didn't > >>> have the above, so its not entirely uncommon for folks to have > >>> *proper independant* console and remote access. > >>> > >>> > >>> I empathize with the Optus team and their customers who have been > >>> negatively impacted by this incident. I sincerely hope that some > >>> positive outcomes can emerge from this situation, including: > >>> > >>> - Attention to critical infrastructure resilience > >>> - BGP clue increases > >>> - Incident management improves > >>> (I'm sure there's more). > >>> > >>> Network is a black box to most people and I think a large chunk of > >>> Australia now knows what it feels like to not have it. > >>> > >>> > >>> On Wed, 8 Nov 2023 at 11:06, Ben Buxton > wrote: > >>>> > >>>> > >>>> > >>>> On Wed, 8 Nov 2023 at 10:14, DaZZa wrote: > >>>>> > >>>>> Yeah, I'd be willing to bet that it's a change which wasn't > thoroughly > >>>>> tested before being rolled out, and which had an inadequate backout > >>>>> plan. > >>>> > >>>> > >>>> Also, "Our on-site technician is actively prioritising > >>>> establishing a console connection.". > >>>> > >>>> I mean come on, it's nearly 2024 and a [major] telco does not > >>>> have remote console access? Whilst I'm > >>>> looking forward to enthusiastically reading the PM, I'll have to > >>>> book a physio appointment in advance due to > >>>> neck strain from all the head shaking it'll likely induce. > >>>> > >>>> BB > >>>> > >>>> > >>>>> > >>>>> > >>>>> Interestingly, my Optus mobile actually had a valid connection for a > >>>>> short time - wasn't able to actually DO anything, but was connected > to > >>>>> the OPtus network - but it's now gone to "SOS" mode. > >>>>> > >>>>> D > >>>>> > >>>>> On Wed, 8 Nov 2023 at 10:01, John Edwards > wrote: > >>>>> > > >>>>> > The 4am Wednesday morning outage start looks suspiciously like > >>>>> a firmware upgrade window. > >>>>> > > >>>>> > I note that Optus devices where I am are showing "SoS" which > >>>>> indicates the tower is unable to reach the location register, > >>>>> which presumably is on a private network and indicative of a > >>>>> pretty major fault rather than just IP. > >>>>> > > >>>>> > John > >>>>> > > >>>>> > > >>>>> > On Wed, 8 Nov 2023 at 09:10, DaZZa wrote: > >>>>> >> > >>>>> >> The Optus hamster finally died of old age. > >>>>> >> > >>>>> >> I would suggest your SMS issues would be caused by whoever is > issuing > >>>>> >> the SMS using Optus - not so much by the Telstra end receiving it. > >>>>> >> > >>>>> >> Anecdotally, Optus enterprise/wholesale appears to be still > functional > >>>>> >> - at least my link appears to be working fine - and my BGP > >>>>> >> advertisements are still being seen overseas - seems to be only > NBN > >>>>> >> and mobile based services which are busted > >>>>> >> > >>>>> >> D > >>>>> >> > >>>>> >> On Wed, 8 Nov 2023 at 09:27, wrote: > >>>>> >> > > >>>>> >> > Morning all, > >>>>> >> > Hope the chaos isn't too hard on your work/family. > >>>>> >> > I have had trouble with a couple of SMS verifications > >>>>> coming through to me, my Telstra number. Is this related? > >>>>> >> > > >>>>> >> > Any general banter around the downtime would be fine too - > >>>>> looks like it all began at 4.07am AEDT? > >>>>> >> > > >>>>> >> > Cheers > >>>>> >> > > >>>>> >> > -- > >>>>> >> > > >>>>> >> > francisfides at mailup.net > >>>>> >> > _______________________________________________ > >>>>> >> > AusNOG mailing list > >>>>> >> > AusNOG at lists.ausnog.net > >>>>> >> > https://lists.ausnog.net/mailman/listinfo/ausnog > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> -- > >>>>> >> veg?e?tar?i?an: > >>>>> >> Ancient tribal slang for the village idiot who can't hunt, > >>>>> fish or ride > >>>>> >> _______________________________________________ > >>>>> >> AusNOG mailing list > >>>>> >> AusNOG at lists.ausnog.net > >>>>> >> https://lists.ausnog.net/mailman/listinfo/ausnog > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> veg?e?tar?i?an: > >>>>> Ancient tribal slang for the village idiot who can't hunt, fish or > ride > >>>>> _______________________________________________ > >>>>> AusNOG mailing list > >>>>> AusNOG at lists.ausnog.net > >>>>> https://lists.ausnog.net/mailman/listinfo/ausnog > >>>> > >>>> _______________________________________________ > >>>> AusNOG mailing list > >>>> AusNOG at lists.ausnog.net > >>>> https://lists.ausnog.net/mailman/listinfo/ausnog > >> > >> > >> > >> -- > >> veg?e?tar?i?an: > >> Ancient tribal slang for the village idiot who can't hunt, fish or ride > >> _______________________________________________ > >> AusNOG mailing list > >> AusNOG at lists.ausnog.net > >> https://lists.ausnog.net/mailman/listinfo/ausnog > > _______________________________________________ > > AusNOG mailing list > > AusNOG at lists.ausnog.net > > https://lists.ausnog.net/mailman/listinfo/ausnog > > > > > > > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From luke.t at tncrew.com.au Wed Nov 15 11:01:18 2023 From: luke.t at tncrew.com.au (Luke Thompson) Date: Wed, 15 Nov 2023 10:01:18 +1000 Subject: [AusNOG] Optus downtime chat + affecting SMS verification toTelstra? In-Reply-To: References: <4f8931e0-fd97-41fc-8489-c4ec5b7dec90@app.fastmail.com> <20231114130235.01014a8se84okgc4@horde-2.ozonline.com.au> Message-ID: <18bd04713e8.28b2.934dbec5914d2454830359510b9fa213@tncrew.com.au> They've blamed Singtel Internet Exchange (STiX) for the international peering route updates, at least going by anonymous sources cited by SMH. https://www.smh.com.au/technology/identity-of-third-party-who-brought-down-optus-network-revealed-20231114-p5ejy1.html Luke On 14 November 2023 12:37:30 pm Ben Buxton wrote: > > Blaming routing updates from peers is a scapegoat and never is the cause of > an outage - public BGP is the wild west and you're always getting broken > information - it's your responsibility to filter those updates and (unless > it's a zero-day poison packet bug) you only have yourself to blame if you > fall over from them. > > If I were an optus business customer, reading that outage page would just > make me even more determined to move elsewhere. > > They vaguely categorised the "what" of the outage into a big bucket > (software upgrade related), but gave absolutely no useful information or > explain the "why" which would regain my confidence. > > Why did this upgrade trigger an outage? > - Was there a behaviour/feature change they neglected to take into account? > - Did the upgrade require a config change that broke? > - Were they neglectful in following config best practices? (filtering, > prefix limits, restarts, etc?)- Did the new software have an unidentified bug? > - Why did testing not catch this problem (they do test changes...right?) > - How did progressive rollout still lead to this impact? (they do > progressive rollouts over N days/weeks...right?) > > Why did mitigation take so long? > - What detection/telemetry measures led them to realise the scope of the > outage? (news reports dont count)- Were they dependent on the downed > network for oncall paging & comms? > > - Why did their rollback plan fail? (they had a rollback plan...right?) > - Why was remote console/power access not working? (they have both...right?) > - Were they dependent on the downed network for said access? > - Were their playbooks/credential access dependent on the downed network? > > "We have made changes to the network to address this issue so that it > cannot occur again." ... this smells like "whoops forgot to set max-prefix > (with restart!)". > > Bugs, config stuff-ups, etc happen, and they will continue to happen - it > is a lie to state that outages will never happen again. This is the > culmination of monumental failures in the trigger, prevention and > mitigation measures which cannot be fixed in a couple of days, it sounds > like much deeper architectural and organisational issues need addressing. > Many of the above failures are things that a young network will experience > and learn from, but for Optus these should all be well planned for already. > > I suspect any government investigation will simply add more bureaucracy and > boxes to tick rather than effect meaningful change, but one can always be > hopeful... > > BB > > On Tue, 14 Nov 2023 at 13:02, Michael Bethune wrote: > "Optus network received changes to routing information from an > international peering network following a software upgrade" > > I note they are very careful to avoid nominating whose software upgrade. > > I also note that when they say they received routing updates, > don't they limit the number of prefixes accepted by their BGP from > any given peer? > > Sounds like a carefully crafted statement to enable them to point fingers > elsewhere, not unexpected. > > - Michael. > > Quoting francisfides at mailup.net: > >> Looks like it was a software upgrade: >> https://www.abc.net.au/news/2023-11-13/optus-identifies-cause-of-nationwide-outage-software-upgrade/103099902 >> >> Nothing in their media centre, just appears as a new box on their >> outage response page: https://www.optus.com.au/notices/outage-response >> >> Cheers >> >> ---- >> Text: >> >> "We have been working to understand what caused the outage on >> Wednesday, and we now know what the cause was and have taken steps >> to ensure it will not happen again. We apologise sincerely for >> letting our customers down and the inconvenience it caused. >> >> At around 4.05am Wednesday morning, the Optus network received >> changes to routing information from an international peering network >> following a software upgrade. These routing information changes >> propagated through multiple layers in our network and exceeded >> preset safety levels on key routers. This resulted in those routers >> disconnecting from the Optus IP Core network to protect themselves. >> >> The restoration required a large-scale effort of the team and in >> some cases required Optus to reconnect or reboot routers physically, >> requiring the dispatch of people across a number of sites in >> Australia. This is why restoration was progressive over the afternoon. >> >> Given the widespread impact of the outage, our investigations into >> the issue took longer than we would have liked as we examined >> several different paths to restoration. The restoration of the >> network was at all times our priority and we subsequently >> established the cause working together with our partners. We have >> made changes to the network to address this issue so that it cannot >> occur again. >> >> We are committed to learning from what has occurred and continuing >> to work with our international vendors and partners to increase the >> resilience of our network. We will also support and fully cooperate >> with the reviews being undertaken by the Government and the Senate. >> >> We continue to invest heavily to improve the resiliency of our >> network and services." >> >> -- >> >> francisfides at mailup.net >> >> On Thu, Nov 9, 2023, at 07:15, DaZZa wrote: >>> I have all three you're asking about. >>> >>> But I'm very small potatoes compared to most of the members of this >>> list, and my required remote footprint is correspondingly small, so >>> it's easy to maintain. >>> >>> D >>> >>> On Thu, 9 Nov 2023 at 06:18, Phillip Grasso >>> wrote: >>>>> >>>>> I mean come on, it's nearly 2024 and a [major] telco does not >>>>> have remote console access? >>>> >>>> >>>> If we send a poll out to this community, how many would be able to >>>> genuinely honestly answer: >>>> >>>> Do you have a console or appropriate control plane access into all >>>> your critical infrastructure? >>>> Do you have independant out of band that does not share any >>>> infrastructure with your current system(s) - with exemption for >>>> physical location and power. >>>> Do you have the ability to remote power control your devices? >>>> >>>> We know from the facebook outage in 2021 that they probably didn't >>>> have the above, so its not entirely uncommon for folks to have >>>> *proper independant* console and remote access. >>>> >>>> >>>> I empathize with the Optus team and their customers who have been >>>> negatively impacted by this incident. I sincerely hope that some >>>> positive outcomes can emerge from this situation, including: >>>> >>>> - Attention to critical infrastructure resilience >>>> - BGP clue increases >>>> - Incident management improves >>>> (I'm sure there's more). >>>> >>>> Network is a black box to most people and I think a large chunk of >>>> Australia now knows what it feels like to not have it. >>>> >>>> >>>> On Wed, 8 Nov 2023 at 11:06, Ben Buxton wrote: >>>>> >>>>> >>>>> >>>>> On Wed, 8 Nov 2023 at 10:14, DaZZa wrote: >>>>>> >>>>>> Yeah, I'd be willing to bet that it's a change which wasn't thoroughly >>>>>> tested before being rolled out, and which had an inadequate backout >>>>>> plan. >>>>> >>>>> >>>>> Also, "Our on-site technician is actively prioritising >>>>> establishing a console connection.". >>>>> >>>>> I mean come on, it's nearly 2024 and a [major] telco does not >>>>> have remote console access? Whilst I'm >>>>> looking forward to enthusiastically reading the PM, I'll have to >>>>> book a physio appointment in advance due to >>>>> neck strain from all the head shaking it'll likely induce. >>>>> >>>>> BB >>>>> >>>>> >>>>>> >>>>>> >>>>>> Interestingly, my Optus mobile actually had a valid connection for a >>>>>> short time - wasn't able to actually DO anything, but was connected to >>>>>> the OPtus network - but it's now gone to "SOS" mode. >>>>>> >>>>>> D >>>>>> >>>>>> On Wed, 8 Nov 2023 at 10:01, John Edwards wrote: >>>>>> > >>>>>> > The 4am Wednesday morning outage start looks suspiciously like >>>>>> a firmware upgrade window. >>>>>> > >>>>>> > I note that Optus devices where I am are showing "SoS" which >>>>>> indicates the tower is unable to reach the location register, >>>>>> which presumably is on a private network and indicative of a >>>>>> pretty major fault rather than just IP. >>>>>> > >>>>>> > John >>>>>> > >>>>>> > >>>>>> > On Wed, 8 Nov 2023 at 09:10, DaZZa wrote: >>>>>> >> >>>>>> >> The Optus hamster finally died of old age. >>>>>> >> >>>>>> >> I would suggest your SMS issues would be caused by whoever is issuing >>>>>> >> the SMS using Optus - not so much by the Telstra end receiving it. >>>>>> >> >>>>>> >> Anecdotally, Optus enterprise/wholesale appears to be still functional >>>>>> >> - at least my link appears to be working fine - and my BGP >>>>>> >> advertisements are still being seen overseas - seems to be only NBN >>>>>> >> and mobile based services which are busted >>>>>> >> >>>>>> >> D >>>>>> >> >>>>>> >> On Wed, 8 Nov 2023 at 09:27, wrote: >>>>>> >> > >>>>>> >> > Morning all, >>>>>> >> > Hope the chaos isn't too hard on your work/family. >>>>>> >> > I have had trouble with a couple of SMS verifications >>>>>> coming through to me, my Telstra number. Is this related? >>>>>> >> > >>>>>> >> > Any general banter around the downtime would be fine too - >>>>>> looks like it all began at 4.07am AEDT? >>>>>> >> > >>>>>> >> > Cheers >>>>>> >> > >>>>>> >> > -- >>>>>> >> > >>>>>> >> > francisfides at mailup.net >>>>>> >> > _______________________________________________ >>>>>> >> > AusNOG mailing list >>>>>> >> > AusNOG at lists.ausnog.net >>>>>> >> > https://lists.ausnog.net/mailman/listinfo/ausnog >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> -- >>>>>> >> veg?e?tar?i?an: >>>>>> >> Ancient tribal slang for the village idiot who can't hunt, >>>>>> fish or ride >>>>>> >> _______________________________________________ >>>>>> >> AusNOG mailing list >>>>>> >> AusNOG at lists.ausnog.net >>>>>> >> https://lists.ausnog.net/mailman/listinfo/ausnog >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> veg?e?tar?i?an: >>>>>> Ancient tribal slang for the village idiot who can't hunt, fish or ride >>>>>> _______________________________________________ >>>>>> AusNOG mailing list >>>>>> AusNOG at lists.ausnog.net >>>>>> https://lists.ausnog.net/mailman/listinfo/ausnog >>>>> >>>>> _______________________________________________ >>>>> AusNOG mailing list >>>>> AusNOG at lists.ausnog.net >>>>> https://lists.ausnog.net/mailman/listinfo/ausnog >>> >>> >>> >>> -- >>> veg?e?tar?i?an: >>> Ancient tribal slang for the village idiot who can't hunt, fish or ride >>> _______________________________________________ >>> AusNOG mailing list >>> AusNOG at lists.ausnog.net >>> https://lists.ausnog.net/mailman/listinfo/ausnog >> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog >> > > > > > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog -------------- next part -------------- An HTML attachment was scrubbed... URL: From joe at ausnog.net Wed Nov 15 11:28:32 2023 From: joe at ausnog.net (Joe Wooller) Date: Wed, 15 Nov 2023 08:28:32 +0800 Subject: [AusNOG] AusNOG 2024 Program Committee Message-ID: Hey Ausnoggers, Reflecting on the success of AusNOG 2023, we extend our gratitude to all participants for contributing to an exceptional event. If you missed the action, catch up on the insightful presentations at https://www.ausnog.net/events/ausnog-2023/program. Looking ahead, AusNOG 2024 is set to take place in Sydney and we're thrilled to announce an opportunity for you to be part of the Program Committee. We are seeking 2 dynamic individuals who share our passion for the industry and want to play a key role in shaping AusNOG 2024. *Criteria for Program Committee Members:* *Industry Expertise:* You need to be actively involved in the industry. *AusNOG Participation:* Attend at least 2 previous AusNOG conferences. *Membership:* Sign up as an AusNOG member https://member.ausnog.net/signup/ (reach out if you encounter any issues). *Commitment:* Allocate time to assist with PC duties, including identifying compelling topics and presenters, and collaborating effectively within the team. *Attendance:* Commit to attending AusNOG 2024. *Session Hosting:* Be ready to host sessions at AusNOG 2024. *Travel and Accommodation Note:* All AusNOG PC members will receive a complimentary ticket to AusNOG 2024, ensuring you won't miss out on the action. However, please note that you will need to arrange your own flights and accommodation for the event. If you are enthusiastic about shaping the future of AusNOG and meet the criteria, we invite you to express your interest by reaching out off-list ( program-chair at ausnog.net). Let's make AusNOG 2024 another outstanding gathering! Regards, Joe -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at thesysadmin.dev Thu Nov 16 14:42:00 2023 From: chris at thesysadmin.dev (Christopher Hawker) Date: Thu, 16 Nov 2023 03:42:00 +0000 Subject: [AusNOG] =?windows-1252?q?Your_Input_Needed=3A_Can_ROA_Replace_LO?= =?windows-1252?q?A=3F_=96_Short_Survey_=287_mins=29?= Message-ID: Hello everyone, Aftab Siddiqui is currently exploring the possibility of using Route Object Authorisations (ROAs) as a potential replacement to LOAs. Separate to this (and unknowing of Aftab's research), I had started a discussion on the RPKI Community guild on Discord (https://discord.gg/9jYcqpbdRE) discussing the usage of ROAs instead of LOAs. An LOA, or "Letter of Authority" / "Letter of Authorization," is a formal document granting permission for third parties to take specific actions regarding network resources or services. In the service provider industry, its primary use is for advertising address resources (IPv4/v6 and ASN). When an organization intends to announce its IP prefixes through its own or a transit provider's ASN to the global internet, it typically needs to provide an LOA to their transit provider, confirming their custodianship or ownership of the resources. RPKI ROA, stands for "Resource Public Key Infrastructure Route Origin Authorization," is part of a security framework designed to validate the authenticity of internet routing information. It involves a digitally signed object that specifies which Autonomous Systems (ASes) are permitted to announce specific IP address prefixes. Could you please take a moment to fill out our brief survey? Your feedback will play a crucial role in our understanding of this topic. Survey Link: https://www.surveymonkey.com/r/JCHLWBB Thanks, Christopher Hawker -------------- next part -------------- An HTML attachment was scrubbed... URL: From dazzagibbs at gmail.com Fri Nov 17 11:14:38 2023 From: dazzagibbs at gmail.com (DaZZa) Date: Fri, 17 Nov 2023 11:14:38 +1100 Subject: [AusNOG] Optus downtime chat + affecting SMS verification toTelstra? In-Reply-To: <18bd04713e8.28b2.934dbec5914d2454830359510b9fa213@tncrew.com.au> References: <4f8931e0-fd97-41fc-8489-c4ec5b7dec90@app.fastmail.com> <20231114130235.01014a8se84okgc4@horde-2.ozonline.com.au> <18bd04713e8.28b2.934dbec5914d2454830359510b9fa213@tncrew.com.au> Message-ID: And now Singtel have returned serve and are denying it was them. https://www.zdnet.com/article/singtel-refutes-reports-that-its-system-upgrade-caused-optus-outage/ It's like watching kids trying to blame each other for who broke the window with the cricket ball. D On Wed, 15 Nov 2023 at 11:01, Luke Thompson wrote: > > They've blamed Singtel Internet Exchange (STiX) for the international peering route updates, at least going by anonymous sources cited by SMH. > > https://www.smh.com.au/technology/identity-of-third-party-who-brought-down-optus-network-revealed-20231114-p5ejy1.html > > Luke > > On 14 November 2023 12:37:30 pm Ben Buxton wrote: >> >> >> Blaming routing updates from peers is a scapegoat and never is the cause of an outage - public BGP is the wild west and you're always getting broken information - it's your responsibility to filter those updates and (unless it's a zero-day poison packet bug) you only have yourself to blame if you fall over from them. >> >> If I were an optus business customer, reading that outage page would just make me even more determined to move elsewhere. >> >> They vaguely categorised the "what" of the outage into a big bucket (software upgrade related), but gave absolutely no useful information or explain the "why" which would regain my confidence. >> >> Why did this upgrade trigger an outage? >> - Was there a behaviour/feature change they neglected to take into account? >> - Did the upgrade require a config change that broke? >> - Were they neglectful in following config best practices? (filtering, prefix limits, restarts, etc?) >> - Did the new software have an unidentified bug? >> - Why did testing not catch this problem (they do test changes...right?) >> - How did progressive rollout still lead to this impact? (they do progressive rollouts over N days/weeks...right?) >> >> Why did mitigation take so long? >> - What detection/telemetry measures led them to realise the scope of the outage? (news reports dont count) >> - Were they dependent on the downed network for oncall paging & comms? >> - Why did their rollback plan fail? (they had a rollback plan...right?) >> - Why was remote console/power access not working? (they have both...right?) >> - Were they dependent on the downed network for said access? >> - Were their playbooks/credential access dependent on the downed network? >> >> "We have made changes to the network to address this issue so that it cannot occur again." ... this smells like "whoops forgot to set max-prefix (with restart!)". >> >> Bugs, config stuff-ups, etc happen, and they will continue to happen - it is a lie to state that outages will never happen again. This is the culmination of monumental failures in the trigger, prevention and mitigation measures which cannot be fixed in a couple of days, it sounds like much deeper architectural and organisational issues need addressing. >> >> Many of the above failures are things that a young network will experience and learn from, but for Optus these should all be well planned for already. >> >> I suspect any government investigation will simply add more bureaucracy and boxes to tick rather than effect meaningful change, but one can always be hopeful... >> >> BB >> >> On Tue, 14 Nov 2023 at 13:02, Michael Bethune wrote: >>> >>> "Optus network received changes to routing information from an >>> international peering network following a software upgrade" >>> >>> I note they are very careful to avoid nominating whose software upgrade. >>> >>> I also note that when they say they received routing updates, >>> don't they limit the number of prefixes accepted by their BGP from >>> any given peer? >>> >>> Sounds like a carefully crafted statement to enable them to point fingers >>> elsewhere, not unexpected. >>> >>> - Michael. >>> >>> Quoting francisfides at mailup.net: >>> >>> > Looks like it was a software upgrade: >>> > https://www.abc.net.au/news/2023-11-13/optus-identifies-cause-of-nationwide-outage-software-upgrade/103099902 >>> > >>> > Nothing in their media centre, just appears as a new box on their >>> > outage response page: https://www.optus.com.au/notices/outage-response >>> > >>> > Cheers >>> > >>> > ---- >>> > Text: >>> > >>> > "We have been working to understand what caused the outage on >>> > Wednesday, and we now know what the cause was and have taken steps >>> > to ensure it will not happen again. We apologise sincerely for >>> > letting our customers down and the inconvenience it caused. >>> > >>> > At around 4.05am Wednesday morning, the Optus network received >>> > changes to routing information from an international peering network >>> > following a software upgrade. These routing information changes >>> > propagated through multiple layers in our network and exceeded >>> > preset safety levels on key routers. This resulted in those routers >>> > disconnecting from the Optus IP Core network to protect themselves. >>> > >>> > The restoration required a large-scale effort of the team and in >>> > some cases required Optus to reconnect or reboot routers physically, >>> > requiring the dispatch of people across a number of sites in >>> > Australia. This is why restoration was progressive over the afternoon. >>> > >>> > Given the widespread impact of the outage, our investigations into >>> > the issue took longer than we would have liked as we examined >>> > several different paths to restoration. The restoration of the >>> > network was at all times our priority and we subsequently >>> > established the cause working together with our partners. We have >>> > made changes to the network to address this issue so that it cannot >>> > occur again. >>> > >>> > We are committed to learning from what has occurred and continuing >>> > to work with our international vendors and partners to increase the >>> > resilience of our network. We will also support and fully cooperate >>> > with the reviews being undertaken by the Government and the Senate. >>> > >>> > We continue to invest heavily to improve the resiliency of our >>> > network and services." >>> > >>> > -- >>> > >>> > francisfides at mailup.net >>> > >>> > On Thu, Nov 9, 2023, at 07:15, DaZZa wrote: >>> >> I have all three you're asking about. >>> >> >>> >> But I'm very small potatoes compared to most of the members of this >>> >> list, and my required remote footprint is correspondingly small, so >>> >> it's easy to maintain. >>> >> >>> >> D >>> >> >>> >> On Thu, 9 Nov 2023 at 06:18, Phillip Grasso >>> >> wrote: >>> >>>> >>> >>>> I mean come on, it's nearly 2024 and a [major] telco does not >>> >>>> have remote console access? >>> >>> >>> >>> >>> >>> If we send a poll out to this community, how many would be able to >>> >>> genuinely honestly answer: >>> >>> >>> >>> Do you have a console or appropriate control plane access into all >>> >>> your critical infrastructure? >>> >>> Do you have independant out of band that does not share any >>> >>> infrastructure with your current system(s) - with exemption for >>> >>> physical location and power. >>> >>> Do you have the ability to remote power control your devices? >>> >>> >>> >>> We know from the facebook outage in 2021 that they probably didn't >>> >>> have the above, so its not entirely uncommon for folks to have >>> >>> *proper independant* console and remote access. >>> >>> >>> >>> >>> >>> I empathize with the Optus team and their customers who have been >>> >>> negatively impacted by this incident. I sincerely hope that some >>> >>> positive outcomes can emerge from this situation, including: >>> >>> >>> >>> - Attention to critical infrastructure resilience >>> >>> - BGP clue increases >>> >>> - Incident management improves >>> >>> (I'm sure there's more). >>> >>> >>> >>> Network is a black box to most people and I think a large chunk of >>> >>> Australia now knows what it feels like to not have it. >>> >>> >>> >>> >>> >>> On Wed, 8 Nov 2023 at 11:06, Ben Buxton wrote: >>> >>>> >>> >>>> >>> >>>> >>> >>>> On Wed, 8 Nov 2023 at 10:14, DaZZa wrote: >>> >>>>> >>> >>>>> Yeah, I'd be willing to bet that it's a change which wasn't thoroughly >>> >>>>> tested before being rolled out, and which had an inadequate backout >>> >>>>> plan. >>> >>>> >>> >>>> >>> >>>> Also, "Our on-site technician is actively prioritising >>> >>>> establishing a console connection.". >>> >>>> >>> >>>> I mean come on, it's nearly 2024 and a [major] telco does not >>> >>>> have remote console access? Whilst I'm >>> >>>> looking forward to enthusiastically reading the PM, I'll have to >>> >>>> book a physio appointment in advance due to >>> >>>> neck strain from all the head shaking it'll likely induce. >>> >>>> >>> >>>> BB >>> >>>> >>> >>>> >>> >>>>> >>> >>>>> >>> >>>>> Interestingly, my Optus mobile actually had a valid connection for a >>> >>>>> short time - wasn't able to actually DO anything, but was connected to >>> >>>>> the OPtus network - but it's now gone to "SOS" mode. >>> >>>>> >>> >>>>> D >>> >>>>> >>> >>>>> On Wed, 8 Nov 2023 at 10:01, John Edwards wrote: >>> >>>>> > >>> >>>>> > The 4am Wednesday morning outage start looks suspiciously like >>> >>>>> a firmware upgrade window. >>> >>>>> > >>> >>>>> > I note that Optus devices where I am are showing "SoS" which >>> >>>>> indicates the tower is unable to reach the location register, >>> >>>>> which presumably is on a private network and indicative of a >>> >>>>> pretty major fault rather than just IP. >>> >>>>> > >>> >>>>> > John >>> >>>>> > >>> >>>>> > >>> >>>>> > On Wed, 8 Nov 2023 at 09:10, DaZZa wrote: >>> >>>>> >> >>> >>>>> >> The Optus hamster finally died of old age. >>> >>>>> >> >>> >>>>> >> I would suggest your SMS issues would be caused by whoever is issuing >>> >>>>> >> the SMS using Optus - not so much by the Telstra end receiving it. >>> >>>>> >> >>> >>>>> >> Anecdotally, Optus enterprise/wholesale appears to be still functional >>> >>>>> >> - at least my link appears to be working fine - and my BGP >>> >>>>> >> advertisements are still being seen overseas - seems to be only NBN >>> >>>>> >> and mobile based services which are busted >>> >>>>> >> >>> >>>>> >> D >>> >>>>> >> >>> >>>>> >> On Wed, 8 Nov 2023 at 09:27, wrote: >>> >>>>> >> > >>> >>>>> >> > Morning all, >>> >>>>> >> > Hope the chaos isn't too hard on your work/family. >>> >>>>> >> > I have had trouble with a couple of SMS verifications >>> >>>>> coming through to me, my Telstra number. Is this related? >>> >>>>> >> > >>> >>>>> >> > Any general banter around the downtime would be fine too - >>> >>>>> looks like it all began at 4.07am AEDT? >>> >>>>> >> > >>> >>>>> >> > Cheers >>> >>>>> >> > >>> >>>>> >> > -- >>> >>>>> >> > >>> >>>>> >> > francisfides at mailup.net >>> >>>>> >> > _______________________________________________ >>> >>>>> >> > AusNOG mailing list >>> >>>>> >> > AusNOG at lists.ausnog.net >>> >>>>> >> > https://lists.ausnog.net/mailman/listinfo/ausnog >>> >>>>> >> >>> >>>>> >> >>> >>>>> >> >>> >>>>> >> -- >>> >>>>> >> veg?e?tar?i?an: >>> >>>>> >> Ancient tribal slang for the village idiot who can't hunt, >>> >>>>> fish or ride >>> >>>>> >> _______________________________________________ >>> >>>>> >> AusNOG mailing list >>> >>>>> >> AusNOG at lists.ausnog.net >>> >>>>> >> https://lists.ausnog.net/mailman/listinfo/ausnog >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> -- >>> >>>>> veg?e?tar?i?an: >>> >>>>> Ancient tribal slang for the village idiot who can't hunt, fish or ride >>> >>>>> _______________________________________________ >>> >>>>> AusNOG mailing list >>> >>>>> AusNOG at lists.ausnog.net >>> >>>>> https://lists.ausnog.net/mailman/listinfo/ausnog >>> >>>> >>> >>>> _______________________________________________ >>> >>>> AusNOG mailing list >>> >>>> AusNOG at lists.ausnog.net >>> >>>> https://lists.ausnog.net/mailman/listinfo/ausnog >>> >> >>> >> >>> >> >>> >> -- >>> >> veg?e?tar?i?an: >>> >> Ancient tribal slang for the village idiot who can't hunt, fish or ride >>> >> _______________________________________________ >>> >> AusNOG mailing list >>> >> AusNOG at lists.ausnog.net >>> >> https://lists.ausnog.net/mailman/listinfo/ausnog >>> > _______________________________________________ >>> > AusNOG mailing list >>> > AusNOG at lists.ausnog.net >>> > https://lists.ausnog.net/mailman/listinfo/ausnog >>> > >>> >>> >>> >>> >>> _______________________________________________ >>> AusNOG mailing list >>> AusNOG at lists.ausnog.net >>> https://lists.ausnog.net/mailman/listinfo/ausnog >> >> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog >> > > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog -- veg?e?tar?i?an: Ancient tribal slang for the village idiot who can't hunt, fish or ride From andrew at oakeley.com.au Fri Nov 17 11:31:47 2023 From: andrew at oakeley.com.au (Andrew Oakeley) Date: Fri, 17 Nov 2023 00:31:47 +0000 Subject: [AusNOG] Optus downtime chat + affecting SMS verification toTelstra? In-Reply-To: References: <4f8931e0-fd97-41fc-8489-c4ec5b7dec90@app.fastmail.com> <20231114130235.01014a8se84okgc4@horde-2.ozonline.com.au> <18bd04713e8.28b2.934dbec5914d2454830359510b9fa213@tncrew.com.au> Message-ID: And in the senate enquiry this morning they both blamed Cisco "The trigger was the Singtel outage, but the root cause was Cisco." https://www.abc.net.au/news/2023-11-17/asx-markets-business-live-news-optus-outage-senate-inquiry/103115518 -----Original Message----- From: AusNOG On Behalf Of DaZZa Sent: Friday, November 17, 2023 8:15 AM To: Luke Thompson Cc: michael.bethune at australiaonline.au; ausnog at lists.ausnog.net Subject: Re: [AusNOG] Optus downtime chat + affecting SMS verification toTelstra? And now Singtel have returned serve and are denying it was them. https://www.zdnet.com/article/singtel-refutes-reports-that-its-system-upgrade-caused-optus-outage/ It's like watching kids trying to blame each other for who broke the window with the cricket ball. D On Wed, 15 Nov 2023 at 11:01, Luke Thompson wrote: > > They've blamed Singtel Internet Exchange (STiX) for the international peering route updates, at least going by anonymous sources cited by SMH. > > https://www.smh.com.au/technology/identity-of-third-party-who-brought- > down-optus-network-revealed-20231114-p5ejy1.html > > Luke > > On 14 November 2023 12:37:30 pm Ben Buxton wrote: >> >> >> Blaming routing updates from peers is a scapegoat and never is the cause of an outage - public BGP is the wild west and you're always getting broken information - it's your responsibility to filter those updates and (unless it's a zero-day poison packet bug) you only have yourself to blame if you fall over from them. >> >> If I were an optus business customer, reading that outage page would just make me even more determined to move elsewhere. >> >> They vaguely categorised the "what" of the outage into a big bucket (software upgrade related), but gave absolutely no useful information or explain the "why" which would regain my confidence. >> >> Why did this upgrade trigger an outage? >> - Was there a behaviour/feature change they neglected to take into account? >> - Did the upgrade require a config change that broke? >> - Were they neglectful in following config best practices? (filtering, prefix limits, restarts, etc?) >> - Did the new software have an unidentified bug? >> - Why did testing not catch this problem (they do test changes...right?) >> - How did progressive rollout still lead to this impact? (they do >> progressive rollouts over N days/weeks...right?) >> >> Why did mitigation take so long? >> - What detection/telemetry measures led them to realise the scope of the outage? (news reports dont count) >> - Were they dependent on the downed network for oncall paging & comms? >> - Why did their rollback plan fail? (they had a rollback plan...right?) >> - Why was remote console/power access not working? (they have both...right?) >> - Were they dependent on the downed network for said access? >> - Were their playbooks/credential access dependent on the downed network? >> >> "We have made changes to the network to address this issue so that it cannot occur again." ... this smells like "whoops forgot to set max-prefix (with restart!)". >> >> Bugs, config stuff-ups, etc happen, and they will continue to happen - it is a lie to state that outages will never happen again. This is the culmination of monumental failures in the trigger, prevention and mitigation measures which cannot be fixed in a couple of days, it sounds like much deeper architectural and organisational issues need addressing. >> >> Many of the above failures are things that a young network will experience and learn from, but for Optus these should all be well planned for already. >> >> I suspect any government investigation will simply add more bureaucracy and boxes to tick rather than effect meaningful change, but one can always be hopeful... >> >> BB >> >> On Tue, 14 Nov 2023 at 13:02, Michael Bethune wrote: >>> >>> "Optus network received changes to routing information from an >>> international peering network following a software upgrade" >>> >>> I note they are very careful to avoid nominating whose software upgrade. >>> >>> I also note that when they say they received routing updates, don't >>> they limit the number of prefixes accepted by their BGP from any >>> given peer? >>> >>> Sounds like a carefully crafted statement to enable them to point >>> fingers elsewhere, not unexpected. >>> >>> - Michael. >>> >>> Quoting francisfides at mailup.net: >>> >>> > Looks like it was a software upgrade: >>> > https://www.abc.net.au/news/2023-11-13/optus-identifies-cause-of-n >>> > ationwide-outage-software-upgrade/103099902 >>> > >>> > Nothing in their media centre, just appears as a new box on their >>> > outage response page: >>> > https://www.optus.com.au/notices/outage-response >>> > >>> > Cheers >>> > >>> > ---- >>> > Text: >>> > >>> > "We have been working to understand what caused the outage on >>> > Wednesday, and we now know what the cause was and have taken steps >>> > to ensure it will not happen again. We apologise sincerely for >>> > letting our customers down and the inconvenience it caused. >>> > >>> > At around 4.05am Wednesday morning, the Optus network received >>> > changes to routing information from an international peering >>> > network following a software upgrade. These routing information >>> > changes propagated through multiple layers in our network and >>> > exceeded preset safety levels on key routers. This resulted in >>> > those routers disconnecting from the Optus IP Core network to protect themselves. >>> > >>> > The restoration required a large-scale effort of the team and in >>> > some cases required Optus to reconnect or reboot routers >>> > physically, requiring the dispatch of people across a number of >>> > sites in Australia. This is why restoration was progressive over the afternoon. >>> > >>> > Given the widespread impact of the outage, our investigations into >>> > the issue took longer than we would have liked as we examined >>> > several different paths to restoration. The restoration of the >>> > network was at all times our priority and we subsequently >>> > established the cause working together with our partners. We have >>> > made changes to the network to address this issue so that it >>> > cannot occur again. >>> > >>> > We are committed to learning from what has occurred and continuing >>> > to work with our international vendors and partners to increase >>> > the resilience of our network. We will also support and fully >>> > cooperate with the reviews being undertaken by the Government and the Senate. >>> > >>> > We continue to invest heavily to improve the resiliency of our >>> > network and services." >>> > >>> > -- >>> > >>> > francisfides at mailup.net >>> > >>> > On Thu, Nov 9, 2023, at 07:15, DaZZa wrote: >>> >> I have all three you're asking about. >>> >> >>> >> But I'm very small potatoes compared to most of the members of >>> >> this list, and my required remote footprint is correspondingly >>> >> small, so it's easy to maintain. >>> >> >>> >> D >>> >> >>> >> On Thu, 9 Nov 2023 at 06:18, Phillip Grasso >>> >> wrote: >>> >>>> >>> >>>> I mean come on, it's nearly 2024 and a [major] telco does not >>> >>>> have remote console access? >>> >>> >>> >>> >>> >>> If we send a poll out to this community, how many would be able >>> >>> to genuinely honestly answer: >>> >>> >>> >>> Do you have a console or appropriate control plane access into >>> >>> all your critical infrastructure? >>> >>> Do you have independant out of band that does not share any >>> >>> infrastructure with your current system(s) - with exemption for >>> >>> physical location and power. >>> >>> Do you have the ability to remote power control your devices? >>> >>> >>> >>> We know from the facebook outage in 2021 that they probably >>> >>> didn't have the above, so its not entirely uncommon for folks >>> >>> to have *proper independant* console and remote access. >>> >>> >>> >>> >>> >>> I empathize with the Optus team and their customers who have >>> >>> been negatively impacted by this incident. I sincerely hope that >>> >>> some positive outcomes can emerge from this situation, including: >>> >>> >>> >>> - Attention to critical infrastructure resilience >>> >>> - BGP clue increases >>> >>> - Incident management improves >>> >>> (I'm sure there's more). >>> >>> >>> >>> Network is a black box to most people and I think a large chunk >>> >>> of Australia now knows what it feels like to not have it. >>> >>> >>> >>> >>> >>> On Wed, 8 Nov 2023 at 11:06, Ben Buxton wrote: >>> >>>> >>> >>>> >>> >>>> >>> >>>> On Wed, 8 Nov 2023 at 10:14, DaZZa wrote: >>> >>>>> >>> >>>>> Yeah, I'd be willing to bet that it's a change which wasn't >>> >>>>> thoroughly tested before being rolled out, and which had an >>> >>>>> inadequate backout plan. >>> >>>> >>> >>>> >>> >>>> Also, "Our on-site technician is actively prioritising >>> >>>> establishing a console connection.". >>> >>>> >>> >>>> I mean come on, it's nearly 2024 and a [major] telco does not >>> >>>> have remote console access? Whilst I'm looking forward to >>> >>>> enthusiastically reading the PM, I'll have to book a physio >>> >>>> appointment in advance due to neck strain from all the head >>> >>>> shaking it'll likely induce. >>> >>>> >>> >>>> BB >>> >>>> >>> >>>> >>> >>>>> >>> >>>>> >>> >>>>> Interestingly, my Optus mobile actually had a valid connection >>> >>>>> for a short time - wasn't able to actually DO anything, but >>> >>>>> was connected to the OPtus network - but it's now gone to "SOS" mode. >>> >>>>> >>> >>>>> D >>> >>>>> >>> >>>>> On Wed, 8 Nov 2023 at 10:01, John Edwards wrote: >>> >>>>> > >>> >>>>> > The 4am Wednesday morning outage start looks suspiciously >>> >>>>> > like >>> >>>>> a firmware upgrade window. >>> >>>>> > >>> >>>>> > I note that Optus devices where I am are showing "SoS" which >>> >>>>> indicates the tower is unable to reach the location register, >>> >>>>> which presumably is on a private network and indicative of a >>> >>>>> pretty major fault rather than just IP. >>> >>>>> > >>> >>>>> > John >>> >>>>> > >>> >>>>> > >>> >>>>> > On Wed, 8 Nov 2023 at 09:10, DaZZa wrote: >>> >>>>> >> >>> >>>>> >> The Optus hamster finally died of old age. >>> >>>>> >> >>> >>>>> >> I would suggest your SMS issues would be caused by whoever >>> >>>>> >> is issuing the SMS using Optus - not so much by the Telstra end receiving it. >>> >>>>> >> >>> >>>>> >> Anecdotally, Optus enterprise/wholesale appears to be still >>> >>>>> >> functional >>> >>>>> >> - at least my link appears to be working fine - and my BGP >>> >>>>> >> advertisements are still being seen overseas - seems to be >>> >>>>> >> only NBN and mobile based services which are busted >>> >>>>> >> >>> >>>>> >> D >>> >>>>> >> >>> >>>>> >> On Wed, 8 Nov 2023 at 09:27, wrote: >>> >>>>> >> > >>> >>>>> >> > Morning all, >>> >>>>> >> > Hope the chaos isn't too hard on your work/family. >>> >>>>> >> > I have had trouble with a couple of SMS verifications >>> >>>>> coming through to me, my Telstra number. Is this related? >>> >>>>> >> > >>> >>>>> >> > Any general banter around the downtime would be fine too >>> >>>>> >> > - >>> >>>>> looks like it all began at 4.07am AEDT? >>> >>>>> >> > >>> >>>>> >> > Cheers >>> >>>>> >> > >>> >>>>> >> > -- >>> >>>>> >> > >>> >>>>> >> > francisfides at mailup.net >>> >>>>> >> > _______________________________________________ >>> >>>>> >> > AusNOG mailing list >>> >>>>> >> > AusNOG at lists.ausnog.net >>> >>>>> >> > https://lists.ausnog.net/mailman/listinfo/ausnog >>> >>>>> >> >>> >>>>> >> >>> >>>>> >> >>> >>>>> >> -- >>> >>>>> >> veg?e?tar?i?an: >>> >>>>> >> Ancient tribal slang for the village idiot who can't hunt, >>> >>>>> fish or ride >>> >>>>> >> _______________________________________________ >>> >>>>> >> AusNOG mailing list >>> >>>>> >> AusNOG at lists.ausnog.net >>> >>>>> >> https://lists.ausnog.net/mailman/listinfo/ausnog >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> -- >>> >>>>> veg?e?tar?i?an: >>> >>>>> Ancient tribal slang for the village idiot who can't hunt, >>> >>>>> fish or ride _______________________________________________ >>> >>>>> AusNOG mailing list >>> >>>>> AusNOG at lists.ausnog.net >>> >>>>> https://lists.ausnog.net/mailman/listinfo/ausnog >>> >>>> >>> >>>> _______________________________________________ >>> >>>> AusNOG mailing list >>> >>>> AusNOG at lists.ausnog.net >>> >>>> https://lists.ausnog.net/mailman/listinfo/ausnog >>> >> >>> >> >>> >> >>> >> -- >>> >> veg?e?tar?i?an: >>> >> Ancient tribal slang for the village idiot who can't hunt, fish >>> >> or ride _______________________________________________ >>> >> AusNOG mailing list >>> >> AusNOG at lists.ausnog.net >>> >> https://lists.ausnog.net/mailman/listinfo/ausnog >>> > _______________________________________________ >>> > AusNOG mailing list >>> > AusNOG at lists.ausnog.net >>> > https://lists.ausnog.net/mailman/listinfo/ausnog >>> > >>> >>> >>> >>> >>> _______________________________________________ >>> AusNOG mailing list >>> AusNOG at lists.ausnog.net >>> https://lists.ausnog.net/mailman/listinfo/ausnog >> >> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog >> > > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog -- veg?e?tar?i?an: Ancient tribal slang for the village idiot who can't hunt, fish or ride _______________________________________________ AusNOG mailing list AusNOG at lists.ausnog.net https://lists.ausnog.net/mailman/listinfo/ausnog From dazzagibbs at gmail.com Fri Nov 17 12:37:39 2023 From: dazzagibbs at gmail.com (DaZZa) Date: Fri, 17 Nov 2023 12:37:39 +1100 Subject: [AusNOG] Optus downtime chat + affecting SMS verification toTelstra? In-Reply-To: References: <4f8931e0-fd97-41fc-8489-c4ec5b7dec90@app.fastmail.com> <20231114130235.01014a8se84okgc4@horde-2.ozonline.com.au> <18bd04713e8.28b2.934dbec5914d2454830359510b9fa213@tncrew.com.au> Message-ID: What a load of crap. The root cause was they're morons, and configured the routers incorrectly. Cisco had nothing to do with it. I'll bet the routers behaved exactly as they were intended to behave. Post the config snippets you claim caused it, Optus, and let people who know what they're talking about prove you right or wrong. D On Fri, 17 Nov 2023 at 11:31, Andrew Oakeley wrote: > > And in the senate enquiry this morning they both blamed Cisco > > "The trigger was the Singtel outage, but the root cause was Cisco." > > https://www.abc.net.au/news/2023-11-17/asx-markets-business-live-news-optus-outage-senate-inquiry/103115518 > > -----Original Message----- > From: AusNOG On Behalf Of DaZZa > Sent: Friday, November 17, 2023 8:15 AM > To: Luke Thompson > Cc: michael.bethune at australiaonline.au; ausnog at lists.ausnog.net > Subject: Re: [AusNOG] Optus downtime chat + affecting SMS verification toTelstra? > > And now Singtel have returned serve and are denying it was them. > > https://www.zdnet.com/article/singtel-refutes-reports-that-its-system-upgrade-caused-optus-outage/ > > It's like watching kids trying to blame each other for who broke the window with the cricket ball. > > D > > On Wed, 15 Nov 2023 at 11:01, Luke Thompson wrote: > > > > They've blamed Singtel Internet Exchange (STiX) for the international peering route updates, at least going by anonymous sources cited by SMH. > > > > https://www.smh.com.au/technology/identity-of-third-party-who-brought- > > down-optus-network-revealed-20231114-p5ejy1.html > > > > Luke > > > > On 14 November 2023 12:37:30 pm Ben Buxton wrote: > >> > >> > >> Blaming routing updates from peers is a scapegoat and never is the cause of an outage - public BGP is the wild west and you're always getting broken information - it's your responsibility to filter those updates and (unless it's a zero-day poison packet bug) you only have yourself to blame if you fall over from them. > >> > >> If I were an optus business customer, reading that outage page would just make me even more determined to move elsewhere. > >> > >> They vaguely categorised the "what" of the outage into a big bucket (software upgrade related), but gave absolutely no useful information or explain the "why" which would regain my confidence. > >> > >> Why did this upgrade trigger an outage? > >> - Was there a behaviour/feature change they neglected to take into account? > >> - Did the upgrade require a config change that broke? > >> - Were they neglectful in following config best practices? (filtering, prefix limits, restarts, etc?) > >> - Did the new software have an unidentified bug? > >> - Why did testing not catch this problem (they do test changes...right?) > >> - How did progressive rollout still lead to this impact? (they do > >> progressive rollouts over N days/weeks...right?) > >> > >> Why did mitigation take so long? > >> - What detection/telemetry measures led them to realise the scope of the outage? (news reports dont count) > >> - Were they dependent on the downed network for oncall paging & comms? > >> - Why did their rollback plan fail? (they had a rollback plan...right?) > >> - Why was remote console/power access not working? (they have both...right?) > >> - Were they dependent on the downed network for said access? > >> - Were their playbooks/credential access dependent on the downed network? > >> > >> "We have made changes to the network to address this issue so that it cannot occur again." ... this smells like "whoops forgot to set max-prefix (with restart!)". > >> > >> Bugs, config stuff-ups, etc happen, and they will continue to happen - it is a lie to state that outages will never happen again. This is the culmination of monumental failures in the trigger, prevention and mitigation measures which cannot be fixed in a couple of days, it sounds like much deeper architectural and organisational issues need addressing. > >> > >> Many of the above failures are things that a young network will experience and learn from, but for Optus these should all be well planned for already. > >> > >> I suspect any government investigation will simply add more bureaucracy and boxes to tick rather than effect meaningful change, but one can always be hopeful... > >> > >> BB > >> > >> On Tue, 14 Nov 2023 at 13:02, Michael Bethune wrote: > >>> > >>> "Optus network received changes to routing information from an > >>> international peering network following a software upgrade" > >>> > >>> I note they are very careful to avoid nominating whose software upgrade. > >>> > >>> I also note that when they say they received routing updates, don't > >>> they limit the number of prefixes accepted by their BGP from any > >>> given peer? > >>> > >>> Sounds like a carefully crafted statement to enable them to point > >>> fingers elsewhere, not unexpected. > >>> > >>> - Michael. > >>> > >>> Quoting francisfides at mailup.net: > >>> > >>> > Looks like it was a software upgrade: > >>> > https://www.abc.net.au/news/2023-11-13/optus-identifies-cause-of-n > >>> > ationwide-outage-software-upgrade/103099902 > >>> > > >>> > Nothing in their media centre, just appears as a new box on their > >>> > outage response page: > >>> > https://www.optus.com.au/notices/outage-response > >>> > > >>> > Cheers > >>> > > >>> > ---- > >>> > Text: > >>> > > >>> > "We have been working to understand what caused the outage on > >>> > Wednesday, and we now know what the cause was and have taken steps > >>> > to ensure it will not happen again. We apologise sincerely for > >>> > letting our customers down and the inconvenience it caused. > >>> > > >>> > At around 4.05am Wednesday morning, the Optus network received > >>> > changes to routing information from an international peering > >>> > network following a software upgrade. These routing information > >>> > changes propagated through multiple layers in our network and > >>> > exceeded preset safety levels on key routers. This resulted in > >>> > those routers disconnecting from the Optus IP Core network to protect themselves. > >>> > > >>> > The restoration required a large-scale effort of the team and in > >>> > some cases required Optus to reconnect or reboot routers > >>> > physically, requiring the dispatch of people across a number of > >>> > sites in Australia. This is why restoration was progressive over the afternoon. > >>> > > >>> > Given the widespread impact of the outage, our investigations into > >>> > the issue took longer than we would have liked as we examined > >>> > several different paths to restoration. The restoration of the > >>> > network was at all times our priority and we subsequently > >>> > established the cause working together with our partners. We have > >>> > made changes to the network to address this issue so that it > >>> > cannot occur again. > >>> > > >>> > We are committed to learning from what has occurred and continuing > >>> > to work with our international vendors and partners to increase > >>> > the resilience of our network. We will also support and fully > >>> > cooperate with the reviews being undertaken by the Government and the Senate. > >>> > > >>> > We continue to invest heavily to improve the resiliency of our > >>> > network and services." > >>> > > >>> > -- > >>> > > >>> > francisfides at mailup.net > >>> > > >>> > On Thu, Nov 9, 2023, at 07:15, DaZZa wrote: > >>> >> I have all three you're asking about. > >>> >> > >>> >> But I'm very small potatoes compared to most of the members of > >>> >> this list, and my required remote footprint is correspondingly > >>> >> small, so it's easy to maintain. > >>> >> > >>> >> D > >>> >> > >>> >> On Thu, 9 Nov 2023 at 06:18, Phillip Grasso > >>> >> wrote: > >>> >>>> > >>> >>>> I mean come on, it's nearly 2024 and a [major] telco does not > >>> >>>> have remote console access? > >>> >>> > >>> >>> > >>> >>> If we send a poll out to this community, how many would be able > >>> >>> to genuinely honestly answer: > >>> >>> > >>> >>> Do you have a console or appropriate control plane access into > >>> >>> all your critical infrastructure? > >>> >>> Do you have independant out of band that does not share any > >>> >>> infrastructure with your current system(s) - with exemption for > >>> >>> physical location and power. > >>> >>> Do you have the ability to remote power control your devices? > >>> >>> > >>> >>> We know from the facebook outage in 2021 that they probably > >>> >>> didn't have the above, so its not entirely uncommon for folks > >>> >>> to have *proper independant* console and remote access. > >>> >>> > >>> >>> > >>> >>> I empathize with the Optus team and their customers who have > >>> >>> been negatively impacted by this incident. I sincerely hope that > >>> >>> some positive outcomes can emerge from this situation, including: > >>> >>> > >>> >>> - Attention to critical infrastructure resilience > >>> >>> - BGP clue increases > >>> >>> - Incident management improves > >>> >>> (I'm sure there's more). > >>> >>> > >>> >>> Network is a black box to most people and I think a large chunk > >>> >>> of Australia now knows what it feels like to not have it. > >>> >>> > >>> >>> > >>> >>> On Wed, 8 Nov 2023 at 11:06, Ben Buxton wrote: > >>> >>>> > >>> >>>> > >>> >>>> > >>> >>>> On Wed, 8 Nov 2023 at 10:14, DaZZa wrote: > >>> >>>>> > >>> >>>>> Yeah, I'd be willing to bet that it's a change which wasn't > >>> >>>>> thoroughly tested before being rolled out, and which had an > >>> >>>>> inadequate backout plan. > >>> >>>> > >>> >>>> > >>> >>>> Also, "Our on-site technician is actively prioritising > >>> >>>> establishing a console connection.". > >>> >>>> > >>> >>>> I mean come on, it's nearly 2024 and a [major] telco does not > >>> >>>> have remote console access? Whilst I'm looking forward to > >>> >>>> enthusiastically reading the PM, I'll have to book a physio > >>> >>>> appointment in advance due to neck strain from all the head > >>> >>>> shaking it'll likely induce. > >>> >>>> > >>> >>>> BB > >>> >>>> > >>> >>>> > >>> >>>>> > >>> >>>>> > >>> >>>>> Interestingly, my Optus mobile actually had a valid connection > >>> >>>>> for a short time - wasn't able to actually DO anything, but > >>> >>>>> was connected to the OPtus network - but it's now gone to "SOS" mode. > >>> >>>>> > >>> >>>>> D > >>> >>>>> > >>> >>>>> On Wed, 8 Nov 2023 at 10:01, John Edwards wrote: > >>> >>>>> > > >>> >>>>> > The 4am Wednesday morning outage start looks suspiciously > >>> >>>>> > like > >>> >>>>> a firmware upgrade window. > >>> >>>>> > > >>> >>>>> > I note that Optus devices where I am are showing "SoS" which > >>> >>>>> indicates the tower is unable to reach the location register, > >>> >>>>> which presumably is on a private network and indicative of a > >>> >>>>> pretty major fault rather than just IP. > >>> >>>>> > > >>> >>>>> > John > >>> >>>>> > > >>> >>>>> > > >>> >>>>> > On Wed, 8 Nov 2023 at 09:10, DaZZa wrote: > >>> >>>>> >> > >>> >>>>> >> The Optus hamster finally died of old age. > >>> >>>>> >> > >>> >>>>> >> I would suggest your SMS issues would be caused by whoever > >>> >>>>> >> is issuing the SMS using Optus - not so much by the Telstra end receiving it. > >>> >>>>> >> > >>> >>>>> >> Anecdotally, Optus enterprise/wholesale appears to be still > >>> >>>>> >> functional > >>> >>>>> >> - at least my link appears to be working fine - and my BGP > >>> >>>>> >> advertisements are still being seen overseas - seems to be > >>> >>>>> >> only NBN and mobile based services which are busted > >>> >>>>> >> > >>> >>>>> >> D > >>> >>>>> >> > >>> >>>>> >> On Wed, 8 Nov 2023 at 09:27, wrote: > >>> >>>>> >> > > >>> >>>>> >> > Morning all, > >>> >>>>> >> > Hope the chaos isn't too hard on your work/family. > >>> >>>>> >> > I have had trouble with a couple of SMS verifications > >>> >>>>> coming through to me, my Telstra number. Is this related? > >>> >>>>> >> > > >>> >>>>> >> > Any general banter around the downtime would be fine too > >>> >>>>> >> > - > >>> >>>>> looks like it all began at 4.07am AEDT? > >>> >>>>> >> > > >>> >>>>> >> > Cheers > >>> >>>>> >> > > >>> >>>>> >> > -- > >>> >>>>> >> > > >>> >>>>> >> > francisfides at mailup.net > >>> >>>>> >> > _______________________________________________ > >>> >>>>> >> > AusNOG mailing list > >>> >>>>> >> > AusNOG at lists.ausnog.net > >>> >>>>> >> > https://lists.ausnog.net/mailman/listinfo/ausnog > >>> >>>>> >> > >>> >>>>> >> > >>> >>>>> >> > >>> >>>>> >> -- > >>> >>>>> >> veg?e?tar?i?an: > >>> >>>>> >> Ancient tribal slang for the village idiot who can't hunt, > >>> >>>>> fish or ride > >>> >>>>> >> _______________________________________________ > >>> >>>>> >> AusNOG mailing list > >>> >>>>> >> AusNOG at lists.ausnog.net > >>> >>>>> >> https://lists.ausnog.net/mailman/listinfo/ausnog > >>> >>>>> > >>> >>>>> > >>> >>>>> > >>> >>>>> -- > >>> >>>>> veg?e?tar?i?an: > >>> >>>>> Ancient tribal slang for the village idiot who can't hunt, > >>> >>>>> fish or ride _______________________________________________ > >>> >>>>> AusNOG mailing list > >>> >>>>> AusNOG at lists.ausnog.net > >>> >>>>> https://lists.ausnog.net/mailman/listinfo/ausnog > >>> >>>> > >>> >>>> _______________________________________________ > >>> >>>> AusNOG mailing list > >>> >>>> AusNOG at lists.ausnog.net > >>> >>>> https://lists.ausnog.net/mailman/listinfo/ausnog > >>> >> > >>> >> > >>> >> > >>> >> -- > >>> >> veg?e?tar?i?an: > >>> >> Ancient tribal slang for the village idiot who can't hunt, fish > >>> >> or ride _______________________________________________ > >>> >> AusNOG mailing list > >>> >> AusNOG at lists.ausnog.net > >>> >> https://lists.ausnog.net/mailman/listinfo/ausnog > >>> > _______________________________________________ > >>> > AusNOG mailing list > >>> > AusNOG at lists.ausnog.net > >>> > https://lists.ausnog.net/mailman/listinfo/ausnog > >>> > > >>> > >>> > >>> > >>> > >>> _______________________________________________ > >>> AusNOG mailing list > >>> AusNOG at lists.ausnog.net > >>> https://lists.ausnog.net/mailman/listinfo/ausnog > >> > >> _______________________________________________ > >> AusNOG mailing list > >> AusNOG at lists.ausnog.net > >> https://lists.ausnog.net/mailman/listinfo/ausnog > >> > > > > _______________________________________________ > > AusNOG mailing list > > AusNOG at lists.ausnog.net > > https://lists.ausnog.net/mailman/listinfo/ausnog > > > > -- > veg?e?tar?i?an: > Ancient tribal slang for the village idiot who can't hunt, fish or ride _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog -- veg?e?tar?i?an: Ancient tribal slang for the village idiot who can't hunt, fish or ride From tony at wicks.co.nz Fri Nov 17 13:02:03 2023 From: tony at wicks.co.nz (Tony Wicks) Date: Fri, 17 Nov 2023 15:02:03 +1300 Subject: [AusNOG] Optus downtime chat + affecting SMS verification toTelstra? In-Reply-To: References: <4f8931e0-fd97-41fc-8489-c4ec5b7dec90@app.fastmail.com> <20231114130235.01014a8se84okgc4@horde-2.ozonline.com.au> <18bd04713e8.28b2.934dbec5914d2454830359510b9fa213@tncrew.com.au> Message-ID: <018701da18fa$1149c9b0$33dd5d10$@wicks.co.nz> To be fair, Assuming there were config issues (i.e. the lack of maximum-prefixes and the lack of filtering preventing large route tables hitting devices that can not carry full tables) the behaviour of a network device when its RIB/FIB or memory is exceeded also significantly comes into play. Dropping BGP is fine, crashing the router so it requires a hard reset is another case entirely. In my experience (I have not used Cisco's in a telco environment for many years however) Cisco devices have been much more pre-disposed to crash catastrophically than over vendor devices like Nokia or Juniper. -----Original Message----- From: AusNOG On Behalf Of DaZZa Sent: Friday, November 17, 2023 2:38 PM To: Andrew Oakeley Cc: michael.bethune at australiaonline.au; Luke Thompson ; ausnog at lists.ausnog.net Subject: Re: [AusNOG] Optus downtime chat + affecting SMS verification toTelstra? What a load of crap. The root cause was they're morons, and configured the routers incorrectly. Cisco had nothing to do with it. I'll bet the routers behaved exactly as they were intended to behave. From casper.oshea at gmail.com Fri Nov 17 13:36:02 2023 From: casper.oshea at gmail.com (Christopher O'Shea) Date: Fri, 17 Nov 2023 02:36:02 +0000 Subject: [AusNOG] Optus downtime chat + affecting SMS verification toTelstra? In-Reply-To: <018701da18fa$1149c9b0$33dd5d10$@wicks.co.nz> References: <4f8931e0-fd97-41fc-8489-c4ec5b7dec90@app.fastmail.com> <20231114130235.01014a8se84okgc4@horde-2.ozonline.com.au> <18bd04713e8.28b2.934dbec5914d2454830359510b9fa213@tncrew.com.au> <018701da18fa$1149c9b0$33dd5d10$@wicks.co.nz> Message-ID: I wouldn't be so quick to blame it on a single thing. We have all been there, An incident always comes down to many things not going the way you think. Reading between the lines, I see that a peer's network creates larger than "normal" routes, and seeing they called out IPv6 in their submission to Senate [1] Lack of filtering of v6 for that peer due to an oversight or misunderstanding of the template/group between v4 and v6. Then, when it was shared with their PE routers (Which seem to be Cisco) On the ASK9K (Not sure what they use), the default limit of 524288 [2] for v6 could lead to the session's termination by default. We should read these reports and understand if the same thing could happen to your network, what protection you have to stop this, and your device's default behaviour. I would like to know more about their out-of-band and why it had issues. (Could it be that DNS broke, issue getting to internal documentation or was the password vault access broken, or the IP limit of the OOB device was too tight). Chris O'Shea [1] https://www.aph.gov.au/DocumentStore.ashx?id=2ed95079-023d-49d5-87fd-d9029740629b&subId=750333 reports of the Optus outage [2] https://www.cisco.com/c/en/us/td/docs/routers/asr9000/software/routing/command/reference/b-routing-cr-asr9000/bgp-commands.html#wp3192417938 On Fri, Nov 17, 2023 at 2:02?AM Tony Wicks wrote: > To be fair, Assuming there were config issues (i.e. the lack of > maximum-prefixes and the lack of filtering preventing large route tables > hitting devices that can not carry full tables) the behaviour of a network > device when its RIB/FIB or memory is exceeded also significantly comes into > play. Dropping BGP is fine, crashing the router so it requires a hard reset > is another case entirely. In my experience (I have not used Cisco's in a > telco environment for many years however) Cisco devices have been much more > pre-disposed to crash catastrophically than over vendor devices like Nokia > or Juniper. > > > > -----Original Message----- > From: AusNOG On Behalf Of DaZZa > Sent: Friday, November 17, 2023 2:38 PM > To: Andrew Oakeley > Cc: michael.bethune at australiaonline.au; Luke Thompson < > luke.t at tncrew.com.au>; ausnog at lists.ausnog.net > Subject: Re: [AusNOG] Optus downtime chat + affecting SMS verification > toTelstra? > > What a load of crap. > > The root cause was they're morons, and configured the routers incorrectly. > > Cisco had nothing to do with it. I'll bet the routers behaved exactly as > they were intended to behave. > > > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From luke.t at tncrew.com.au Fri Nov 17 14:39:36 2023 From: luke.t at tncrew.com.au (Luke Thompson) Date: Fri, 17 Nov 2023 13:39:36 +1000 Subject: [AusNOG] Optus downtime chat + affecting SMS verification toTelstra? In-Reply-To: References: <4f8931e0-fd97-41fc-8489-c4ec5b7dec90@app.fastmail.com> <20231114130235.01014a8se84okgc4@horde-2.ozonline.com.au> <18bd04713e8.28b2.934dbec5914d2454830359510b9fa213@tncrew.com.au> Message-ID: <574aebe4-51ef-4d28-86c5-42d18471bc38@tncrew.com.au> I'd muse they pay enough that there's an agreement made to wear that. Once it's blown over, it's just another outage blip in the past. They do happen; no person nor network is infallible. As Ben highlights though, Optus seems rough. Luke On 17/11/2023 10:31 am, Andrew Oakeley wrote: > And in the senate enquiry this morning they both blamed Cisco > > "The trigger was the Singtel outage, but the root cause was Cisco." > > https://www.abc.net.au/news/2023-11-17/asx-markets-business-live-news-optus-outage-senate-inquiry/103115518 > > -----Original Message----- > From: AusNOG On Behalf Of DaZZa > Sent: Friday, November 17, 2023 8:15 AM > To: Luke Thompson > Cc: michael.bethune at australiaonline.au; ausnog at lists.ausnog.net > Subject: Re: [AusNOG] Optus downtime chat + affecting SMS verification toTelstra? > > And now Singtel have returned serve and are denying it was them. > > https://www.zdnet.com/article/singtel-refutes-reports-that-its-system-upgrade-caused-optus-outage/ > > It's like watching kids trying to blame each other for who broke the window with the cricket ball. > > D > > On Wed, 15 Nov 2023 at 11:01, Luke Thompson wrote: >> They've blamed Singtel Internet Exchange (STiX) for the international peering route updates, at least going by anonymous sources cited by SMH. >> >> https://www.smh.com.au/technology/identity-of-third-party-who-brought- >> down-optus-network-revealed-20231114-p5ejy1.html >> >> Luke >> >> On 14 November 2023 12:37:30 pm Ben Buxton wrote: >>> >>> Blaming routing updates from peers is a scapegoat and never is the cause of an outage - public BGP is the wild west and you're always getting broken information - it's your responsibility to filter those updates and (unless it's a zero-day poison packet bug) you only have yourself to blame if you fall over from them. >>> >>> If I were an optus business customer, reading that outage page would just make me even more determined to move elsewhere. >>> >>> They vaguely categorised the "what" of the outage into a big bucket (software upgrade related), but gave absolutely no useful information or explain the "why" which would regain my confidence. >>> >>> Why did this upgrade trigger an outage? >>> - Was there a behaviour/feature change they neglected to take into account? >>> - Did the upgrade require a config change that broke? >>> - Were they neglectful in following config best practices? (filtering, prefix limits, restarts, etc?) >>> - Did the new software have an unidentified bug? >>> - Why did testing not catch this problem (they do test changes...right?) >>> - How did progressive rollout still lead to this impact? (they do >>> progressive rollouts over N days/weeks...right?) >>> >>> Why did mitigation take so long? >>> - What detection/telemetry measures led them to realise the scope of the outage? (news reports dont count) >>> - Were they dependent on the downed network for oncall paging & comms? >>> - Why did their rollback plan fail? (they had a rollback plan...right?) >>> - Why was remote console/power access not working? (they have both...right?) >>> - Were they dependent on the downed network for said access? >>> - Were their playbooks/credential access dependent on the downed network? >>> >>> "We have made changes to the network to address this issue so that it cannot occur again." ... this smells like "whoops forgot to set max-prefix (with restart!)". >>> >>> Bugs, config stuff-ups, etc happen, and they will continue to happen - it is a lie to state that outages will never happen again. This is the culmination of monumental failures in the trigger, prevention and mitigation measures which cannot be fixed in a couple of days, it sounds like much deeper architectural and organisational issues need addressing. >>> >>> Many of the above failures are things that a young network will experience and learn from, but for Optus these should all be well planned for already. >>> >>> I suspect any government investigation will simply add more bureaucracy and boxes to tick rather than effect meaningful change, but one can always be hopeful... >>> >>> BB >>> >>> On Tue, 14 Nov 2023 at 13:02, Michael Bethune wrote: >>>> "Optus network received changes to routing information from an >>>> international peering network following a software upgrade" >>>> >>>> I note they are very careful to avoid nominating whose software upgrade. >>>> >>>> I also note that when they say they received routing updates, don't >>>> they limit the number of prefixes accepted by their BGP from any >>>> given peer? >>>> >>>> Sounds like a carefully crafted statement to enable them to point >>>> fingers elsewhere, not unexpected. >>>> >>>> - Michael. >>>> >>>> Quoting francisfides at mailup.net: >>>> >>>>> Looks like it was a software upgrade: >>>>> https://www.abc.net.au/news/2023-11-13/optus-identifies-cause-of-n >>>>> ationwide-outage-software-upgrade/103099902 >>>>> >>>>> Nothing in their media centre, just appears as a new box on their >>>>> outage response page: >>>>> https://www.optus.com.au/notices/outage-response >>>>> >>>>> Cheers >>>>> >>>>> ---- >>>>> Text: >>>>> >>>>> "We have been working to understand what caused the outage on >>>>> Wednesday, and we now know what the cause was and have taken steps >>>>> to ensure it will not happen again. We apologise sincerely for >>>>> letting our customers down and the inconvenience it caused. >>>>> >>>>> At around 4.05am Wednesday morning, the Optus network received >>>>> changes to routing information from an international peering >>>>> network following a software upgrade. These routing information >>>>> changes propagated through multiple layers in our network and >>>>> exceeded preset safety levels on key routers. This resulted in >>>>> those routers disconnecting from the Optus IP Core network to protect themselves. >>>>> >>>>> The restoration required a large-scale effort of the team and in >>>>> some cases required Optus to reconnect or reboot routers >>>>> physically, requiring the dispatch of people across a number of >>>>> sites in Australia. This is why restoration was progressive over the afternoon. >>>>> >>>>> Given the widespread impact of the outage, our investigations into >>>>> the issue took longer than we would have liked as we examined >>>>> several different paths to restoration. The restoration of the >>>>> network was at all times our priority and we subsequently >>>>> established the cause working together with our partners. We have >>>>> made changes to the network to address this issue so that it >>>>> cannot occur again. >>>>> >>>>> We are committed to learning from what has occurred and continuing >>>>> to work with our international vendors and partners to increase >>>>> the resilience of our network. We will also support and fully >>>>> cooperate with the reviews being undertaken by the Government and the Senate. >>>>> >>>>> We continue to invest heavily to improve the resiliency of our >>>>> network and services." >>>>> >>>>> -- >>>>> >>>>> francisfides at mailup.net >>>>> >>>>> On Thu, Nov 9, 2023, at 07:15, DaZZa wrote: >>>>>> I have all three you're asking about. >>>>>> >>>>>> But I'm very small potatoes compared to most of the members of >>>>>> this list, and my required remote footprint is correspondingly >>>>>> small, so it's easy to maintain. >>>>>> >>>>>> D >>>>>> >>>>>> On Thu, 9 Nov 2023 at 06:18, Phillip Grasso >>>>>> wrote: >>>>>>>> I mean come on, it's nearly 2024 and a [major] telco does not >>>>>>>> have remote console access? >>>>>>> >>>>>>> If we send a poll out to this community, how many would be able >>>>>>> to genuinely honestly answer: >>>>>>> >>>>>>> Do you have a console or appropriate control plane access into >>>>>>> all your critical infrastructure? >>>>>>> Do you have independant out of band that does not share any >>>>>>> infrastructure with your current system(s) - with exemption for >>>>>>> physical location and power. >>>>>>> Do you have the ability to remote power control your devices? >>>>>>> >>>>>>> We know from the facebook outage in 2021 that they probably >>>>>>> didn't have the above, so its not entirely uncommon for folks >>>>>>> to have *proper independant* console and remote access. >>>>>>> >>>>>>> >>>>>>> I empathize with the Optus team and their customers who have >>>>>>> been negatively impacted by this incident. I sincerely hope that >>>>>>> some positive outcomes can emerge from this situation, including: >>>>>>> >>>>>>> - Attention to critical infrastructure resilience >>>>>>> - BGP clue increases >>>>>>> - Incident management improves >>>>>>> (I'm sure there's more). >>>>>>> >>>>>>> Network is a black box to most people and I think a large chunk >>>>>>> of Australia now knows what it feels like to not have it. >>>>>>> >>>>>>> >>>>>>> On Wed, 8 Nov 2023 at 11:06, Ben Buxton wrote: >>>>>>>> >>>>>>>> >>>>>>>> On Wed, 8 Nov 2023 at 10:14, DaZZa wrote: >>>>>>>>> Yeah, I'd be willing to bet that it's a change which wasn't >>>>>>>>> thoroughly tested before being rolled out, and which had an >>>>>>>>> inadequate backout plan. >>>>>>>> >>>>>>>> Also, "Our on-site technician is actively prioritising >>>>>>>> establishing a console connection.". >>>>>>>> >>>>>>>> I mean come on, it's nearly 2024 and a [major] telco does not >>>>>>>> have remote console access? Whilst I'm looking forward to >>>>>>>> enthusiastically reading the PM, I'll have to book a physio >>>>>>>> appointment in advance due to neck strain from all the head >>>>>>>> shaking it'll likely induce. >>>>>>>> >>>>>>>> BB >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Interestingly, my Optus mobile actually had a valid connection >>>>>>>>> for a short time - wasn't able to actually DO anything, but >>>>>>>>> was connected to the OPtus network - but it's now gone to "SOS" mode. >>>>>>>>> >>>>>>>>> D >>>>>>>>> >>>>>>>>> On Wed, 8 Nov 2023 at 10:01, John Edwards wrote: >>>>>>>>>> The 4am Wednesday morning outage start looks suspiciously >>>>>>>>>> like >>>>>>>>> a firmware upgrade window. >>>>>>>>>> I note that Optus devices where I am are showing "SoS" which >>>>>>>>> indicates the tower is unable to reach the location register, >>>>>>>>> which presumably is on a private network and indicative of a >>>>>>>>> pretty major fault rather than just IP. >>>>>>>>>> John >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, 8 Nov 2023 at 09:10, DaZZa wrote: >>>>>>>>>>> The Optus hamster finally died of old age. >>>>>>>>>>> >>>>>>>>>>> I would suggest your SMS issues would be caused by whoever >>>>>>>>>>> is issuing the SMS using Optus - not so much by the Telstra end receiving it. >>>>>>>>>>> >>>>>>>>>>> Anecdotally, Optus enterprise/wholesale appears to be still >>>>>>>>>>> functional >>>>>>>>>>> - at least my link appears to be working fine - and my BGP >>>>>>>>>>> advertisements are still being seen overseas - seems to be >>>>>>>>>>> only NBN and mobile based services which are busted >>>>>>>>>>> >>>>>>>>>>> D >>>>>>>>>>> >>>>>>>>>>> On Wed, 8 Nov 2023 at 09:27, wrote: >>>>>>>>>>>> Morning all, >>>>>>>>>>>> Hope the chaos isn't too hard on your work/family. >>>>>>>>>>>> I have had trouble with a couple of SMS verifications >>>>>>>>> coming through to me, my Telstra number. Is this related? >>>>>>>>>>>> Any general banter around the downtime would be fine too >>>>>>>>>>>> - >>>>>>>>> looks like it all began at 4.07am AEDT? >>>>>>>>>>>> Cheers >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> francisfides at mailup.net >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> AusNOG mailing list >>>>>>>>>>>> AusNOG at lists.ausnog.net >>>>>>>>>>>> https://lists.ausnog.net/mailman/listinfo/ausnog >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> veg?e?tar?i?an: >>>>>>>>>>> Ancient tribal slang for the village idiot who can't hunt, >>>>>>>>> fish or ride >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> AusNOG mailing list >>>>>>>>>>> AusNOG at lists.ausnog.net >>>>>>>>>>> https://lists.ausnog.net/mailman/listinfo/ausnog >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> veg?e?tar?i?an: >>>>>>>>> Ancient tribal slang for the village idiot who can't hunt, >>>>>>>>> fish or ride _______________________________________________ >>>>>>>>> AusNOG mailing list >>>>>>>>> AusNOG at lists.ausnog.net >>>>>>>>> https://lists.ausnog.net/mailman/listinfo/ausnog >>>>>>>> _______________________________________________ >>>>>>>> AusNOG mailing list >>>>>>>> AusNOG at lists.ausnog.net >>>>>>>> https://lists.ausnog.net/mailman/listinfo/ausnog >>>>>> >>>>>> >>>>>> -- >>>>>> veg?e?tar?i?an: >>>>>> Ancient tribal slang for the village idiot who can't hunt, fish >>>>>> or ride _______________________________________________ >>>>>> AusNOG mailing list >>>>>> AusNOG at lists.ausnog.net >>>>>> https://lists.ausnog.net/mailman/listinfo/ausnog >>>>> _______________________________________________ >>>>> AusNOG mailing list >>>>> AusNOG at lists.ausnog.net >>>>> https://lists.ausnog.net/mailman/listinfo/ausnog >>>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> AusNOG mailing list >>>> AusNOG at lists.ausnog.net >>>> https://lists.ausnog.net/mailman/listinfo/ausnog >>> _______________________________________________ >>> AusNOG mailing list >>> AusNOG at lists.ausnog.net >>> https://lists.ausnog.net/mailman/listinfo/ausnog >>> >> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog > > > -- > veg?e?tar?i?an: > Ancient tribal slang for the village idiot who can't hunt, fish or ride _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog From notroger at gmail.com Fri Nov 17 14:52:06 2023 From: notroger at gmail.com (Roger) Date: Fri, 17 Nov 2023 14:52:06 +1100 Subject: [AusNOG] Rack Mounting Brackets for arista 7050 Message-ID: Hey All, Need to find some rack mounting brackets for an arista 7050 in Sydney, does any one have any spare for sale or know where to source them? thanks, Roger. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bb.ausnog at bb.cactii.net Fri Nov 17 15:56:37 2023 From: bb.ausnog at bb.cactii.net (Ben Buxton) Date: Fri, 17 Nov 2023 15:56:37 +1100 Subject: [AusNOG] Optus downtime chat + affecting SMS verification toTelstra? In-Reply-To: References: <4f8931e0-fd97-41fc-8489-c4ec5b7dec90@app.fastmail.com> <20231114130235.01014a8se84okgc4@horde-2.ozonline.com.au> <18bd04713e8.28b2.934dbec5914d2454830359510b9fa213@tncrew.com.au> <018701da18fa$1149c9b0$33dd5d10$@wicks.co.nz> Message-ID: It looks like the outage was largely due to a max-prefix issue then (or lack thereof). And their change management processes don't seem to come into play (except perhaps during restoration?). Given that this was from prefixes received over an exchange, I'm curious to know why no-one else seems to have suffered as it's unlikely just 1 peer would be affected. Something glaringly missing from the Senate submission is information about why the restoration took so long. 6 hours is an embarrasingly long time to fix what was essentially a max-prefix trip. I would really like to know more details about: - OOB access - Remote power / reboot capability - Potential issues about comms between engineers and otherwise accessing a downed network - i bet it took a long time to contact some key engineers. Again it looks like they explained what happened (max prefix trip and then engineers working + onsite for 6 hours to mitigate). But not why they feel 6 hours was an acceptable duration - the submission seems to imply 6 hours is a normal investigation time. This aspect really needs to be picked apart further. Outages happen - it's a fact of life. But prevention only goes so far, you need to build and test robust mitigation strategies and incident management plans. On Fri, 17 Nov 2023 at 13:36, Christopher O'Shea wrote: > I wouldn't be so quick to blame it on a single thing. We have all been > there, An incident always comes down to many things not going the way you > think. > > Reading between the lines, I see that a peer's network creates larger than > "normal" routes, and seeing they called out IPv6 in their submission to > Senate [1] > Lack of filtering of v6 for that peer due to an oversight or > misunderstanding of the template/group between v4 and v6. > > Then, when it was shared with their PE routers (Which seem to be Cisco) On > the ASK9K (Not sure what they use), the default limit of 524288 [2] for v6 > could lead to the session's termination by default. > > We should read these reports and understand if the same thing could happen > to your network, what protection you have to stop this, and your device's > default behaviour. > > I would like to know more about their out-of-band and why it had issues. > (Could it be that DNS broke, issue getting to internal documentation or was > the password vault access broken, or the IP limit of the OOB device was too > tight). > > Chris O'Shea > > [1] > https://www.aph.gov.au/DocumentStore.ashx?id=2ed95079-023d-49d5-87fd-d9029740629b&subId=750333 > reports of the Optus outage > [2] > https://www.cisco.com/c/en/us/td/docs/routers/asr9000/software/routing/command/reference/b-routing-cr-asr9000/bgp-commands.html#wp3192417938 > > > > On Fri, Nov 17, 2023 at 2:02?AM Tony Wicks wrote: > >> To be fair, Assuming there were config issues (i.e. the lack of >> maximum-prefixes and the lack of filtering preventing large route tables >> hitting devices that can not carry full tables) the behaviour of a network >> device when its RIB/FIB or memory is exceeded also significantly comes into >> play. Dropping BGP is fine, crashing the router so it requires a hard reset >> is another case entirely. In my experience (I have not used Cisco's in a >> telco environment for many years however) Cisco devices have been much more >> pre-disposed to crash catastrophically than over vendor devices like Nokia >> or Juniper. >> >> >> >> -----Original Message----- >> From: AusNOG On Behalf Of DaZZa >> Sent: Friday, November 17, 2023 2:38 PM >> To: Andrew Oakeley >> Cc: michael.bethune at australiaonline.au; Luke Thompson < >> luke.t at tncrew.com.au>; ausnog at lists.ausnog.net >> Subject: Re: [AusNOG] Optus downtime chat + affecting SMS verification >> toTelstra? >> >> What a load of crap. >> >> The root cause was they're morons, and configured the routers incorrectly. >> >> Cisco had nothing to do with it. I'll bet the routers behaved exactly as >> they were intended to behave. >> >> >> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog >> > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From romislam at gmail.com Wed Nov 22 11:29:30 2023 From: romislam at gmail.com (Nurul Islam) Date: Wed, 22 Nov 2023 10:29:30 +1000 Subject: [AusNOG] Full Internet routing table router recommendation Message-ID: Hi All, Good day. We are looking at an option to accept full Internet routing table (approx 1Million Plus) from two ISP. Total route the edge router probably needs to handle will be around 2 Million. What is the Cisco 1001-X equivalent from the current router range? Has anyone used Cisco C8500L-8S4X with 16GB RAM (with performance licence) and it handle this load on 2Mill internet routes? Regards -N -------------- next part -------------- An HTML attachment was scrubbed... URL: From cameron.murray at gmail.com Wed Nov 22 11:58:28 2023 From: cameron.murray at gmail.com (Cameron Murray) Date: Wed, 22 Nov 2023 10:58:28 +1000 Subject: [AusNOG] Full Internet routing table router recommendation In-Reply-To: References: Message-ID: Hi Nurul, What sort of throughput and port density are you looking for? Kind Regards Cameron On Wed, Nov 22, 2023 at 10:30?AM Nurul Islam wrote: > Hi All, > > Good day. We are looking at an option to accept full Internet routing > table (approx 1Million Plus) from two ISP. Total route the edge router > probably needs to handle will be around 2 Million. > > What is the Cisco 1001-X equivalent from the current router range? Has > anyone used Cisco C8500L-8S4X with 16GB RAM (with performance licence) and > it handle this load on 2Mill internet routes? > > Regards > > > -N > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mitchkelly24 at gmail.com Wed Nov 22 16:15:37 2023 From: mitchkelly24 at gmail.com (Mitch Kelly) Date: Wed, 22 Nov 2023 13:15:37 +0800 Subject: [AusNOG] Full Internet routing table router recommendation In-Reply-To: References: Message-ID: Hi, We use the Mik CCR2216 with a global table, Routing table is using about 1GB Ram out of 16Gb, Might still be out of the price range (5k), But it sure does beat 50k for the 8500L with only a base license. On Wed, Nov 22, 2023 at 8:58?AM Cameron Murray wrote: > Hi Nurul, > > What sort of throughput and port density are you looking for? > > Kind Regards > > Cameron > > > On Wed, Nov 22, 2023 at 10:30?AM Nurul Islam wrote: > >> Hi All, >> >> Good day. We are looking at an option to accept full Internet routing >> table (approx 1Million Plus) from two ISP. Total route the edge router >> probably needs to handle will be around 2 Million. >> >> What is the Cisco 1001-X equivalent from the current router range? Has >> anyone used Cisco C8500L-8S4X with 16GB RAM (with performance licence) and >> it handle this load on 2Mill internet routes? >> >> Regards >> >> >> -N >> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog >> > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex at samad.com.au Thu Nov 23 10:41:37 2023 From: alex at samad.com.au (Alex Samad) Date: Thu, 23 Nov 2023 10:41:37 +1100 Subject: [AusNOG] Full Internet routing table router recommendation In-Reply-To: References: Message-ID: Hi I have used CCR1072 + CCR1036, taken 4 full feeds from upstream. No a problem. the issue I found with the is convergence, the older models with routeros 6 from memory was single threaded bgp daemon. I believe in routeros 7 it was meant to have moved to multhreaded and faster convergence Alex On Wed, Nov 22, 2023 at 4:16?PM Mitch Kelly wrote: > Hi, > > We use the Mik CCR2216 with a global table, Routing table is using about > 1GB Ram out of 16Gb, Might still be out of the price range (5k), But it > sure does beat 50k for the 8500L with only a base license. > > On Wed, Nov 22, 2023 at 8:58?AM Cameron Murray > wrote: > >> Hi Nurul, >> >> What sort of throughput and port density are you looking for? >> >> Kind Regards >> >> Cameron >> >> >> On Wed, Nov 22, 2023 at 10:30?AM Nurul Islam wrote: >> >>> Hi All, >>> >>> Good day. We are looking at an option to accept full Internet routing >>> table (approx 1Million Plus) from two ISP. Total route the edge router >>> probably needs to handle will be around 2 Million. >>> >>> What is the Cisco 1001-X equivalent from the current router range? Has >>> anyone used Cisco C8500L-8S4X with 16GB RAM (with performance licence) and >>> it handle this load on 2Mill internet routes? >>> >>> Regards >>> >>> >>> -N >>> _______________________________________________ >>> AusNOG mailing list >>> AusNOG at lists.ausnog.net >>> https://lists.ausnog.net/mailman/listinfo/ausnog >>> >> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog >> > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike at ozonline.com.au Fri Nov 24 09:35:27 2023 From: mike at ozonline.com.au (Michael Bethune) Date: Fri, 24 Nov 2023 09:35:27 +1100 Subject: [AusNOG] Question: Multilink PPP using L2 PPP authenticated TC4 like AAPT NWB? Message-ID: <20231124093527.zwjy8rez0o8040cs@horde-2.ozonline.com.au> Question: multi link PPP using L2 PPP authenticated TC4 like with AAPT NWB? Hi my question, is it possible to bond multiple L2 NBN TC4 links acquired via an upstream wholesale aggregator such as AAPT. Ideally I'd have multi link ppp CPE on customer site and our Layer 2 LNS would either terminate the multi link ppp itself or multi hop it off to another LNS that does. My understanding is that the PPP of the individual links in multi link isn't plain vanilla PPP, has additional packet headers, so the wholesaler, for instance AAPT, may not pass it on faithfully only being primed to pass vanilla PPP and so it mayn't work. Or is there a better way of doing this or no way? Thoughts? - Michael. From ltd at interlink.com.au Fri Nov 24 11:35:02 2023 From: ltd at interlink.com.au (Lincoln Dale) Date: Fri, 24 Nov 2023 11:35:02 +1100 Subject: [AusNOG] Question: Multilink PPP using L2 PPP authenticated TC4 like AAPT NWB? In-Reply-To: <20231124093527.zwjy8rez0o8040cs@horde-2.ozonline.com.au> References: <20231124093527.zwjy8rez0o8040cs@horde-2.ozonline.com.au> Message-ID: On Fri, Nov 24, 2023 at 9:36?AM Michael Bethune wrote: > Or is there a better way of doing this or no way? > Thoughts? > Do you really need to stretch L2? I'd suggest you'd be far better off running an overlay over the top if you had to. But in fact, you're really just better off not doing L2. Unless you really like troubleshooting. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sapage at sapage.net Fri Nov 24 11:46:31 2023 From: sapage at sapage.net (S T) Date: Fri, 24 Nov 2023 10:46:31 +1000 Subject: [AusNOG] Question: Multilink PPP using L2 PPP authenticated TC4 like AAPT NWB? In-Reply-To: References: <20231124093527.zwjy8rez0o8040cs@horde-2.ozonline.com.au> Message-ID: Friends do not let friends build large layer 2 networks! ST On Fri, Nov 24, 2023 at 10:35?AM Lincoln Dale wrote: > On Fri, Nov 24, 2023 at 9:36?AM Michael Bethune > wrote: > >> Or is there a better way of doing this or no way? >> Thoughts? >> > > Do you really need to stretch L2? I'd suggest you'd be far better off > running an overlay over the top if you had to. > But in fact, you're really just better off not doing L2. Unless you really > like troubleshooting. > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From admin at comkal.net Fri Nov 24 22:20:06 2023 From: admin at comkal.net (Ian Manners) Date: Fri, 24 Nov 2023 19:20:06 +0800 (AWS) Subject: [AusNOG] Stop shaming service providers for outages... Message-ID: <100.485f0300e6866065.003@comkal.net> Excellent article on The Register. Be nice to figure out how to start something as Geoff has mentioned, and how to fund it. Cheers Ian Manners From justintwiss at gmail.com Fri Nov 24 22:30:52 2023 From: justintwiss at gmail.com (Justin Twiss) Date: Fri, 24 Nov 2023 19:30:52 +0800 Subject: [AusNOG] Fwd: Would anyone in NextDC P1 happen to have a USB serial (+console) cable adapter we could borrow? In-Reply-To: References: Message-ID: Hey guys, Would anyone in NextDC P1 happen to have a USB serial (+console) cable adapter we could borrow urgently? One of our after-hours guys has been caught on the hop without a USB serial console cable (RJ45 termination) out at NextDC P1 -- Was wondering if anyone had one on-site we could borrow for a short period of time? Have checked the vending machine and whilst there's the great stocks of ethernet and fibre, there unfortunately isn't any USB to serial adapters. -JT -------------- next part -------------- An HTML attachment was scrubbed... URL: From jenn at jenn.id.au Fri Nov 24 22:32:33 2023 From: jenn at jenn.id.au (Jennifer Sims) Date: Fri, 24 Nov 2023 22:32:33 +1100 Subject: [AusNOG] Fwd: Would anyone in NextDC P1 happen to have a USB serial (+console) cable adapter we could borrow? In-Reply-To: References: Message-ID: Did you check with NextDC staff on site? They may be able to loan one out? On Fri, Nov 24, 2023 at 10:31?PM Justin Twiss wrote: > > > Hey guys, > > Would anyone in NextDC P1 happen to have a USB serial (+console) cable > adapter we could borrow urgently? > > One of our after-hours guys has been caught on the hop without a USB > serial console cable (RJ45 termination) out at NextDC P1 -- Was wondering > if anyone had one on-site we could borrow for a short period of time? > > Have checked the vending machine and whilst there's the great stocks of > ethernet and fibre, there unfortunately isn't any USB to serial adapters. > > > -JT > > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mitchkelly24 at gmail.com Fri Nov 24 22:35:55 2023 From: mitchkelly24 at gmail.com (Mitch Kelly) Date: Fri, 24 Nov 2023 19:35:55 +0800 Subject: [AusNOG] Fwd: Would anyone in NextDC P1 happen to have a USB serial (+console) cable adapter we could borrow? In-Reply-To: References: Message-ID: Might be one in the vending machine. Else il around the corner in Vic Park. On Fri, 24 Nov 2023, 7:33?pm Jennifer Sims, wrote: > Did you check with NextDC staff on site? They may be able to loan one out? > > On Fri, Nov 24, 2023 at 10:31?PM Justin Twiss > wrote: > >> >> >> Hey guys, >> >> Would anyone in NextDC P1 happen to have a USB serial (+console) cable >> adapter we could borrow urgently? >> >> One of our after-hours guys has been caught on the hop without a USB >> serial console cable (RJ45 termination) out at NextDC P1 -- Was wondering >> if anyone had one on-site we could borrow for a short period of time? >> >> Have checked the vending machine and whilst there's the great stocks of >> ethernet and fibre, there unfortunately isn't any USB to serial adapters. >> >> >> -JT >> >> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog >> > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew at levart.com.au Mon Nov 27 16:38:09 2023 From: andrew at levart.com.au (Andrew Simmonds) Date: Mon, 27 Nov 2023 13:38:09 +0800 Subject: [AusNOG] OOB over LTE in Data Centres Message-ID: <2c7df850-1f56-49ed-a82b-307eb5518aa9@levart.com.au> Hi All, We are reviewing OOB access at our DC locations. Do you deploy LTE/5G in the data centre racks as a last-resort? (i.e. via OpenGear's LTE models). If so, had success with a in-rack or an external LTE antenna? Whilst waiting for permission from the DC the support rep. has mentioned that this is not a common request and that other tenants may just utilise independent OOB cross-connects. It would be great to hear your thoughts. - Andrew -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4228 bytes Desc: S/MIME Cryptographic Signature URL: From Nathan.Brookfield at iperium.com.au Mon Nov 27 16:47:30 2023 From: Nathan.Brookfield at iperium.com.au (Nathan Brookfield) Date: Mon, 27 Nov 2023 05:47:30 +0000 Subject: [AusNOG] OOB over LTE in Data Centres In-Reply-To: <2c7df850-1f56-49ed-a82b-307eb5518aa9@levart.com.au> References: <2c7df850-1f56-49ed-a82b-307eb5518aa9@levart.com.au> Message-ID: It's likely an unusual request because nobody asks permission, they just put there antenna on the top of the rack or yep we actually try and use partner networks for our OOB in most locations. -----Original Message----- From: AusNOG On Behalf Of Andrew Simmonds Sent: Monday, November 27, 2023 4:38 PM To: ausnog at lists.ausnog.net Subject: [AusNOG] OOB over LTE in Data Centres Hi All, We are reviewing OOB access at our DC locations. Do you deploy LTE/5G in the data centre racks as a last-resort? (i.e. via OpenGear's LTE models). If so, had success with a in-rack or an external LTE antenna? Whilst waiting for permission from the DC the support rep. has mentioned that this is not a common request and that other tenants may just utilise independent OOB cross-connects. It would be great to hear your thoughts. - Andrew From jaden.roberts at serversaustralia.com.au Mon Nov 27 16:56:04 2023 From: jaden.roberts at serversaustralia.com.au (Jaden Roberts) Date: Mon, 27 Nov 2023 15:56:04 +1000 Subject: [AusNOG] OOB over LTE in Data Centres In-Reply-To: <2c7df850-1f56-49ed-a82b-307eb5518aa9@levart.com.au> References: <2c7df850-1f56-49ed-a82b-307eb5518aa9@levart.com.au> Message-ID: We've deployed Opengears in all locations with LTE interfaces on standby for failover. We find the in rack antenna's to be sufficient. The biggest thing is just selecting a provider that has good coverage in the facility. On Mon, 27 Nov 2023, 4:38?pm Andrew Simmonds, wrote: > Hi All, > > We are reviewing OOB access at our DC locations. > > Do you deploy LTE/5G in the data centre racks as a last-resort? (i.e. > via OpenGear's LTE models). If so, had success with a in-rack or an > external LTE antenna? > > Whilst waiting for permission from the DC the support rep. has mentioned > that this is not a common request and that other tenants may just > utilise independent OOB cross-connects. > > It would be great to hear your thoughts. > > - Andrew > > > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jloh at fastmail.com Mon Nov 27 17:43:37 2023 From: jloh at fastmail.com (James Loh) Date: Mon, 27 Nov 2023 17:43:37 +1100 Subject: [AusNOG] OOB over LTE in Data Centres In-Reply-To: References: <2c7df850-1f56-49ed-a82b-307eb5518aa9@levart.com.au> Message-ID: <56fb88d8-d3db-4d53-bff5-f4414972828d@betaapp.fastmail.com> Don't do DC work anymore but when I did we did the same as Jaden. Opengears with LTE interfaces. Sometimes we ran into issues where ICMP would be blocked so monitoring the OOB network was "online" was difficult, otherwise it worked great. Cheers, James On Mon, 27 Nov 2023, at 4:56 PM, Jaden Roberts wrote: > We've deployed Opengears in all locations with LTE interfaces on standby for failover. > > We find the in rack antenna's to be sufficient. The biggest thing is just selecting a provider that has good coverage in the facility. > > On Mon, 27 Nov 2023, 4:38?pm Andrew Simmonds, wrote: >> Hi All, >> >> We are reviewing OOB access at our DC locations. >> >> Do you deploy LTE/5G in the data centre racks as a last-resort? (i.e. >> via OpenGear's LTE models). If so, had success with a in-rack or an >> external LTE antenna? >> >> Whilst waiting for permission from the DC the support rep. has mentioned >> that this is not a common request and that other tenants may just >> utilise independent OOB cross-connects. >> >> It would be great to hear your thoughts. >> >> - Andrew >> >> >> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbabich at datamossa.com Mon Nov 27 17:57:44 2023 From: bbabich at datamossa.com (Ben Babich) Date: Mon, 27 Nov 2023 13:57:44 +0700 Subject: [AusNOG] OOB over LTE in Data Centres In-Reply-To: <56fb88d8-d3db-4d53-bff5-f4414972828d@betaapp.fastmail.com> References: <2c7df850-1f56-49ed-a82b-307eb5518aa9@levart.com.au> <56fb88d8-d3db-4d53-bff5-f4414972828d@betaapp.fastmail.com> Message-ID: Hi Andrew et al, We've found the RUT* range from Teltonika to work pretty well in most locations. It also helps that they're better priced and easier to obtain than their opengear equivalents. https://teltonika-networks.com/products The cost of these + a cellular service for a year is almost equivalent to the cost of just the XC install + one month alone, then of course you need optics, cables and someone to put a service on it. It really gets down to what you're building, what you want it to support and how reliable you need it to be. Regards, Ben Babich w: https://datamossa.com | e: bbabich at datamossa.com p: +61 2 8188 5111 | m: +61 412 501 110 o: Suite 1, Level 31, 31 Market St. Sydney, NSW 2000. On Mon, 27 Nov 2023 at 13:44, James Loh wrote: > Don't do DC work anymore but when I did we did the same as Jaden. > Opengears with LTE interfaces. > > Sometimes we ran into issues where ICMP would be blocked so monitoring the > OOB network was "online" was difficult, otherwise it worked great. > > Cheers, > James > > On Mon, 27 Nov 2023, at 4:56 PM, Jaden Roberts wrote: > > We've deployed Opengears in all locations with LTE interfaces on standby > for failover. > > We find the in rack antenna's to be sufficient. The biggest thing is just > selecting a provider that has good coverage in the facility. > > On Mon, 27 Nov 2023, 4:38?pm Andrew Simmonds, > wrote: > > Hi All, > > We are reviewing OOB access at our DC locations. > > Do you deploy LTE/5G in the data centre racks as a last-resort? (i.e. > via OpenGear's LTE models). If so, had success with a in-rack or an > external LTE antenna? > > Whilst waiting for permission from the DC the support rep. has mentioned > that this is not a common request and that other tenants may just > utilise independent OOB cross-connects. > > It would be great to hear your thoughts. > > - Andrew > > > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > > > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lidder86 at gmail.com Mon Nov 27 19:04:57 2023 From: lidder86 at gmail.com (Michael Baker) Date: Mon, 27 Nov 2023 19:04:57 +1100 Subject: [AusNOG] OOB over LTE in Data Centres In-Reply-To: <2c7df850-1f56-49ed-a82b-307eb5518aa9@levart.com.au> References: <2c7df850-1f56-49ed-a82b-307eb5518aa9@levart.com.au> Message-ID: Back when I was atba certain MSP ;) we had another client put a SRX110 in the DC with a 2mb optus link then ipsec over that to HO and vpn clients which gave us direct oob over the vpn which also included console access... Andrew you should know which MSP ping me directly if you want the technical side of it Been a long long time! On Mon, 27 Nov 2023, 16:38 Andrew Simmonds, wrote: > Hi All, > > We are reviewing OOB access at our DC locations. > > Do you deploy LTE/5G in the data centre racks as a last-resort? (i.e. > via OpenGear's LTE models). If so, had success with a in-rack or an > external LTE antenna? > > Whilst waiting for permission from the DC the support rep. has mentioned > that this is not a common request and that other tenants may just > utilise independent OOB cross-connects. > > It would be great to hear your thoughts. > > - Andrew > > > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stavros at staff.esc.net.au Tue Nov 28 08:58:06 2023 From: stavros at staff.esc.net.au (Stavros Patiniotis) Date: Tue, 28 Nov 2023 08:28:06 +1030 Subject: [AusNOG] OOB over LTE in Data Centres In-Reply-To: <2c7df850-1f56-49ed-a82b-307eb5518aa9@levart.com.au> References: <2c7df850-1f56-49ed-a82b-307eb5518aa9@levart.com.au> Message-ID: <06ee01da217c$cd578fe0$6806afa0$@staff.esc.net.au> Hi Andrew, We're using devices with external antennas and we have no issues with signal quality from within the rack at all our locations (although a few years back we did have issues within NextDC P1). NextDC don?t like antennas sticking out of the rack either, so that?s not an option. We also use LTE as primary comms for our OOB solution which is similar in design to https://nlnog.net/static/nlnog_live_summer_2020/NLNOG_Live_Job_Snijders_NTT_IP_OOB.pdf. Regards -----Original Message----- From: AusNOG On Behalf Of Andrew Simmonds Sent: Monday, 27 November 2023 4:08 PM To: ausnog at lists.ausnog.net Subject: [AusNOG] OOB over LTE in Data Centres Hi All, We are reviewing OOB access at our DC locations. Do you deploy LTE/5G in the data centre racks as a last-resort? (i.e. via OpenGear's LTE models). If so, had success with a in-rack or an external LTE antenna? Whilst waiting for permission from the DC the support rep. has mentioned that this is not a common request and that other tenants may just utilise independent OOB cross-connects. It would be great to hear your thoughts. - Andrew From kauer at biplane.com.au Tue Nov 28 12:19:06 2023 From: kauer at biplane.com.au (Karl Auer) Date: Tue, 28 Nov 2023 12:19:06 +1100 Subject: [AusNOG] A very small ending Message-ID: Sorry if this is not appropriate for AusNOG, but if not here then where? I just advised APNIC that I would not be renewing 203.26.128.0/24 after holding it for (I think) twenty-seven years. The fees were OK for what amounted to sentimental reasons (since I haven't routed it for a long while) but not after the recent 225% hike, so... The earliest routing I can find for it was by Spirit Networks (hi Richard!) over a permanent dialup line in June 1996. A reverse delegation for it was actioned in July 1996, apparently by Geoff Huston's own fair hand. So I probably got the original allocation, probably from Geoff, sometime in early 1996. There ya go. Regards, K. -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Karl Auer (kauer at biplane.com.au) http://www.biplane.com.au/kauer