From hudrob at gmail.com Fri Feb 7 14:29:52 2025 From: hudrob at gmail.com (Robert Hudson) Date: Fri, 7 Feb 2025 14:29:52 +1100 Subject: [AusNOG] AWS CloudFront Issues Message-ID: Hi all, Is anyone else seeing AWS CloudFront "fronted" domains being marked as malicious or hosting phishing? We have one domain being marked as such right now after four new IP addresses which were previously hosting malware and phishing attempts were apparently added by AWS to a pool used by CloudFront. It's causing quite the drama for us, was just wondering if it's a bit more widespread... -------------- next part -------------- An HTML attachment was scrubbed... URL: From lloyd.wood at yahoo.co.uk Fri Feb 7 16:17:09 2025 From: lloyd.wood at yahoo.co.uk (lloyd.wood at yahoo.co.uk) Date: Fri, 7 Feb 2025 05:17:09 +0000 (UTC) Subject: [AusNOG] nbn vDSL daily-drop weirdness References: <947170683.16170073.1738905429996.ref@mail.yahoo.com> Message-ID: <947170683.16170073.1738905429996@mail.yahoo.com> Longtime lurker here. This isn't the usual ISP crossconnectivity problem, but I think it's weird or interesting enough to be worth noting to Ausnog, since it's NBN, which we are all supposed to use now. Hey, I'd even ask Whirlpool if I thought it would help. Because, I'm that desperate. I have a residential Fibre-to-the-Node vDSL NBN/RSP link. Several times a week (4+), the connection will drop once, and then selfrestore in under ten minutes. But this always happens at around 3pm Sydney time. Today, 3:08pm. The wiring is good, and has been for years? -- if it wasn't, it would drop at other times or fade, surely. NBN and the RSP have both sent out technicians, had the modem replaced with a different model... it still does that daily drop around that time in a one-hour window. My best guess is some kind of weird not-quite-24-hours modem DHCP lease issue --- but nbn procedures are simply to send a tech out to troubleshoot the last mile. If it's not fixable in the last mile, it's not fixable. So, I've had a number of cases closed on me, I open new cases with the RSP assurance team to log further drops and the time when they happen, I've spoken to case managers and explained my experience and view of things and gotten them to nod along... but everyone is boxed in by their procedures, and I can't get someone to give an engineering view of this and think outside the box. When you have weird stuff with your nbn connection, how do escalate to get it resolved? Offline advice appreciated. thanks. Lloyd Wood lloyd.wood at yahoo.co.uk From ltd at interlink.com.au Fri Feb 7 17:37:50 2025 From: ltd at interlink.com.au (Lincoln Dale) Date: Thu, 6 Feb 2025 22:37:50 -0800 Subject: [AusNOG] AWS CloudFront Issues In-Reply-To: References: Message-ID: I assume you raised this with support or abuse? https://repost.aws/knowledge-center/report-aws-abuse If thats not getting you anywhere send it to me off list, but due to being in transit i cant do anything about it immediately. On Thu, Feb 6, 2025 at 7:30?PM Robert Hudson wrote: > Hi all, > > Is anyone else seeing AWS CloudFront "fronted" domains being marked as > malicious or hosting phishing? > > We have one domain being marked as such right now after four new IP > addresses which were previously hosting malware and phishing attempts were > apparently added by AWS to a pool used by CloudFront. > > It's causing quite the drama for us, was just wondering if it's a bit more > widespread... > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tony at wicks.co.nz Fri Feb 7 22:40:49 2025 From: tony at wicks.co.nz (Tony Wicks) Date: Sat, 8 Feb 2025 00:40:49 +1300 Subject: [AusNOG] nbn vDSL daily-drop weirdness In-Reply-To: <947170683.16170073.1738905429996@mail.yahoo.com> References: <947170683.16170073.1738905429996.ref@mail.yahoo.com>, <947170683.16170073.1738905429996@mail.yahoo.com> Message-ID: <9973BCB6-C3D3-4647-8208-73CEE719B347@hxcore.ol> An HTML attachment was scrubbed... URL: From munniche at amazon.com Sat Feb 8 07:24:21 2025 From: munniche at amazon.com (Muennich, Etienne) Date: Fri, 7 Feb 2025 20:24:21 +0000 Subject: [AusNOG] AWS CloudFront Issues In-Reply-To: References:

Message-ID: <86F34E91-1843-4866-913E-2A7F01F451A7@amazon.com> Please reach out to me off list too. -- Etienne M?nnich Sr. Edge Specialist Solutions Architect | AWS My calendar availability is here: https://prelude.amazon.com/s/gieADvSwB/fw1QK/-/meeting-with-etienne Please do not feel obligated to reply outside of your normal work schedule. From: AusNOG on behalf of Lincoln Dale Date: Friday 7 February 2025 at 5:38?pm To: Robert Hudson Cc: "AusNOG at lists.ausnog.net" Subject: RE: [EXTERNAL] [AusNOG] AWS CloudFront Issues CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. I assume you raised this with support or abuse? https://repost.aws/knowledge-center/report-aws-abuse If thats not getting you anywhere send it to me off list, but due to being in transit i cant do anything about it immediately. On Thu, Feb 6, 2025 at 7:30?PM Robert Hudson > wrote: Hi all, Is anyone else seeing AWS CloudFront "fronted" domains being marked as malicious or hosting phishing? We have one domain being marked as such right now after four new IP addresses which were previously hosting malware and phishing attempts were apparently added by AWS to a pool used by CloudFront. It's causing quite the drama for us, was just wondering if it's a bit more widespread... _______________________________________________ AusNOG mailing list AusNOG at lists.ausnog.net https://lists.ausnog.net/mailman/listinfo/ausnog -------------- next part -------------- An HTML attachment was scrubbed... URL: From hudrob at gmail.com Sat Feb 8 08:48:02 2025 From: hudrob at gmail.com (Robert Hudson) Date: Sat, 8 Feb 2025 08:48:02 +1100 Subject: [AusNOG] AWS CloudFront Issues In-Reply-To: References: Message-ID: As a follow-up. Yes, we raised a ticket with AWS for this. The compounding issue was that the IPs were then associated with a number of domains/sub-domains, some of which are not only presented via CloudFront, and it took some time to get agreement on this point. The IPs were removed, and security services are slowly backing down (we started with 7 services as tracked by VirusTotal marking us as malicious, it crept up to 12, its now down to 11). Hopefully we're on the path to redemption. But it's a slow journey. I suspect the longer term solution to prevent this occurring again is to move to static IP assignments where we use CloudFront - not exactly cheap, but cheaper than what's happened here. On Fri, 7 Feb 2025, 2:29?pm Robert Hudson, wrote: > Hi all, > > Is anyone else seeing AWS CloudFront "fronted" domains being marked as > malicious or hosting phishing? > > We have one domain being marked as such right now after four new IP > addresses which were previously hosting malware and phishing attempts were > apparently added by AWS to a pool used by CloudFront. > > It's causing quite the drama for us, was just wondering if it's a bit more > widespread... > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jenn at jenn.id.au Sat Feb 8 10:04:40 2025 From: jenn at jenn.id.au (Jennifer Sims) Date: Sat, 8 Feb 2025 10:04:40 +1100 Subject: [AusNOG] AWS CloudFront Issues In-Reply-To: References: Message-ID: <6FB7D9B2-A9C8-4E71-ABE3-FDD0641344E0@jenn.id.au> An HTML attachment was scrubbed... URL: From hudrob at gmail.com Sat Feb 8 12:56:42 2025 From: hudrob at gmail.com (Robert Hudson) Date: Sat, 8 Feb 2025 12:56:42 +1100 Subject: [AusNOG] AWS CloudFront Issues In-Reply-To: <6FB7D9B2-A9C8-4E71-ABE3-FDD0641344E0@jenn.id.au> References: <6FB7D9B2-A9C8-4E71-ABE3-FDD0641344E0@jenn.id.au> Message-ID: Thanks for the heads-up Jennifer. This is the primary reason I raised the issue with the AusNOG community - to see if we're alone in seeing this, and to get information on this out there for discussion (and to hopefully help some others who were seeing similar things and a bit stuck). The splash damage from this is horrendous - we've had legitimate domains (and sub-domains) that offer legitimate services to corporate customers now flagged as phishing because once the eye of sauron saw us, it took a good hard look at everything we do, and a bunch of legitmate sites are now being flagged as "potentially" phishing after a single report (when some of these sites have run for years now). We'll have to change how we do a few things - but the pain the simple deployment of a few IPs with a bad reputation has caused will ripple through our business for months now. On Sat, 8 Feb 2025 at 10:05, Jennifer Sims wrote: > As a side note, I've had 7 emails from AWS SES hosted domains trying to > phish for information. Looks like there has been a spate of insecure > systems again on the web being used by bad actors. It wouldn't shock me > given the bucket issues also reported on as well that some dodgy phishing > sites are being hidden behind cloud front. > > As I found a heap behind Akamai. > > Sent from my iPhone > > On 8 Feb 2025, at 08:48, Robert Hudson wrote: > > ? > As a follow-up. > > Yes, we raised a ticket with AWS for this. > > The compounding issue was that the IPs were then associated with a number > of domains/sub-domains, some of which are not only presented via > CloudFront, and it took some time to get agreement on this point. > > The IPs were removed, and security services are slowly backing down (we > started with 7 services as tracked by VirusTotal marking us as malicious, > it crept up to 12, its now down to 11). > > Hopefully we're on the path to redemption. But it's a slow journey. > > I suspect the longer term solution to prevent this occurring again is to > move to static IP assignments where we use CloudFront - not exactly cheap, > but cheaper than what's happened here. > > On Fri, 7 Feb 2025, 2:29?pm Robert Hudson, wrote: > >> Hi all, >> >> Is anyone else seeing AWS CloudFront "fronted" domains being marked as >> malicious or hosting phishing? >> >> We have one domain being marked as such right now after four new IP >> addresses which were previously hosting malware and phishing attempts were >> apparently added by AWS to a pool used by CloudFront. >> >> It's causing quite the drama for us, was just wondering if it's a bit >> more widespread... >> > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From diosbejgli at gmail.com Sat Feb 8 13:08:58 2025 From: diosbejgli at gmail.com (Andras Toth) Date: Sat, 8 Feb 2025 13:08:58 +1100 Subject: [AusNOG] AWS CloudFront Issues Message-ID: <3D84FAA9-098D-460C-B809-5CA8122ABBCB@gmail.com> An HTML attachment was scrubbed... URL: From spoofer-info at caida.org Sun Feb 9 05:00:29 2025 From: spoofer-info at caida.org (CAIDA Spoofer Project) Date: Sat, 8 Feb 2025 10:00:29 -0800 Subject: [AusNOG] Spoofer Report for AusNOG for Jan 2025 Message-ID: <1739037629.166137.26093.nullmailer@caida.org> In response to feedback from operational security communities, CAIDA's source address validation measurement project (https://spoofer.caida.org) is automatically generating monthly reports of ASes originating prefixes in BGP for systems from which we received packets with a spoofed source address. We are publishing these reports to network and security operations lists in order to ensure this information reaches operational contacts in these ASes. This report summarises tests conducted within aus. Inferred improvements during Jan 2025: ASN Name Fixed-By 152107 2025-01-31 Further information for the inferred remediation is available at: https://spoofer.caida.org/remedy.php Source Address Validation issues inferred during Jan 2025: ASN Name First-Spoofed Last-Spoofed 7545 TPG-INTERNET-AP 2016-11-11 2025-01-06 152107 2024-02-25 2025-01-06 9336 WWWIRES-AU 2025-01-04 2025-01-04 150369 2025-01-30 2025-01-30 Further information for these tests where we received spoofed packets is available at: https://spoofer.caida.org/recent_tests.php?country_include=aus&no_block=1 Please send any feedback or suggestions to spoofer-info at caida.org From hudrob at gmail.com Sun Feb 9 17:23:56 2025 From: hudrob at gmail.com (Robert Hudson) Date: Sun, 9 Feb 2025 17:23:56 +1100 Subject: [AusNOG] AWS CloudFront Issues In-Reply-To: <3D84FAA9-098D-460C-B809-5CA8122ABBCB@gmail.com> References: <3D84FAA9-098D-460C-B809-5CA8122ABBCB@gmail.com> Message-ID: Agree entirely. It's gotten worse (sadly) rather than better - sibling domains (including one that the DNS is public, but only resolves to RFC1918 IPs) that didn't share the IPs in question are now being reported as hosting malicious or phishing content. On Sat, 8 Feb 2025 at 13:09, Andras Toth wrote: > ?This is why IP based reputation and filtering just doesn't work in > today's world of public clouds with shared tenancy. This problem isn't > unique to AWS nor CloudFront. > > Andras > > On 8 Feb 2025, at 12:57, Robert Hudson wrote: > > ? > Thanks for the heads-up Jennifer. This is the primary reason I raised the > issue with the AusNOG community - to see if we're alone in seeing this, and > to get information on this out there for discussion (and to hopefully help > some others who were seeing similar things and a bit stuck). > > The splash damage from this is horrendous - we've had legitimate domains > (and sub-domains) that offer legitimate services to corporate customers now > flagged as phishing because once the eye of sauron saw us, it took a good > hard look at everything we do, and a bunch of legitmate sites are now being > flagged as "potentially" phishing after a single report (when some of these > sites have run for years now). > > We'll have to change how we do a few things - but the pain the simple > deployment of a few IPs with a bad reputation has caused will ripple > through our business for months now. > > On Sat, 8 Feb 2025 at 10:05, Jennifer Sims wrote: > >> As a side note, I've had 7 emails from AWS SES hosted domains trying to >> phish for information. Looks like there has been a spate of insecure >> systems again on the web being used by bad actors. It wouldn't shock me >> given the bucket issues also reported on as well that some dodgy phishing >> sites are being hidden behind cloud front. >> >> As I found a heap behind Akamai. >> >> Sent from my iPhone >> >> On 8 Feb 2025, at 08:48, Robert Hudson wrote: >> >> ? >> As a follow-up. >> >> Yes, we raised a ticket with AWS for this. >> >> The compounding issue was that the IPs were then associated with a number >> of domains/sub-domains, some of which are not only presented via >> CloudFront, and it took some time to get agreement on this point. >> >> The IPs were removed, and security services are slowly backing down (we >> started with 7 services as tracked by VirusTotal marking us as malicious, >> it crept up to 12, its now down to 11). >> >> Hopefully we're on the path to redemption. But it's a slow journey. >> >> I suspect the longer term solution to prevent this occurring again is to >> move to static IP assignments where we use CloudFront - not exactly cheap, >> but cheaper than what's happened here. >> >> On Fri, 7 Feb 2025, 2:29?pm Robert Hudson, wrote: >> >>> Hi all, >>> >>> Is anyone else seeing AWS CloudFront "fronted" domains being marked as >>> malicious or hosting phishing? >>> >>> We have one domain being marked as such right now after four new IP >>> addresses which were previously hosting malware and phishing attempts were >>> apparently added by AWS to a pool used by CloudFront. >>> >>> It's causing quite the drama for us, was just wondering if it's a bit >>> more widespread... >>> >> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog >> >> _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ausnog at studio442.com.au Sun Feb 9 17:55:53 2025 From: ausnog at studio442.com.au (Julien Goodwin) Date: Sun, 9 Feb 2025 17:55:53 +1100 Subject: [AusNOG] nbn vDSL daily-drop weirdness In-Reply-To: <947170683.16170073.1738905429996@mail.yahoo.com> References: <947170683.16170073.1738905429996.ref@mail.yahoo.com> <947170683.16170073.1738905429996@mail.yahoo.com> Message-ID: On 7/2/25 4:17 pm, lloyd.wood at yahoo.co.uk wrote: > Longtime lurker here. This isn't the usual ISP crossconnectivity problem, but I think it's weird or interesting enough to be worth noting to Ausnog, since it's NBN, which we are all supposed to use now. Hey, I'd even ask Whirlpool if I thought it would help. Because, I'm that desperate. > > I have a residential Fibre-to-the-Node vDSL NBN/RSP link. > > Several times a week (4+), the connection will drop once, and then selfrestore in under ten minutes. But this always happens at around 3pm Sydney time. Today, 3:08pm. > > The wiring is good, and has been for years? -- if it wasn't, it would drop at other times or fade, surely. NBN and the RSP have both sent out technicians, had the modem replaced with a different model... it still does that daily drop around that time in a one-hour window. If it was FTTC I'd guess what used to happen here where the power sharing seems to not work great and always resets whenever my neighbour bounces their NTD (took _ages_ to figure out this was happening). 3PM is an odd time though, if it shifts with the seasons or DST I'd be very suspicious of the cabinet being in direct sun and just barely overheating. You'd hope NBN would have looked to see if it really is just you or everyone on that DSLAM though. From mitchkelly24 at gmail.com Sun Feb 9 21:19:54 2025 From: mitchkelly24 at gmail.com (Mitch Kelly) Date: Sun, 9 Feb 2025 18:19:54 +0800 Subject: [AusNOG] AWS CloudFront Issues In-Reply-To: References: <3D84FAA9-098D-460C-B809-5CA8122ABBCB@gmail.com> Message-ID: Sadly also having issues with CloudFront. Issues started to show their head Tuesday last week and have been getting worse. With many sites not working at all. On Sun, 9 Feb 2025, 2:24?pm Robert Hudson, wrote: > Agree entirely. > > It's gotten worse (sadly) rather than better - sibling domains (including > one that the DNS is public, but only resolves to RFC1918 IPs) that didn't > share the IPs in question are now being reported as hosting malicious or > phishing content. > > On Sat, 8 Feb 2025 at 13:09, Andras Toth wrote: > >> ?This is why IP based reputation and filtering just doesn't work in >> today's world of public clouds with shared tenancy. This problem isn't >> unique to AWS nor CloudFront. >> >> Andras >> >> On 8 Feb 2025, at 12:57, Robert Hudson wrote: >> >> ? >> Thanks for the heads-up Jennifer. This is the primary reason I raised >> the issue with the AusNOG community - to see if we're alone in seeing this, >> and to get information on this out there for discussion (and to hopefully >> help some others who were seeing similar things and a bit stuck). >> >> The splash damage from this is horrendous - we've had legitimate domains >> (and sub-domains) that offer legitimate services to corporate customers now >> flagged as phishing because once the eye of sauron saw us, it took a good >> hard look at everything we do, and a bunch of legitmate sites are now being >> flagged as "potentially" phishing after a single report (when some of these >> sites have run for years now). >> >> We'll have to change how we do a few things - but the pain the simple >> deployment of a few IPs with a bad reputation has caused will ripple >> through our business for months now. >> >> On Sat, 8 Feb 2025 at 10:05, Jennifer Sims wrote: >> >>> As a side note, I've had 7 emails from AWS SES hosted domains trying to >>> phish for information. Looks like there has been a spate of insecure >>> systems again on the web being used by bad actors. It wouldn't shock me >>> given the bucket issues also reported on as well that some dodgy phishing >>> sites are being hidden behind cloud front. >>> >>> As I found a heap behind Akamai. >>> >>> Sent from my iPhone >>> >>> On 8 Feb 2025, at 08:48, Robert Hudson wrote: >>> >>> ? >>> As a follow-up. >>> >>> Yes, we raised a ticket with AWS for this. >>> >>> The compounding issue was that the IPs were then associated with a >>> number of domains/sub-domains, some of which are not only presented via >>> CloudFront, and it took some time to get agreement on this point. >>> >>> The IPs were removed, and security services are slowly backing down (we >>> started with 7 services as tracked by VirusTotal marking us as malicious, >>> it crept up to 12, its now down to 11). >>> >>> Hopefully we're on the path to redemption. But it's a slow journey. >>> >>> I suspect the longer term solution to prevent this occurring again is to >>> move to static IP assignments where we use CloudFront - not exactly cheap, >>> but cheaper than what's happened here. >>> >>> On Fri, 7 Feb 2025, 2:29?pm Robert Hudson, wrote: >>> >>>> Hi all, >>>> >>>> Is anyone else seeing AWS CloudFront "fronted" domains being marked as >>>> malicious or hosting phishing? >>>> >>>> We have one domain being marked as such right now after four new IP >>>> addresses which were previously hosting malware and phishing attempts were >>>> apparently added by AWS to a pool used by CloudFront. >>>> >>>> It's causing quite the drama for us, was just wondering if it's a bit >>>> more widespread... >>>> >>> _______________________________________________ >>> AusNOG mailing list >>> AusNOG at lists.ausnog.net >>> https://lists.ausnog.net/mailman/listinfo/ausnog >>> >>> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog >> >> _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jenn at jenn.id.au Sun Feb 9 23:43:22 2025 From: jenn at jenn.id.au (Jennifer Sims) Date: Sun, 9 Feb 2025 23:43:22 +1100 Subject: [AusNOG] AWS CloudFront Issues In-Reply-To: References: <3D84FAA9-098D-460C-B809-5CA8122ABBCB@gmail.com>

Message-ID: Sadly these days, bad actors will target any machine they can get into, it's not unique to AWS, Cloudfront, Akamai etc etc etc. On Sun, Feb 9, 2025 at 9:20?PM Mitch Kelly wrote: > Sadly also having issues with CloudFront. Issues started to show their > head Tuesday last week and have been getting worse. With many sites not > working at all. > > On Sun, 9 Feb 2025, 2:24?pm Robert Hudson, wrote: > >> Agree entirely. >> >> It's gotten worse (sadly) rather than better - sibling domains (including >> one that the DNS is public, but only resolves to RFC1918 IPs) that didn't >> share the IPs in question are now being reported as hosting malicious or >> phishing content. >> >> On Sat, 8 Feb 2025 at 13:09, Andras Toth wrote: >> >>> ?This is why IP based reputation and filtering just doesn't work in >>> today's world of public clouds with shared tenancy. This problem isn't >>> unique to AWS nor CloudFront. >>> >>> Andras >>> >>> On 8 Feb 2025, at 12:57, Robert Hudson wrote: >>> >>> ? >>> Thanks for the heads-up Jennifer. This is the primary reason I raised >>> the issue with the AusNOG community - to see if we're alone in seeing this, >>> and to get information on this out there for discussion (and to hopefully >>> help some others who were seeing similar things and a bit stuck). >>> >>> The splash damage from this is horrendous - we've had legitimate domains >>> (and sub-domains) that offer legitimate services to corporate customers now >>> flagged as phishing because once the eye of sauron saw us, it took a good >>> hard look at everything we do, and a bunch of legitmate sites are now being >>> flagged as "potentially" phishing after a single report (when some of these >>> sites have run for years now). >>> >>> We'll have to change how we do a few things - but the pain the simple >>> deployment of a few IPs with a bad reputation has caused will ripple >>> through our business for months now. >>> >>> On Sat, 8 Feb 2025 at 10:05, Jennifer Sims wrote: >>> >>>> As a side note, I've had 7 emails from AWS SES hosted domains trying to >>>> phish for information. Looks like there has been a spate of insecure >>>> systems again on the web being used by bad actors. It wouldn't shock me >>>> given the bucket issues also reported on as well that some dodgy phishing >>>> sites are being hidden behind cloud front. >>>> >>>> As I found a heap behind Akamai. >>>> >>>> Sent from my iPhone >>>> >>>> On 8 Feb 2025, at 08:48, Robert Hudson wrote: >>>> >>>> ? >>>> As a follow-up. >>>> >>>> Yes, we raised a ticket with AWS for this. >>>> >>>> The compounding issue was that the IPs were then associated with a >>>> number of domains/sub-domains, some of which are not only presented via >>>> CloudFront, and it took some time to get agreement on this point. >>>> >>>> The IPs were removed, and security services are slowly backing down (we >>>> started with 7 services as tracked by VirusTotal marking us as malicious, >>>> it crept up to 12, its now down to 11). >>>> >>>> Hopefully we're on the path to redemption. But it's a slow journey. >>>> >>>> I suspect the longer term solution to prevent this occurring again is >>>> to move to static IP assignments where we use CloudFront - not exactly >>>> cheap, but cheaper than what's happened here. >>>> >>>> On Fri, 7 Feb 2025, 2:29?pm Robert Hudson, wrote: >>>> >>>>> Hi all, >>>>> >>>>> Is anyone else seeing AWS CloudFront "fronted" domains being marked as >>>>> malicious or hosting phishing? >>>>> >>>>> We have one domain being marked as such right now after four new IP >>>>> addresses which were previously hosting malware and phishing attempts were >>>>> apparently added by AWS to a pool used by CloudFront. >>>>> >>>>> It's causing quite the drama for us, was just wondering if it's a bit >>>>> more widespread... >>>>> >>>> _______________________________________________ >>>> AusNOG mailing list >>>> AusNOG at lists.ausnog.net >>>> https://lists.ausnog.net/mailman/listinfo/ausnog >>>> >>>> _______________________________________________ >>> AusNOG mailing list >>> AusNOG at lists.ausnog.net >>> https://lists.ausnog.net/mailman/listinfo/ausnog >>> >>> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog >> > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hudrob at gmail.com Mon Feb 10 07:42:28 2025 From: hudrob at gmail.com (Robert Hudson) Date: Mon, 10 Feb 2025 07:42:28 +1100 Subject: [AusNOG] AWS CloudFront Issues In-Reply-To: References: <3D84FAA9-098D-460C-B809-5CA8122ABBCB@gmail.com>

Message-ID: Absolutely. Doesn't make it easier though when certain providers spread the problem around by re-using compromised IPs. On Sun, 9 Feb 2025 at 23:43, Jennifer Sims wrote: > Sadly these days, bad actors will target any machine they can get into, > it's not unique to AWS, Cloudfront, Akamai etc etc etc. > > On Sun, Feb 9, 2025 at 9:20?PM Mitch Kelly wrote: > >> Sadly also having issues with CloudFront. Issues started to show their >> head Tuesday last week and have been getting worse. With many sites not >> working at all. >> >> On Sun, 9 Feb 2025, 2:24?pm Robert Hudson, wrote: >> >>> Agree entirely. >>> >>> It's gotten worse (sadly) rather than better - sibling domains >>> (including one that the DNS is public, but only resolves to RFC1918 IPs) >>> that didn't share the IPs in question are now being reported as hosting >>> malicious or phishing content. >>> >>> On Sat, 8 Feb 2025 at 13:09, Andras Toth wrote: >>> >>>> ?This is why IP based reputation and filtering just doesn't work in >>>> today's world of public clouds with shared tenancy. This problem isn't >>>> unique to AWS nor CloudFront. >>>> >>>> Andras >>>> >>>> On 8 Feb 2025, at 12:57, Robert Hudson wrote: >>>> >>>> ? >>>> Thanks for the heads-up Jennifer. This is the primary reason I raised >>>> the issue with the AusNOG community - to see if we're alone in seeing this, >>>> and to get information on this out there for discussion (and to hopefully >>>> help some others who were seeing similar things and a bit stuck). >>>> >>>> The splash damage from this is horrendous - we've had legitimate >>>> domains (and sub-domains) that offer legitimate services to corporate >>>> customers now flagged as phishing because once the eye of sauron saw us, it >>>> took a good hard look at everything we do, and a bunch of legitmate sites >>>> are now being flagged as "potentially" phishing after a single report (when >>>> some of these sites have run for years now). >>>> >>>> We'll have to change how we do a few things - but the pain the simple >>>> deployment of a few IPs with a bad reputation has caused will ripple >>>> through our business for months now. >>>> >>>> On Sat, 8 Feb 2025 at 10:05, Jennifer Sims wrote: >>>> >>>>> As a side note, I've had 7 emails from AWS SES hosted domains trying >>>>> to phish for information. Looks like there has been a spate of insecure >>>>> systems again on the web being used by bad actors. It wouldn't shock me >>>>> given the bucket issues also reported on as well that some dodgy phishing >>>>> sites are being hidden behind cloud front. >>>>> >>>>> As I found a heap behind Akamai. >>>>> >>>>> Sent from my iPhone >>>>> >>>>> On 8 Feb 2025, at 08:48, Robert Hudson wrote: >>>>> >>>>> ? >>>>> As a follow-up. >>>>> >>>>> Yes, we raised a ticket with AWS for this. >>>>> >>>>> The compounding issue was that the IPs were then associated with a >>>>> number of domains/sub-domains, some of which are not only presented via >>>>> CloudFront, and it took some time to get agreement on this point. >>>>> >>>>> The IPs were removed, and security services are slowly backing down >>>>> (we started with 7 services as tracked by VirusTotal marking us as >>>>> malicious, it crept up to 12, its now down to 11). >>>>> >>>>> Hopefully we're on the path to redemption. But it's a slow journey. >>>>> >>>>> I suspect the longer term solution to prevent this occurring again is >>>>> to move to static IP assignments where we use CloudFront - not exactly >>>>> cheap, but cheaper than what's happened here. >>>>> >>>>> On Fri, 7 Feb 2025, 2:29?pm Robert Hudson, wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> Is anyone else seeing AWS CloudFront "fronted" domains being marked >>>>>> as malicious or hosting phishing? >>>>>> >>>>>> We have one domain being marked as such right now after four new IP >>>>>> addresses which were previously hosting malware and phishing attempts were >>>>>> apparently added by AWS to a pool used by CloudFront. >>>>>> >>>>>> It's causing quite the drama for us, was just wondering if it's a bit >>>>>> more widespread... >>>>>> >>>>> _______________________________________________ >>>>> AusNOG mailing list >>>>> AusNOG at lists.ausnog.net >>>>> https://lists.ausnog.net/mailman/listinfo/ausnog >>>>> >>>>> _______________________________________________ >>>> AusNOG mailing list >>>> AusNOG at lists.ausnog.net >>>> https://lists.ausnog.net/mailman/listinfo/ausnog >>>> >>>> _______________________________________________ >>> AusNOG mailing list >>> AusNOG at lists.ausnog.net >>> https://lists.ausnog.net/mailman/listinfo/ausnog >>> >> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ltd at interlink.com.au Mon Feb 10 08:45:45 2025 From: ltd at interlink.com.au (Lincoln Dale) Date: Mon, 10 Feb 2025 08:45:45 +1100 Subject: [AusNOG] AWS CloudFront Issues In-Reply-To: References: <3D84FAA9-098D-460C-B809-5CA8122ABBCB@gmail.com>

Message-ID: On Mon, Feb 10, 2025 at 7:43?AM Robert Hudson wrote: > Doesn't make it easier though when certain providers spread the problem > around by re-using compromised IPs. > The fault here lies with VirusTotal (likely actually Google, its parent) who are doing outmoded/outdated attaching "trust" to an IP address. Specifically, since RFC 2817 (almost 25 ago) one has not needed to use a "dedicated IP per SSL certificate". And assigning reputation on an IP of what is clearly a CDN is going to have its issues. There is no such thing here as a "compromised IP". The reputation for the IP is "This IP address has been detected as a proxy connection, which could be hurting your IP reputation", which is pretty much the definition of a CDN. The reality is CDNs as a general rule don't dedicate IPs to domains or customers, because there's way more domains than there are IP addresses allocated to serve them. I found the ticket internally, looks resolved, reach out if any issues. On Sun, Feb 9, 2025 at 9:20?PM Mitch Kelly wrote: > Sadly also having issues with CloudFront. Issues started to show their >>> head Tuesday last week and have been getting worse. With many sites not >>> working at all. >>> >> Not aware of any widespread issue. Same offer as to Robert, if you are looking for assistance, send it though to me. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hudrob at gmail.com Mon Feb 10 09:01:54 2025 From: hudrob at gmail.com (Robert Hudson) Date: Mon, 10 Feb 2025 09:01:54 +1100 Subject: [AusNOG] AWS CloudFront Issues In-Reply-To: References: <3D84FAA9-098D-460C-B809-5CA8122ABBCB@gmail.com>

Message-ID: I agree that AWS shouldn't need to worry about the IPs - but as much as it should t be the case in theory, in practice is is the case. This is burning my employer badly. I'm actually not interested in blame (the fact AWS attached IPs with a bad reputation to our services is a fact, they themselves acknowledge it and have removed them) but rather just in a fix. I appreciate your offer to assist, I've reached out. On Mon, 10 Feb 2025, 8:45?am Lincoln Dale, wrote: > > On Mon, Feb 10, 2025 at 7:43?AM Robert Hudson wrote: > >> Doesn't make it easier though when certain providers spread the problem >> around by re-using compromised IPs. >> > > The fault here lies with VirusTotal (likely actually Google, its parent) > who are doing outmoded/outdated attaching "trust" to an IP address. > Specifically, since RFC 2817 (almost 25 ago) one has not needed to use a > "dedicated IP per SSL certificate". And assigning reputation on an IP of > what is clearly a CDN is going to have its issues. > > There is no such thing here as a "compromised IP". The reputation for the > IP is "This IP address has been detected as a proxy connection, which could > be hurting your IP reputation", which is pretty much the definition of a > CDN. The reality is CDNs as a general rule don't dedicate IPs to domains or > customers, because there's way more domains than there are IP addresses > allocated to serve them. > I found the ticket internally, looks resolved, reach out if any issues. > > > On Sun, Feb 9, 2025 at 9:20?PM Mitch Kelly wrote: > >> Sadly also having issues with CloudFront. Issues started to show their >>>> head Tuesday last week and have been getting worse. With many sites not >>>> working at all. >>>> >>> Not aware of any widespread issue. > Same offer as to Robert, if you are looking for assistance, send it though > to me. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Ben.Ricardo at acs.com.au Mon Feb 10 17:50:39 2025 From: Ben.Ricardo at acs.com.au (Ben Ricardo) Date: Mon, 10 Feb 2025 06:50:39 +0000 Subject: [AusNOG] TPG Outage 5:30pm AEDT Message-ID: <4cd6418b9bdd4b64843e9fe95f64711b@acs.com.au> Hi Folks, It would appear to me that TPG have some sort of problem starting approx. 5:25pm Anyone else for an early mark today? Ben Ben Ricardo?| Senior Technician |? M Net&SysAdmin, MCITP-SA, CEHv8, ITIL Australian Computer Solutions Pty Ltd | 2/28 Barralong Rd Erina NSW 2250?| P: 02 4365 2727 or 1300-807-131 | F: 02 4365 2304 | E: ben.ricardo at acs.com.au From Nathan.Brookfield at iperium.com.au Mon Feb 10 17:59:54 2025 From: Nathan.Brookfield at iperium.com.au (Nathan Brookfield) Date: Mon, 10 Feb 2025 06:59:54 +0000 Subject: [AusNOG] TPG Outage 5:30pm AEDT In-Reply-To: <4cd6418b9bdd4b64843e9fe95f64711b@acs.com.au> References: <4cd6418b9bdd4b64843e9fe95f64711b@acs.com.au> Message-ID: Can certainly confirm, apparently there helpdesk, website and large amounts of other customer facing systems are down. From: AusNOG on behalf of Ben Ricardo Date: Monday, 10 February 2025 at 16:51 To: ausnog at lists.ausnog.net Subject: [AusNOG] TPG Outage 5:30pm AEDT Hi Folks, It would appear to me that TPG have some sort of problem starting approx. 5:25pm Anyone else for an early mark today? Ben Ben Ricardo | Senior Technician | M Net&SysAdmin, MCITP-SA, CEHv8, ITIL Australian Computer Solutions Pty Ltd | 2/28 Barralong Rd Erina NSW 2250 | P: 02 4365 2727 or 1300-807-131 | F: 02 4365 2304 | E: ben.ricardo at acs.com.au _______________________________________________ AusNOG mailing list AusNOG at lists.ausnog.net https://lists.ausnog.net/mailman/listinfo/ausnog -------------- next part -------------- An HTML attachment was scrubbed... URL: From Steven.Waite at comtel.com.au Mon Feb 10 18:00:02 2025 From: Steven.Waite at comtel.com.au (Steven Waite) Date: Mon, 10 Feb 2025 07:00:02 +0000 Subject: [AusNOG] TPG Outage 5:30pm AEDT In-Reply-To: <4cd6418b9bdd4b64843e9fe95f64711b@acs.com.au> References: <4cd6418b9bdd4b64843e9fe95f64711b@acs.com.au> Message-ID: <5b3b8eb1cc9e4641b348459cb02fed0d@comtel.com.au> We lost our peer to them around the same time too. Seems to be coming back now Thanks Steve -----Original Message----- From: AusNOG On Behalf Of Ben Ricardo Sent: Monday, 10 February 2025 4:51 PM To: ausnog at lists.ausnog.net Subject: [AusNOG] TPG Outage 5:30pm AEDT Hi Folks, It would appear to me that TPG have some sort of problem starting approx. 5:25pm Anyone else for an early mark today? Ben Ben Ricardo?| Senior Technician |? M Net&SysAdmin, MCITP-SA, CEHv8, ITIL Australian Computer Solutions Pty Ltd | 2/28 Barralong Rd Erina NSW 2250?| P: 02 4365 2727 or 1300-807-131 | F: 02 4365 2304 | E: ben.ricardo at acs.com.au _______________________________________________ AusNOG mailing list AusNOG at lists.ausnog.net https://lists.ausnog.net/mailman/listinfo/ausnog From chris at thesysadmin.au Mon Feb 10 18:06:36 2025 From: chris at thesysadmin.au (Christopher Hawker) Date: Mon, 10 Feb 2025 07:06:36 +0000 Subject: [AusNOG] TPG Outage 5:30pm AEDT In-Reply-To: References: <4cd6418b9bdd4b64843e9fe95f64711b@acs.com.au> Message-ID: Can confirm, have seen F1000 services in Sydney and Melbourne all go down about the same time. On a F1000 service in NSW, I did notice that a v4 BGP session lost about 10k routes, other than that it still appears to be working as expected. Regards, Christopher Hawker ________________________________ From: AusNOG on behalf of Nathan Brookfield Sent: Monday, February 10, 2025 5:59 PM To: Ben Ricardo ; ausnog at lists.ausnog.net Subject: Re: [AusNOG] TPG Outage 5:30pm AEDT Can certainly confirm, apparently there helpdesk, website and large amounts of other customer facing systems are down. From: AusNOG on behalf of Ben Ricardo Date: Monday, 10 February 2025 at 16:51 To: ausnog at lists.ausnog.net Subject: [AusNOG] TPG Outage 5:30pm AEDT Hi Folks, It would appear to me that TPG have some sort of problem starting approx. 5:25pm Anyone else for an early mark today? Ben Ben Ricardo | Senior Technician | M Net&SysAdmin, MCITP-SA, CEHv8, ITIL Australian Computer Solutions Pty Ltd | 2/28 Barralong Rd Erina NSW 2250 | P: 02 4365 2727 or 1300-807-131 | F: 02 4365 2304 | E: ben.ricardo at acs.com.au _______________________________________________ AusNOG mailing list AusNOG at lists.ausnog.net https://lists.ausnog.net/mailman/listinfo/ausnog -------------- next part -------------- An HTML attachment was scrubbed... URL: From claytongee at me.com Mon Feb 10 18:11:39 2025 From: claytongee at me.com (Clayton Gee) Date: Mon, 10 Feb 2025 18:11:39 +1100 Subject: [AusNOG] TPG Outage 5:30pm AEDT In-Reply-To: References: <4cd6418b9bdd4b64843e9fe95f64711b@acs.com.au> Message-ID: <6AE34897-14A4-4E0F-94AB-5D9D020D01FF@me.com> Can confirm here too. Large corporate using TPG and we lost all connections. They?re back online now but performing rather slow. > On 10 Feb 2025, at 6:06?pm, Christopher Hawker wrote: > > Can confirm, have seen F1000 services in Sydney and Melbourne all go down about the same time. On a F1000 service in NSW, I did notice that a v4 BGP session lost about 10k routes, other than that it still appears to be working as expected. > > Regards, > Christopher Hawker > From: AusNOG on behalf of Nathan Brookfield > Sent: Monday, February 10, 2025 5:59 PM > To: Ben Ricardo ; ausnog at lists.ausnog.net > Subject: Re: [AusNOG] TPG Outage 5:30pm AEDT > > Can certainly confirm, apparently there helpdesk, website and large amounts of other customer facing systems are down. > > From: AusNOG on behalf of Ben Ricardo > Date: Monday, 10 February 2025 at 16:51 > To: ausnog at lists.ausnog.net > Subject: [AusNOG] TPG Outage 5:30pm AEDT > > Hi Folks, > It would appear to me that TPG have some sort of problem starting approx. 5:25pm > Anyone else for an early mark today? > > Ben > > Ben Ricardo | Senior Technician | M Net&SysAdmin, MCITP-SA, CEHv8, ITIL > Australian Computer Solutions Pty Ltd | 2/28 Barralong Rd Erina NSW 2250 | > P: 02 4365 2727 or 1300-807-131 | F: 02 4365 2304 | E: ben.ricardo at acs.com.au > > > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog -------------- next part -------------- An HTML attachment was scrubbed... URL: From dazzagibbs at gmail.com Mon Feb 10 18:13:07 2025 From: dazzagibbs at gmail.com (DaZZa) Date: Mon, 10 Feb 2025 18:13:07 +1100 Subject: [AusNOG] TPG Outage 5:30pm AEDT In-Reply-To: <4cd6418b9bdd4b64843e9fe95f64711b@acs.com.au> References: <4cd6418b9bdd4b64843e9fe95f64711b@acs.com.au> Message-ID: Yeah, I've lost my home internet, corporate link via them, their support phone number is reporting as incorrect or disconnected and half their web site doesn't work. Someone is in for a good night. D On Mon, 10 Feb 2025, 5:50?pm Ben Ricardo, wrote: > Hi Folks, > It would appear to me that TPG have some sort of problem starting approx. > 5:25pm > Anyone else for an early mark today? > > Ben > > Ben Ricardo | Senior Technician | M Net&SysAdmin, MCITP-SA, CEHv8, ITIL > Australian Computer Solutions Pty Ltd | 2/28 Barralong Rd Erina NSW 2250 > | > > P: 02 4365 2727 or 1300-807-131 | F: 02 4365 2304 | E: > ben.ricardo at acs.com.au > > > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hudrob at gmail.com Mon Feb 10 18:14:37 2025 From: hudrob at gmail.com (Robert Hudson) Date: Mon, 10 Feb 2025 18:14:37 +1100 Subject: [AusNOG] TPG Outage 5:30pm AEDT In-Reply-To: <6AE34897-14A4-4E0F-94AB-5D9D020D01FF@me.com> References: <4cd6418b9bdd4b64843e9fe95f64711b@acs.com.au> <6AE34897-14A4-4E0F-94AB-5D9D020D01FF@me.com> Message-ID: We suffered an office Internet outage. I know at least one large national customer had an outage in at least one datacentre. Services came back online pretty quickly once we rebooted the router in our office, and we were able to re-connect to the large customer pretty quickly as well. It seem from downdetector.com.au that it was pretty widespread. On Mon, 10 Feb 2025 at 18:12, Clayton Gee wrote: > Can confirm here too. Large corporate using TPG and we lost all > connections. They?re back online now but performing rather slow. > > On 10 Feb 2025, at 6:06?pm, Christopher Hawker > wrote: > > Can confirm, have seen F1000 services in Sydney and Melbourne all go down > about the same time. On a F1000 service in NSW, I did notice that a v4 BGP > session lost about 10k routes, other than that it still appears to be > working as expected. > > Regards, > Christopher Hawker > ------------------------------ > *From:* AusNOG on behalf of Nathan > Brookfield > *Sent:* Monday, February 10, 2025 5:59 PM > *To:* Ben Ricardo ; ausnog at lists.ausnog.net < > ausnog at lists.ausnog.net> > *Subject:* Re: [AusNOG] TPG Outage 5:30pm AEDT > > Can certainly confirm, apparently there helpdesk, website and large > amounts of other customer facing systems are down. > > > > *From: *AusNOG on behalf of Ben Ricardo > > *Date: *Monday, 10 February 2025 at 16:51 > *To: *ausnog at lists.ausnog.net > *Subject: *[AusNOG] TPG Outage 5:30pm AEDT > Hi Folks, > It would appear to me that TPG have some sort of problem starting approx. > 5:25pm > Anyone else for an early mark today? > > Ben > > Ben Ricardo | Senior Technician | M Net&SysAdmin, MCITP-SA, CEHv8, ITIL > Australian Computer Solutions Pty Ltd | 2/28 Barralong Rd Erina NSW 2250 | > > P: 02 4365 2727 or 1300-807-131 | F: 02 4365 2304 | E: > ben.ricardo at acs.com.au > > > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > > > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nathan.Brookfield at iperium.com.au Mon Feb 10 18:17:08 2025 From: Nathan.Brookfield at iperium.com.au (Nathan Brookfield) Date: Mon, 10 Feb 2025 07:17:08 +0000 Subject: [AusNOG] TPG Outage 5:30pm AEDT In-Reply-To: <6AE34897-14A4-4E0F-94AB-5D9D020D01FF@me.com> References: <4cd6418b9bdd4b64843e9fe95f64711b@acs.com.au> <6AE34897-14A4-4E0F-94AB-5D9D020D01FF@me.com> Message-ID: Looks like there primary DNS etc is still down, Frontier and other systems the same. My money is on power related issue in the Glebe, Pymont area being related. From: Clayton Gee Date: Monday, 10 February 2025 at 17:12 To: Christopher Hawker Cc: Nathan Brookfield , Ben Ricardo , ausnog at lists.ausnog.net Subject: Re: [AusNOG] TPG Outage 5:30pm AEDT Can confirm here too. Large corporate using TPG and we lost all connections. They?re back online now but performing rather slow. On 10 Feb 2025, at 6:06?pm, Christopher Hawker wrote: Can confirm, have seen F1000 services in Sydney and Melbourne all go down about the same time. On a F1000 service in NSW, I did notice that a v4 BGP session lost about 10k routes, other than that it still appears to be working as expected. Regards, Christopher Hawker ________________________________ From: AusNOG on behalf of Nathan Brookfield Sent: Monday, February 10, 2025 5:59 PM To: Ben Ricardo ; ausnog at lists.ausnog.net Subject: Re: [AusNOG] TPG Outage 5:30pm AEDT Can certainly confirm, apparently there helpdesk, website and large amounts of other customer facing systems are down. From: AusNOG on behalf of Ben Ricardo Date: Monday, 10 February 2025 at 16:51 To: ausnog at lists.ausnog.net Subject: [AusNOG] TPG Outage 5:30pm AEDT Hi Folks, It would appear to me that TPG have some sort of problem starting approx. 5:25pm Anyone else for an early mark today? Ben Ben Ricardo | Senior Technician | M Net&SysAdmin, MCITP-SA, CEHv8, ITIL Australian Computer Solutions Pty Ltd | 2/28 Barralong Rd Erina NSW 2250 | P: 02 4365 2727 or 1300-807-131 | F: 02 4365 2304 | E: ben.ricardo at acs.com.au _______________________________________________ AusNOG mailing list AusNOG at lists.ausnog.net https://lists.ausnog.net/mailman/listinfo/ausnog _______________________________________________ AusNOG mailing list AusNOG at lists.ausnog.net https://lists.ausnog.net/mailman/listinfo/ausnog -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at thesysadmin.au Mon Feb 10 18:31:20 2025 From: chris at thesysadmin.au (Christopher Hawker) Date: Mon, 10 Feb 2025 07:31:20 +0000 Subject: [AusNOG] TPG Outage 5:30pm AEDT In-Reply-To: References: <4cd6418b9bdd4b64843e9fe95f64711b@acs.com.au> <6AE34897-14A4-4E0F-94AB-5D9D020D01FF@me.com> Message-ID: Ausgrid does report a power outage in Glebe, but surely their national network isn't reliant on a single site, running single-feed power without some form of a UPS sitting between it and their diesel genset... Regards, Christopher Hawker ________________________________ From: Nathan Brookfield Sent: Monday, February 10, 2025 6:17 PM To: Clayton Gee ; Christopher Hawker Cc: Ben Ricardo ; ausnog at lists.ausnog.net Subject: Re: [AusNOG] TPG Outage 5:30pm AEDT Looks like there primary DNS etc is still down, Frontier and other systems the same. My money is on power related issue in the Glebe, Pymont area being related. From: Clayton Gee Date: Monday, 10 February 2025 at 17:12 To: Christopher Hawker Cc: Nathan Brookfield , Ben Ricardo , ausnog at lists.ausnog.net Subject: Re: [AusNOG] TPG Outage 5:30pm AEDT Can confirm here too. Large corporate using TPG and we lost all connections. They?re back online now but performing rather slow. On 10 Feb 2025, at 6:06?pm, Christopher Hawker wrote: Can confirm, have seen F1000 services in Sydney and Melbourne all go down about the same time. On a F1000 service in NSW, I did notice that a v4 BGP session lost about 10k routes, other than that it still appears to be working as expected. Regards, Christopher Hawker ________________________________ From: AusNOG on behalf of Nathan Brookfield Sent: Monday, February 10, 2025 5:59 PM To: Ben Ricardo ; ausnog at lists.ausnog.net Subject: Re: [AusNOG] TPG Outage 5:30pm AEDT Can certainly confirm, apparently there helpdesk, website and large amounts of other customer facing systems are down. From: AusNOG on behalf of Ben Ricardo Date: Monday, 10 February 2025 at 16:51 To: ausnog at lists.ausnog.net Subject: [AusNOG] TPG Outage 5:30pm AEDT Hi Folks, It would appear to me that TPG have some sort of problem starting approx. 5:25pm Anyone else for an early mark today? Ben Ben Ricardo | Senior Technician | M Net&SysAdmin, MCITP-SA, CEHv8, ITIL Australian Computer Solutions Pty Ltd | 2/28 Barralong Rd Erina NSW 2250 | P: 02 4365 2727 or 1300-807-131 | F: 02 4365 2304 | E: ben.ricardo at acs.com.au _______________________________________________ AusNOG mailing list AusNOG at lists.ausnog.net https://lists.ausnog.net/mailman/listinfo/ausnog _______________________________________________ AusNOG mailing list AusNOG at lists.ausnog.net https://lists.ausnog.net/mailman/listinfo/ausnog -------------- next part -------------- An HTML attachment was scrubbed... URL: From Ben.Ricardo at acs.com.au Mon Feb 10 18:43:42 2025 From: Ben.Ricardo at acs.com.au (Ben Ricardo) Date: Mon, 10 Feb 2025 07:43:42 +0000 Subject: [AusNOG] TPG Outage 5:30pm AEDT In-Reply-To: References: <4cd6418b9bdd4b64843e9fe95f64711b@acs.com.au> <6AE34897-14A4-4E0F-94AB-5D9D020D01FF@me.com> , Message-ID: We're seeing services returning to normal now. I don't suppose the power's back on in Glebe -------- Original message -------- From: Christopher Hawker Date: 10/2/25 6:31?pm (GMT+10:00) To: Nathan Brookfield , Clayton Gee Cc: Ben Ricardo , ausnog at lists.ausnog.net Subject: Re: [AusNOG] TPG Outage 5:30pm AEDT Ausgrid does report a power outage in Glebe, but surely their national network isn't reliant on a single site, running single-feed power without some form of a UPS sitting between it and their diesel genset... Regards, Christopher Hawker ________________________________ From: Nathan Brookfield Sent: Monday, February 10, 2025 6:17 PM To: Clayton Gee ; Christopher Hawker Cc: Ben Ricardo ; ausnog at lists.ausnog.net Subject: Re: [AusNOG] TPG Outage 5:30pm AEDT Looks like there primary DNS etc is still down, Frontier and other systems the same. My money is on power related issue in the Glebe, Pymont area being related. From: Clayton Gee Date: Monday, 10 February 2025 at 17:12 To: Christopher Hawker Cc: Nathan Brookfield , Ben Ricardo , ausnog at lists.ausnog.net Subject: Re: [AusNOG] TPG Outage 5:30pm AEDT Can confirm here too. Large corporate using TPG and we lost all connections. They?re back online now but performing rather slow. On 10 Feb 2025, at 6:06?pm, Christopher Hawker wrote: Can confirm, have seen F1000 services in Sydney and Melbourne all go down about the same time. On a F1000 service in NSW, I did notice that a v4 BGP session lost about 10k routes, other than that it still appears to be working as expected. Regards, Christopher Hawker ________________________________ From: AusNOG on behalf of Nathan Brookfield Sent: Monday, February 10, 2025 5:59 PM To: Ben Ricardo ; ausnog at lists.ausnog.net Subject: Re: [AusNOG] TPG Outage 5:30pm AEDT Can certainly confirm, apparently there helpdesk, website and large amounts of other customer facing systems are down. From: AusNOG on behalf of Ben Ricardo Date: Monday, 10 February 2025 at 16:51 To: ausnog at lists.ausnog.net Subject: [AusNOG] TPG Outage 5:30pm AEDT Hi Folks, It would appear to me that TPG have some sort of problem starting approx. 5:25pm Anyone else for an early mark today? Ben Ben Ricardo | Senior Technician | M Net&SysAdmin, MCITP-SA, CEHv8, ITIL Australian Computer Solutions Pty Ltd | 2/28 Barralong Rd Erina NSW 2250 | P: 02 4365 2727 or 1300-807-131 | F: 02 4365 2304 | E: ben.ricardo at acs.com.au _______________________________________________ AusNOG mailing list AusNOG at lists.ausnog.net https://lists.ausnog.net/mailman/listinfo/ausnog _______________________________________________ AusNOG mailing list AusNOG at lists.ausnog.net https://lists.ausnog.net/mailman/listinfo/ausnog -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike at ozonline.com.au Mon Feb 10 19:10:32 2025 From: mike at ozonline.com.au (Michael Bethune) Date: Mon, 10 Feb 2025 19:10:32 +1100 Subject: [AusNOG] TPG Outage 5:30pm AEDT In-Reply-To: References: <4cd6418b9bdd4b64843e9fe95f64711b@acs.com.au> <6AE34897-14A4-4E0F-94AB-5D9D020D01FF@me.com> , Message-ID: <20250210191032.0hjfto6rs444s4ck@horde-2.ozonline.com.au> Well at least now the AAPT phones ring, that's encouring. Frontier is still stuffed and we've not seeing any NWB services authenticate since 17:27. - Michael. Quoting Ben Ricardo : > We're seeing services returning to normal now. > I don't suppose the power's back on in Glebe > > > > > -------- Original message -------- > From: Christopher Hawker > Date: 10/2/25 6:31?pm (GMT+10:00) > To: Nathan Brookfield , Clayton > Gee > Cc: Ben Ricardo , ausnog at lists.ausnog.net > Subject: Re: [AusNOG] TPG Outage 5:30pm AEDT > > Ausgrid does report a power outage in Glebe, but surely their > national network isn't reliant on a single site, running single-feed > power without some form of a UPS sitting between it and their > diesel genset... > > Regards, > Christopher Hawker > ________________________________ > From: Nathan Brookfield > Sent: Monday, February 10, 2025 6:17 PM > To: Clayton Gee ; Christopher Hawker > > Cc: Ben Ricardo ; ausnog at lists.ausnog.net > > Subject: Re: [AusNOG] TPG Outage 5:30pm AEDT > > > Looks like there primary DNS etc is still down, Frontier and other > systems the same. > > > > My money is on power related issue in the Glebe, Pymont area being related. > > > > From: Clayton Gee > Date: Monday, 10 February 2025 at 17:12 > To: Christopher Hawker > Cc: Nathan Brookfield , Ben > Ricardo , ausnog at lists.ausnog.net > > Subject: Re: [AusNOG] TPG Outage 5:30pm AEDT > > Can confirm here too. Large corporate using TPG and we lost all > connections. They?re back online now but performing rather slow. > > > > On 10 Feb 2025, at 6:06?pm, Christopher Hawker wrote: > > > > Can confirm, have seen F1000 services in Sydney and Melbourne all go > down about the same time. On a F1000 service in NSW, I did notice > that a v4 BGP session lost about 10k routes, other than that it > still appears to be working as expected. > > > > Regards, > > Christopher Hawker > > ________________________________ > > From: AusNOG on behalf of Nathan > Brookfield > Sent: Monday, February 10, 2025 5:59 PM > To: Ben Ricardo ; ausnog at lists.ausnog.net > > Subject: Re: [AusNOG] TPG Outage 5:30pm AEDT > > > > Can certainly confirm, apparently there helpdesk, website and large > amounts of other customer facing systems are down. > > > > From: AusNOG on behalf of Ben > Ricardo > Date: Monday, 10 February 2025 at 16:51 > To: ausnog at lists.ausnog.net > Subject: [AusNOG] TPG Outage 5:30pm AEDT > > Hi Folks, > It would appear to me that TPG have some sort of problem starting > approx. 5:25pm > Anyone else for an early mark today? > > Ben > > Ben Ricardo | Senior Technician | M Net&SysAdmin, MCITP-SA, CEHv8, ITIL > Australian Computer Solutions Pty Ltd | 2/28 Barralong Rd Erina NSW 2250 | > P: 02 4365 2727 or 1300-807-131 | F: 02 4365 2304 | E: ben.ricardo at acs.com.au > > > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > > > From mike at ozonline.com.au Mon Feb 10 19:33:57 2025 From: mike at ozonline.com.au (Michael Bethune) Date: Mon, 10 Feb 2025 19:33:57 +1100 Subject: [AusNOG] TPG Outage 5:30pm AEDT In-Reply-To: <20250210191032.0hjfto6rs444s4ck@horde-2.ozonline.com.au> References: <4cd6418b9bdd4b64843e9fe95f64711b@acs.com.au> <6AE34897-14A4-4E0F-94AB-5D9D020D01FF@me.com> , <20250210191032.0hjfto6rs444s4ck@horde-2.ozonline.com.au> Message-ID: <20250210193357.owmyrrecqo0s8csc@horde-2.ozonline.com.au> Nope, AAPT's customer service number "is currently not available, please try later." Not encouraging. - Michael. Quoting Michael Bethune : > > Well at least now the AAPT phones ring, > that's encouring. > Frontier is still stuffed > and we've not seeing any NWB services authenticate since 17:27. > > - Michael. > > Quoting Ben Ricardo : > >> We're seeing services returning to normal now. >> I don't suppose the power's back on in Glebe >> >> >> >> >> -------- Original message -------- >> From: Christopher Hawker >> Date: 10/2/25 6:31?pm (GMT+10:00) >> To: Nathan Brookfield , Clayton >> Gee >> Cc: Ben Ricardo , ausnog at lists.ausnog.net >> Subject: Re: [AusNOG] TPG Outage 5:30pm AEDT >> >> Ausgrid does report a power outage in Glebe, but surely their >> national network isn't reliant on a single site, running >> single-feed power without some form of a UPS sitting between it >> and their diesel genset... >> >> Regards, >> Christopher Hawker >> ________________________________ >> From: Nathan Brookfield >> Sent: Monday, February 10, 2025 6:17 PM >> To: Clayton Gee ; Christopher Hawker >> >> Cc: Ben Ricardo ; ausnog at lists.ausnog.net >> >> Subject: Re: [AusNOG] TPG Outage 5:30pm AEDT >> >> >> Looks like there primary DNS etc is still down, Frontier and other >> systems the same. >> >> >> >> My money is on power related issue in the Glebe, Pymont area being related. >> >> >> >> From: Clayton Gee >> Date: Monday, 10 February 2025 at 17:12 >> To: Christopher Hawker >> Cc: Nathan Brookfield , Ben >> Ricardo , ausnog at lists.ausnog.net >> >> Subject: Re: [AusNOG] TPG Outage 5:30pm AEDT >> >> Can confirm here too. Large corporate using TPG and we lost all >> connections. They?re back online now but performing rather slow. >> >> >> >> On 10 Feb 2025, at 6:06?pm, Christopher Hawker wrote: >> >> >> >> Can confirm, have seen F1000 services in Sydney and Melbourne all >> go down about the same time. On a F1000 service in NSW, I did >> notice that a v4 BGP session lost about 10k routes, other than >> that it still appears to be working as expected. >> >> >> >> Regards, >> >> Christopher Hawker >> >> ________________________________ >> >> From: AusNOG on behalf of Nathan >> Brookfield >> Sent: Monday, February 10, 2025 5:59 PM >> To: Ben Ricardo ; ausnog at lists.ausnog.net >> >> Subject: Re: [AusNOG] TPG Outage 5:30pm AEDT >> >> >> >> Can certainly confirm, apparently there helpdesk, website and large >> amounts of other customer facing systems are down. >> >> >> >> From: AusNOG on behalf of Ben >> Ricardo >> Date: Monday, 10 February 2025 at 16:51 >> To: ausnog at lists.ausnog.net >> Subject: [AusNOG] TPG Outage 5:30pm AEDT >> >> Hi Folks, >> It would appear to me that TPG have some sort of problem starting >> approx. 5:25pm >> Anyone else for an early mark today? >> >> Ben >> >> Ben Ricardo | Senior Technician | M Net&SysAdmin, MCITP-SA, CEHv8, ITIL >> Australian Computer Solutions Pty Ltd | 2/28 Barralong Rd Erina NSW 2250 | >> P: 02 4365 2727 or 1300-807-131 | F: 02 4365 2304 | E: >> ben.ricardo at acs.com.au >> >> >> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog >> >> _______________________________________________ >> AusNOG mailing list >> AusNOG at lists.ausnog.net >> https://lists.ausnog.net/mailman/listinfo/ausnog >> >> >> > > > > > _______________________________________________ > AusNOG mailing list > AusNOG at lists.ausnog.net > https://lists.ausnog.net/mailman/listinfo/ausnog > > ______________________________________________________________________ > This mail has been virus scanned and spam scanned by Australia On Line > see http://www.australiaonline.net.au/spamscanning > > Report this email as spam... > http://spam.ozonline.com.au?mail=mike&1739175055.2684587_1.mailscanner4.ozonline.com.au From Tom.Sykes at tpgtelecom.com.au Mon Feb 10 19:46:24 2025 From: Tom.Sykes at tpgtelecom.com.au (Tom Sykes) Date: Mon, 10 Feb 2025 08:46:24 +0000 Subject: [AusNOG] TPG Update Message-ID: <37CF8245-EFB8-4E88-A7CB-6E9A56713C54@tpgtelecom.com.au> Hi Everyone, I wanted to give you a brief update. Our technical teams are working on a power-related issue in Glebe affecting fixed-line data and voice services and we are working to restore services as quickly as possible. Data services are gradually being restored but there are some intermittent issues with our support telephone numbers which are also being worked on in parallel. We appreciate your patience while we work to restore all services as soon as possible. I am happy to take a call any time and my details are below. Regards, Tom Sykes GM Operations ? Enterprise, Government & Wholesale TPG Telecom Mobile: +61 439307942 Email: tom.sykes at tpgtelecom.com.au Confidential -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 1470 bytes Desc: image001.jpg URL: From bevan at slattery.net.au Mon Feb 10 20:48:56 2025 From: bevan at slattery.net.au (Bevan Slattery) Date: Mon, 10 Feb 2025 09:48:56 +0000 Subject: [AusNOG] TPG Update In-Reply-To: <37CF8245-EFB8-4E88-A7CB-6E9A56713C54@tpgtelecom.com.au> References: <37CF8245-EFB8-4E88-A7CB-6E9A56713C54@tpgtelecom.com.au> Message-ID: Ross St Glebe? The OG?! ________________________________ From: AusNOG on behalf of Tom Sykes Sent: Monday, February 10, 2025 6:46 pm To: ausnog at lists.ausnog.net Subject: [AusNOG] TPG Update Hi Everyone, I wanted to give you a brief update. Our technical teams are working on a power-related issue in Glebe affecting fixed-line data and voice services and we are working to restore services as quickly as possible. Data services are gradually being restored but there are some intermittent issues with our support telephone numbers which are also being worked on in parallel. We appreciate your patience while we work to restore all services as soon as possible. I am happy to take a call any time and my details are below. Regards, Tom Sykes GM Operations ? Enterprise, Government & Wholesale TPG Telecom Mobile: +61 439307942 Email: tom.sykes at tpgtelecom.com.au Confidential -------------- next part -------------- An HTML attachment was scrubbed... URL: From Tom.Sykes at tpgtelecom.com.au Mon Feb 10 23:57:46 2025 From: Tom.Sykes at tpgtelecom.com.au (Tom Sykes) Date: Mon, 10 Feb 2025 12:57:46 +0000 Subject: [AusNOG] TPG Update In-Reply-To: <37CF8245-EFB8-4E88-A7CB-6E9A56713C54@tpgtelecom.com.au> References: <37CF8245-EFB8-4E88-A7CB-6E9A56713C54@tpgtelecom.com.au> Message-ID: Hi Everyone, A further update in relation to tonight's disruption for our AAPT Wholesale customers. We have seen that the vast majority of services are now restored, with most of our fixed data services resolved earlier this evening. We have seen a large proportion of NWB services restore now. Our Wholesale support team can be contacted on 1300 851 299 for any residual issues and are available 24x7. Thanks for your patience whilst we worked through this, and I apologise for the disruption. I am happy to take a call any time and my details are below. Regards, Tom Sykes GM Operations ? Enterprise, Government & Wholesale TPG Telecom Mobile: +61 439307942 Email: tom.sykes at tpgtelecom.com.au Confidential From david at hughes.id Mon Feb 24 16:15:11 2025 From: david at hughes.id (david at hughes.id) Date: Mon, 24 Feb 2025 15:15:11 +1000 Subject: [AusNOG] AusNOG 2025 Sponsorship Opportunities Message-ID: Good afternoon everyone, We'll be kicking off the sponsorship process for the 2025 conference in a few weeks. This year will be bigger and better than ever, with room for 500 members of our industry to attend. If your organisation is interested in promoting itself to our audience, and supporting our not-for-profit activities in the process, please subscribe to the AusNOG Sponsorship mailing list. It's a very low volume list (a few emails per year at most) and we release all the details about sponsorship opportunities to the people on that list. You can join up at the link below: https://www.ausnog.net/lists/sponsorship Regards David Hughes Director, Australian Network Operators Group -------------- next part -------------- An HTML attachment was scrubbed... URL: