<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Consolas","serif";
color:black;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="white" lang="EN-AU" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Yeah we had a few services go down in Sydney and Melbourne. I was out and about last night at 2am and saw a few AAPT techs @ 55 Clarence st.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">All ours are backup now..<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext"> ausnog-bounces@lists.ausnog.net
[mailto:ausnog-bounces@lists.ausnog.net] <b>On Behalf Of </b>Matt Perkins<br>
<b>Sent:</b> Sunday, 1 July 2012 11:32 AM<br>
<b>To:</b> ausnog@lists.ausnog.net<br>
<b>Subject:</b> Re: [AusNOG] AAPT metro Ethernet<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Looks like most of the sites are back up. It appears that "Something" caused many remote switches on there network freeze. They then had to send lots of tech's out to the field all through the night to reset them.
<br>
<br>
A disaster of a night for AAPT im thinking. <br>
<br>
<br>
<br>
On 30/06/12 11:24 PM, Matt Perkins wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">there ptp Ethernet we use for some last miles. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">matt<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
<br>
On 30/06/2012, at 11:21 PM, "Joshua D'Alton" <<a href="mailto:joshua@railgun.com.au">joshua@railgun.com.au</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">Yea you'd think paying $90+/Mbit that they'd be able to put out an official report 7 hours after the start of an incident...
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Word on the street is they've got a configuration issue with a piece of hardware hence the delay and vendor support. Could be another 12 hours for all we know.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">What prefix are the offline services showing?<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Sat, Jun 30, 2012 at 11:16 PM, Matt Perkins <<a href="mailto:matt@spectrum.com.au" target="_blank">matt@spectrum.com.au</a>> wrote:<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">Still showing some services off here. They have this on there portal<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> FOPs onsite assisting IP Support. Gradually restoring services by rebooting nodes. Vendor has also been engaged. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">something like this occurred on metroE a few months back from memory. I wish these guys would communicate with there customers. <o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
<br>
On 30/06/2012, at 11:03 PM, Jared Hirst <<a href="mailto:jared.hirst@serversaustralia.com.au" target="_blank">jared.hirst@serversaustralia.com.au</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p>We saw some odd things happening today for customers connecting to us using their transit... but all is resolved now, were your issues around 2-4 pm?<o:p></o:p></p>
<p>Cheers,<o:p></o:p></p>
<p>Jared.<o:p></o:p></p>
<div>
<p class="MsoNormal">On Jun 30, 2012 10:56 PM, "Matt Perkins" <<a href="mailto:matt@spectrum.com.au" target="_blank">matt@spectrum.com.au</a>> wrote:<o:p></o:p></p>
<p class="MsoNormal">AAPT seem to have a wide spread outage on there Ethernet product. there a bit tight lipped at the moment. anybody know any more.<br>
<br>
<br>
_______________________________________________<br>
AusNOG mailing list<br>
<a href="mailto:AusNOG@lists.ausnog.net" target="_blank">AusNOG@lists.ausnog.net</a><br>
<a href="http://lists.ausnog.net/mailman/listinfo/ausnog" target="_blank">http://lists.ausnog.net/mailman/listinfo/ausnog</a><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
AusNOG mailing list<br>
<a href="mailto:AusNOG@lists.ausnog.net">AusNOG@lists.ausnog.net</a><br>
<a href="http://lists.ausnog.net/mailman/listinfo/ausnog" target="_blank">http://lists.ausnog.net/mailman/listinfo/ausnog</a><o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">_______________________________________________<br>
AusNOG mailing list<br>
<a href="mailto:AusNOG@lists.ausnog.net">AusNOG@lists.ausnog.net</a><br>
<a href="http://lists.ausnog.net/mailman/listinfo/ausnog">http://lists.ausnog.net/mailman/listinfo/ausnog</a><o:p></o:p></p>
</div>
</blockquote>
<p class="MsoNormal"><br>
<br>
<br>
<o:p></o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>AusNOG mailing list<o:p></o:p></pre>
<pre><a href="mailto:AusNOG@lists.ausnog.net">AusNOG@lists.ausnog.net</a><o:p></o:p></pre>
<pre><a href="http://lists.ausnog.net/mailman/listinfo/ausnog">http://lists.ausnog.net/mailman/listinfo/ausnog</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
</div>
</body>
</html>