[AusNOG] Looking for someone who has experience with Nextgen VPLS

Thu Apr 7 10:04:36 EST 2011

These are some of the keys that i was looking for also for win2k8 try from a cmd prompt "nets hint tcp set global autotuninglevel=normal"

Additional resources are also available for tweaking the hell out of windows to make it comparable to linux:
Cablenut: http://testmy.net/ipb/index.php?app=core&module=attach&section=attach&attach_id=9194
Cablenut pre-made settings: http://testmy.net/ipb/index.php?app=core&module=attach&section=attach&attach_id=9194

Like i previously said, the windows box was hitting about 1.7mbit, on an 8mbit connection but the linux box was hitting the 8mbit, with the tweaks above i was about to get the windows box to 8mbit, the only thing i would caution is if you tweak to a certain speed (10mb, 100mb etc) make sure you re-tweak if you ever change speeds dramatically, i know from experience if i ever get shaped @ home, i need to remove my hacks so i have a working sucky internet connection.

James Troy |
Network / Windows Administrator

INTRALOT AUSTRALIA PTY LTD
[A] Level 1  283 Normanby rd
     Port Melbourne VIC 3207
[T] + 61 3 9915 6267
[E] james.troy at intralot.com.au
[F] + 61 3 9645 6066
[W] www.intralot.com.au

 Please consider the environment before printing this e-mail

-----Original Message-----
From: Joshua Small [mailto:JSmall at daraco.com.au]
Sent: Thursday, 7 April 2011 10:00 AM
To: James Troy; Greg M; 'Mark Smith'
Cc: ausnog at lists.ausnog.net
Subject: RE: [AusNOG] Looking for someone who has experience with Nextgen VPLS

Hi,

The common fix for bizarre Windows networking behaviour is to disable it's "should be intelligent but often aren't" networking features:

In the following key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
Right-click EnableTCPChimney, and then click Modify.
In the Value data box, type 0, and then click OK.
Right-click EnableRSS, and then click Modify.
In the Value data box, type 0, and then click OK.
Right-click EnableTCPA, and then click Modify.
In the Value data box, type 0, and then click OK.
Exit Registry Editor, and then restart the computer

You may also surprise yourself in a "Windows over WAN" situation with this reg key:
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Lsa\ Kerberos\Parameters\MaxPacketSize = 1 (DWORD)

This takes a reboot to take effect.

Joshua Small

Daraco Services
IT Consulting and Support
Unit 17, 7 Anella Avenue Castle Hill 2154
Phone: 1300 327 226 Fax: +612 8588 1200
Email: jsmall at daraco.com.au www.daraco.com.au

This message and any attachments are solely for the intended recipient and may contain confidential or privileged information. If you are not the intended recipient, any disclosure, copying, use, or distribution of the information included in this message and any attachments is prohibited. If you have received this communication in error, please notify us by reply e-mail and immediately and permanently delete this message. If you do not wish to receive messages from this email account, then please reply with unsubscribe me in the subject line to: unsubscribe at daraco.com.au

-----Original Message-----
From: ausnog-bounces at lists.ausnog.net [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of James Troy
Sent: Thursday, 7 April 2011 9:49 AM
To: Greg M; 'Mark Smith'
Cc: ausnog at lists.ausnog.net
Subject: Re: [AusNOG] Looking for someone who has experience with Nextgen VPLS

I have had similar issues in the past, two boxes sitting next to each other the linux could hit 10mbit, the windows couldn’t, After some research and a reg hack i had the windows box at full speed, i will see if i can find this today for you. This windows machine was an XP box, however i believe the hack will still work in 2k3, out of curiosity however does your windows 2k8 server suffer the same fate?

James Troy |
Network / Windows Administrator

INTRALOT AUSTRALIA PTY LTD
[A] Level 1  283 Normanby rd
     Port Melbourne VIC 3207
[T] + 61 3 9915 6267
[E] james.troy at intralot.com.au
[F] + 61 3 9645 6066
[W] www.intralot.com.au

 Please consider the environment before printing this e-mail

-----Original Message-----
From: ausnog-bounces at lists.ausnog.net [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Greg M
Sent: Wednesday, 6 April 2011 7:02 PM
To: 'Mark Smith'
Cc: ausnog at lists.ausnog.net
Subject: Re: [AusNOG] Looking for someone who has experience with Nextgen VPLS

Mark, you are correct.

This did not resolve the issue, however it definatelly does not appear to be
an issue with the Sydney<->Perth nextgen link as I can do 100Mbps across it
from a linux box to pretty much anywhere else on my network - however
Windows systems struggle to get even 1Mbps to some locations. I struggle to
understand this behaviour.

We have a win2k3 box in Sydney that only gets 60kb/s cifs speeds to Perth,
while on our previous 8Mbps SHDSL link it was able to get 800KB/s

I've tried putting some old 2811s in between the 2 x 3560's and setting the
adjust-mtu, and window-size options on them, but to no avail.

I'm also being told that due to the latency I need to invest in
TCP/bandwidth accelerators at each end of the link to work around these
issues... I may just be being ignorant about this, but I cant believe this
is the case, and there must be a simple way around this TCP issue that only
appears to affect windows PC's.

Cheers,

Greg

-----Original Message-----
From: Mark Smith
[mailto:nanog at 85d5b20a518b8f6864949bd940457dc124746ddc.nosense.org]
Sent: Tuesday, 5 April 2011 7:34 PM
To: Greg M
Cc: ausnog at lists.ausnog.net
Subject: Re: [AusNOG] Looking for someone who has experience with Nextgen
VPLS

On Tue, 5 Apr 2011 18:48:10 +0800
"Greg M" <gregm at servu.net.au> wrote:

> Thanks to a couple of replies off-list:
>
> ip tcp adjust-mss 1452
>
> One line on our Cisco 3560's resolved everything
>

That's very strange, because I'm pretty sure that doesn't adjust the TCP
window parameters that usually constrain the amount of data in flight i.e.
the bandwidth delay product. Those are the Window Size field in the TCP
header (limited to 16 bits) and the window scale option, used to scale the
Window Size field up so that it can represent values larger than 65535.
Typically this window size is constrained by how much memory the OS
allocates, with various earlier versions of windows not allocating enough to
fill up these large bandwidth delay product links (also known as LFNs or
Large Fat Networks ('elefants'))

http://en.wikipedia.org/wiki/TCP_window_scale_option

If the above has worked, then maybe the 3560 is adjusting more than the MSS
option, as the MSS option is only sent during the TCP 3 way handshake, well
before Window Sizes for in flight data have any influence.

> Greg
>
> -----Original Message-----
> From: Mark Smith
> [mailto:nanog at 85d5b20a518b8f6864949bd940457dc124746ddc.nosense.org]
> Sent: Tuesday, 5 April 2011 6:43 PM
> To: Greg M
> Cc: ausnog at lists.ausnog.net
> Subject: Re: [AusNOG] Looking for someone who has experience with
> Nextgen VPLS
>
> On Tue, 5 Apr 2011 18:27:11 +0800
> "Greg M" <gregm at servu.net.au> wrote:
>
> > Thanks again to the members of the Ausnog community the speed issue
> > is now resolved! :-)
> >
>
> Want to summarise what happen between 19:39 and 19:57 tonight? Did you
> get your Windows TCP windows issues sorted somehow?
>
>
> >
> >
> > Thanks to everyone who replied,
> >
> >
> >
> > Greg
> >
> >
> >
> > From: ausnog-bounces at lists.ausnog.net
> > [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Greg M
> > Sent: Tuesday, 5 April 2011 6:09 PM
> > To: ausnog at lists.ausnog.net
> > Subject: Re: [AusNOG] Looking for someone who has experience with
> > Nextgen VPLS
> >
> >
> >
> > Hi All,
> >
> >
> >
> > On the TCP Window size of things. Is there a way for the border
> > routers/switches on each end of the VPLS to be able to handle this
> > so that manually setting TCP Window and TCP Window scaling on every
> > single Windows Desktop PC/server isn't required?
> >
> >
> >
> > I just did some further tests, and from a Centos/Linux virtual
> > machine in Sydney, I can get 100Mbps to the iiNet FTP, and to both Perth
sites.
> >
> >
> >
> > --2011-04-05 20:03:02--  ftp://ftp.iinet.net.au/test500MB.dat
> >
> >            => `test500MB.dat'
> >
> > Resolving ftp.iinet.net.au... 203.0.178.32
> >
> > Connecting to ftp.iinet.net.au|203.0.178.32|:21... connected.
> >
> > Logging in as anonymous ... Logged in!
> >
> > ==> SYST ... done.    ==> PWD ... done.
> >
> > ==> TYPE I ... done.  ==> CWD not needed.
> >
> > ==> SIZE test500MB.dat ... 500000000
> >
> > ==> PASV ... done.    ==> RETR test500MB.dat ... done.
> >
> > Length: 500000000 (477M)
> >
> >
> >
> > 100%[===============================================================
> > ==
> > ======
> >
> ======================================================================
> ======
> > ================================================>] 500,000,000
> > 10.1M/s
> in
> > 46s
> >
> >
> >
> > 2011-04-05 20:03:48 (10.3 MB/s) - `test500MB.dat' saved [500000000]
> >
> >
> >
> >
> >
> > From a Windows server on the same subnet/switch, I get 4.5MB/s.. So
> > it seems like the link itself is fine, and it is indeed TCP Window
> > related. Why this behaviour is so radically different between
> > Windows and Linux is quite suprising.
> >
> >
> >
> > Thanks,
> >
> >
> >
> > Greg
> >
> >
> >
> > From: ausnog-bounces at lists.ausnog.net
> > [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Greg M
> > Sent: Tuesday, 5 April 2011 4:45 PM
> > To: ausnog at lists.ausnog.net
> > Subject: Re: [AusNOG] Looking for someone who has experience with
> > Nextgen VPLS
> >
> >
> >
> > Thanks for everyone's replies - I was able to do extensive
> > troubleshooting today thanks to the excellent replies on and
> > off-list, and it turns out that there may be multiple issues at
> > play, including too many MAC's on the circuit, et al.
> >
> >
> >
> > It also seems that Telstra has not enabled 802.1q-in-q on their end
> > so we are seeing a lot of encapsulation errors, and can only push
> > traffic across
> > VLAN1 from the Sydney side. They are now telling Nextgen that they
> > cant enable Q-in-q, which could point to Nextgen provisioning the
> > wrong type of Telstra service for us:
> >
> >
> >
> > The order from LOLO states "Telstra Wholesale Business Grade
> > Ethernet Service N275XXXXX for Nextgen Networks"
> >
> >
> >
> > Is this a L3 only service?
> >
> >
> >
> > Since this morning, we have only been able to push 10Mbps through
> > this VPLS circuit as well, so there definitely seems to be some
> > faults on the Nextgen/Telstra side of the circuit, even though they
> > claim everything is configured correctly.
> >
> >
> >
> > Cheers,
> >
> >
> >
> > Greg
> >
> >
> >
> > From: ausnog-bounces at lists.ausnog.net
> > [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Greg M
> > Sent: Monday, 4 April 2011 7:07 PM
> > To: ausnog at lists.ausnog.net
> > Subject: [AusNOG] Looking for someone who has experience with
> > Nextgen VPLS
> >
> >
> >
> > Hi Noggers,
> >
> >
> >
> > I am looking for someone who is able to consult/provide support to a
> > large organisation which has recently deployed a 100M Nextgen VPLS
> > link between an office in Perth & Sydney,  and is experiencing the
> > most bizzare throughput issues.
> >
> >
> >
> > This link has Nextgen fiber in Perth, with Telstra fiber hand off in
> > Sydney, resold via Nextgen as an end-to-end VPLS circuit.
> >
> >
> >
> > The issue we face is relating to speed.
> >
> > -          From Switch A In Sydney to Switch B in Perth, we can do a
> non-df
> > ping of 1500 bytes for 24 hours without a single dropped packet  -
> > so there are no MTU issues. Latency is also always stable, even when
> > maxing the link at 100Mbps.
> >
> > -          Each server in Sydney connected to Switch A, can push the
full
> > VPLS line speed of 100Mbps to servers connected to Switch B in Perth
> > (two endpoints of the VPLS tails)
> >
> > -          There is a 3rd site in Perth that has a 1Gbps Amcom fiber
link
> > directly between Switch C in this site, and Switch B in the other
> > Perth site.
> >
> > -          Some systems on Switch A in Sydney, can push 100Mbps to some
> > servers on Switch C in Perth site #2, yet others can only push
> > 10Mbp, while some can only push 1-3Mbps. There are no speed issues
> > between the two Perth sites, however as they can happily pump a full
> > 1Gbps to each
> other.
> >
> > -          There is a 100Mbps link to WAIX connecting into Switch B in
> > Perth. Perth site #2 can do full 100mbps to a test FTP download
> > (lets say iiNet's 500MB test file on their FTP), yet Sydney can only get
45Mbps.
> >
> >
> >
> > Basically only the directly connected Sydney Switch and Perth switch
> > are able to max out the 100Mbps VPLS circuit. As soon as traffic
> > goes beyond this switch, speed seems to degrade - however it doesn't
> > degrade to all locations, as some systems/servers can obtain a full
> > 100Mbps transfer speed both inbound and outbound via the Sydney to
> > Perth VPLS. In Perth sites #1 and #2, there are no speed issues at
> > all, to any site (there are other remote VPLS and other connected
> > endpoints), it is only this specific site that is having issues.
> >
> >
> >
> > There are no errors on any of the interfaces in any of the Perth
> > sites, however the Sydney site reports a lot of Outbound Discards on
> > the switch port that connects to the Telstra/nextgen VPLS link in
Sydney.
> >
> >
> >
> > Happy to receive responses on or off-list, and we are happy to pay
> > for someone either locally in Sydney or in Perth to help diagnose
> > this - as a lot of money is being burnt on this circuit that is not
> > providing consistent speeds. We have raised these issues to both
> > Nextgen and Telstra and they are pointing the finger at our internal
network.
> >
> >
> >
> > Thanks!
> >
> >
> > Greg
> >
>

_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
http://lists.ausnog.net/mailman/listinfo/ausnog

The information in this electronic mail is confidential, intended only for use of the individual or entity named. If you are not the intended recipient, any dissemination, copying or use of this information is strictly prohibited. If you have received this transmission in error please delete it immediately from your system and inform us by email on reply at intralot.com.au. This email and any attachments have been scanned for computer viruses using anti-viral software, but you should also perform your own scan. We do not accept liability for any loss or damage, whether caused by our own negligence or not, that results from a computer virus or a defect in the transmission of the email or any attached file.
_______________________________________________
AusNOG mailing list
AusNOG at lists.ausnog.net
http://lists.ausnog.net/mailman/listinfo/ausnog

The information in this electronic mail is confidential, intended only for use of the individual or entity named. If you are not the intended recipient, any dissemination, copying or use of this information is strictly prohibited. If you have received this transmission in error please delete it immediately from your system and inform us by email on reply at intralot.com.au. This email and any attachments have been scanned for computer viruses using anti-viral software, but you should also perform your own scan. We do not accept liability for any loss or damage, whether caused by our own negligence or not, that results from a computer virus or a defect in the transmission of the email or any attached file.