[AusNOG] NBN NNI recommendations

Tue Nov 24 16:43:05 EST 2015

For those interested, I believe I have found the reason for the split horizon bridging configuration on our existing setup.

http://www.nbnco.com.au/content/dam/nbnco2/documents/sfaa-wba2-product-catalogue-nebs-product-tech-spec_20151102.pdf

On page 56 it says:
All service frames exiting the NNI (i.e. from the NBN Co Network to the Customer Network through the NNI) must traverse an IP device before being injected back into the NBN Co Network. This is necessary to avoid CPE MAC addresses from appearing as source addresses on traffic ingress to the NNI. This operating restriction must be observed by Customer even if service frames are being switched between VLANs or forwarded via other service provider networks.

I'm guessing this is to avoid the potential of a broadcast storm from a misconfiguration on our network. Because our current configuration bridges all AVC VLANs together (they are in the same subnet), the customer services can normally access each other at layer 2, causing NBN systems to block all traffic.

Thanks everyone who has provided their thoughts both on and off-list. I'm still interested in hearing more thoughts, and will collate the information to share in a more concise format at a suitable time.

Regards,
Philip Loenneker

From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Philip Loenneker
Sent: Tuesday, 24 November 2015 2:14 PM
To: ausnog at ausnog.net
Subject: Re: [AusNOG] NBN NNI recommendations

Hi Matt,

Thanks so much for all of that information. The queuing section of your email in particular will help me in addressing issues we've already identified. I can confirm that software-based queues show similar underperformance to what you describe. I think looking for some new kit that has hardware queues will be top of the todo list.

To clarify a point that I may not have explained very well - we have a WBA and deal direct with NBN for most of our services. We only used a wholesaler for PoIs where we weren't set up yet.

On the point of DHCP, I'm afraid I didn't explain myself there very well either. I'm aware of the Option 82 information, and we successfully have that configured. We aren't assigning static IPs via MAC address as I think you interpreted it. The issue we have is that if a customer changes their router, they have to wait for the DHCP lease to expire (or call us to manually delete the lease) before they can get online again. Reducing the lease time could alleviate that, but could also cause additional problems. I have heard complaints about other providers having the same issue.

I'll have to process your comments on 2-to-2 VLAN swap some more - I don't follow it just yet, but will think on it. My understanding is that NBN will not pass traffic on a service until it has seen a valid DHCP request and response on the link (for an IPoE connection), which I thought would require an intermediate device connected to the customer NTD.

Thanks also for the link to the WBA documentation, I was relying on random other links (many outdated!) instead of having a nice concise list.

Regards,
Philip Loenneker | Network Engineer | TasmaNet
40-50 Innovation Drive, Tas 7010, Australia
P: 03 6165 2542 | M: 0404 097 816
philip.loenneker at tasmanet.com.au<mailto:philip.loenneker at tasmanet.com.au>
www.tasmanet.com.au<http://www.tasmanet.com.au/>

From: Matt Carter [mailto:matt.carter at iseek.com.au]
Sent: Tuesday, 24 November 2015 12:19 PM
To: Philip Loenneker <Philip.Loenneker at tasmanet.com.au>; ausnog at ausnog.net
Subject: RE: NBN NNI recommendations

IMHO getting the queueing/shaping right is the single most important factor. In my experience, using a rate-limit will result in significantly reduced performance amplified by the fact that there is quite often only 1 or 2 users on the end of it with very few streams to spread the policing across. I would be very surprised if you are getting equivalent to the respective speed tier as to avoid NBNcos policer you would need such aggressive values that a single TCP stream will continuously collapse and restart the moment the window size slides out. As per the NBN NNI specification the AVC should be shaped at no more than 10ms of peak burst size (PBS). Additionally the overall CVC should also be shaped at no more than 10ms PBS.  In my experience, this is more like 1-4 ms in Cisco land and you need ASR equivalent to be able to shape correctly (in hardware). If you cannot do this, ask your supplier if they can do it for you.

Why in hardware? The timing interval of the CPU/software based platforms (eg classical LNS) don't have the clock resolution needed to shape at the fine resolutions required by NBNCo. For example if we applied a shaper on a Cisco software based platform which has a minimum clocking interval of 4ms and then sample the traffic at sub ms rates, we can see peaks of 13.7Mbps and such which are getting clipped by NBNCo's policer despite the average rate (over 4ms) being <12mbps (11.5Mbps) (see http://i.imgur.com/NGP6LlG.jpg)  If we were to use a policer we can see even when speedtest fires up 4 parallel streams it cannot get more than 13Mbps out of the link because each one is being repeatedly smacked down due to the aggressive PBS  (see http://i.imgur.com/E8UIpgE.jpg ) if we apply a shaper in hardware at 1ms resolution things look mucho better (see http://i.imgur.com/UeJdwWh.jpg ) (ASR will shape to us values with "lowburst-enabled")

Now, on to the questions at hand.

*         Benefits of terminating the CVC on a switch vs the router that terminates the AVCs (providing DHCP+RADIUS+Queuing)

Really comes down to what your router is and what you want to do at the AVC/CVC level. The main reason I can see for moving the NNI to a switch is MTBF on ye old router vs ye old switch and whether your router has redundant supervisor/linecards and can accommodate for all manner or failures. A switch would allow you to break out to multiple aggregation points and/or potentially do S/C-TAG manipulation that your router may not be able to. Without knowing the switchr/router/topology and objectives this is a bit of a grey one.

*         How to avoid the DHCP lease causing grief for customers that change their on-premises router (MAC address change)

Ideally you want to take the Option 82 information and bounce this off your radius server. So in the DHCP DISCOVER the Option 82 circuit ID will be inserted by an intermediate agent being NBNCo as follows

    Option: (82) Agent Information Option
        Length: 36
        Option 82 Suboption: (1) Agent Circuit ID
            Length: 15
            Agent Circuit ID: 415643303030303031303130303437

Hex to Alphanumeric   415643303030303031303130303437 = AVC000001010047

This then allows you to authenticate the service and provide it an IP regardless of MAC changes on the UNI-D side.

*         Our current config requires split-horizon bridging, otherwise traffic was dropped by nbn - this breaks inter-customer connectivity which we would like to resolve

Ideally I would attempt to preserve the S/C-TAG (or at least C-TAG) all the way through to the aggregation layer so you have discrete services you can work with (as you mention you are doing with your other supplier) Have the supplier indicate to you in the provisioning process what the Vlan tag(s) are for the AVC. This avoids using bridge groups with split horizons at Layer2 and lets routing take care of the inter-CE connectivity in the public table or drop the service into a VRF, MPLS/VPLS/L2TP it elsewhere or basically whatever.  You can still use a common configuration to aggregate all C-Tags under a common S-Tag for example S-Tag 106 C-Tags 1000-3999 ;

interface GigabitEthernet0/2.106
encapsulation dot1Q 106 second-dot1q 1000-3999

All of the NBN specification documents are available here but they won't go into the design you talk of as that is in the realm of the access seeker (ie your supplier) . There is a couple of ways to cook the goose but I suspect what they'll be doing is a 2-to-2 swap with a egress pop. So on the BNG the AVC will be provided by NBNCO in the format of S/C-Tag let's say S-tag 2150 and C-tag 3766 . BNG would then vlan swap the outer to an internal vlan and vlan swap the inner to an incrementing vlan. Eg swap 2150/3766 to 100/1000 . Your traffic is carried through their network on a single Vlan 100 and on the egress side (your AGVC to the supplier) they will pop Vlan 100 exposing the underlying C-Tag so you see a single Vlan per AVC (auto incrementing from 1000)

http://www.nbnco.com.au/sell-nbn-services/supply-agreements/wba2.html

If you need any clarification just holler.

... and I've said it before, and I'll say it again, the silence from NBNCo on this list is deafening. C'mon guys, come to the party  ;) ;)

From: AusNOG [mailto:ausnog-bounces at lists.ausnog.net] On Behalf Of Philip Loenneker
Sent: Tuesday, 24 November 2015 9:05 AM
To: ausnog at ausnog.net<mailto:ausnog at ausnog.net>
Subject: [AusNOG] NBN NNI recommendations

Hi all,

We already supply NBN services to customers, however our NNI configuration is not ideal. We are in the process of building a new NNI for a PoI we will start servicing soon, and rather than just copy what we already have, we're going back to the drawing board to try identify a better setup. I've been reading through the NBN technical documents, but I'm after some advice beyond the scope of their documentation. I wasn't around when it was originally set up so I'm going by what I've been told and have been able to identify so far. Please excuse any foolish questions :)

We are using IPoE, not PPPoE. The CVC is terminating on a single router hooking into Freeradius for static IP and rate limit configs.

I'm particularly interested in getting some advice based on practical experience regarding what works and what should be avoided, including:

*         Benefits of terminating the CVC on a switch vs the router that terminates the AVCs (providing DHCP+RADIUS+Queuing)

*         How to avoid the DHCP lease causing grief for customers that change their on-premises router (MAC address change)

*         Our current config requires split-horizon bridging, otherwise traffic was dropped by nbn - this breaks inter-customer connectivity which we would like to resolve

On a related note, we have previously purchased layer 2 NBN services from another supplier, where they present a VLAN on our cross-connect which is effectively presented untagged on a UNI-D port. We put IPs at each end and can route across it without needing DHCP etc. There is no additional equipment connected to the UNI-D to terminate a VPN, it's just our device with no special port config. I can't find documentation around this type of service (it seems to be available for voice and multicast traffic, but not regular traffic), but it's something we're interested in being able to supply. Does anyone know how this might be done?

Any thoughts would be appreciated.

Regards,
Philip Loenneker
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20151124/d0e3af17/attachment.html>