<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-AU" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Perhaps I’m missing something in your explanation, but we can put in additional collectors at will by essentially cloning the VM and
adding them to the pool – isn’t that horizontal scaling? The netflow traffic is distributed across the pool, not replicated to each one, so adding extra servers reduces the load on the existing servers. The primary goal of this design was more to reduce the
risk of a single failure breaking everything rather than to enable horizontal scaling, but I think it achieves both.
<o:p></o:p></span></p>
<p class="MsoNormal"><br>
On 12 Oct 2015 9:19 am, "Philip Loenneker" <<a href="mailto:Philip.Loenneker@tasmanet.com.au">Philip.Loenneker@tasmanet.com.au</a>> wrote:<br>
><br>
> <snip><br>
> You'll also incur increased collector costs because you have to buy bigger ones of those too as the volume of collection data increases, because you can't spread the collection data across a number of collectors.<br>
> </snip><br>
><br>
> You can spread the load, you just have to be a little creative :)<br>
><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p></o:p></span></p>
<p>You've only spread the load across a number of collectors, and you've done that by adding another element for which the only way to scale is to buy a bigger one, so you're still pursuing vertical scaling. You've also added a another point of failure to the
whole system, because it is another element inline (in serial) with the netflow traffic path.<o:p></o:p></p>
<p>A simple test to work out if you're vertically or horizontally scaling is can you buy another device of the *same* size/capacity and it adds processing capacity to the system? If it does, then your scaling model is horizontal.<o:p></o:p></p>
<p>> A few weeks ago we successfully got a proof-of-concept up and running where we have NetFlow feeds going to a load balancer (NetScaler in our case), which then used source-IP persistence to spread the load over 4 collectors. The collectors run as stateless
UDP servers of course, which makes it difficult to check whether the services are running, so one of our guys set up an additional service running on TCP that would give a response code with the status of the collector, so we have health-checks and auto-failover
working. The collectors we are using are nProbe on Linux, which allows us to dump the raw flows directly into MySQL, so we load balanced that too and set up a cluster of MySQL servers as the destination. Tests have been quite promising, and we get redundancy
in a system that wasn't designed to. All of those servers are virtual, which makes scaling even simpler.<o:p></o:p></p>
</div>
</body>
</html>