[AusNOG] Thought experiment: How would you manage 11 (or 100) million devices?

[Andrew Fort]

Amongst the clever folk I'm presently working with, this has been a
recurring topic of conversation for some time.  We've had all sorts of
ideas that I'd like to share, but first, I figure it's worth posing
the question to operators.  So, here's a question for operators and
tool developers on the list in particular:

How would you go about managing (lets assume via IPvSomething, SNMP,
TL1, CLIs, et al) 11 million devices? *   Assume that you need to
gather performance stats, alerts, and configure the devices to do
stuff.

What sort of architecture would you build?  What technologies would
you use?  What assumptions are you making?  What sorts of team
structures and software development approaches would you use to do it,
and to make sure that you have the skills locally (not in Canada, or
where-ever else) to fix the inevitable problems?

* 11 million devices is the number of nodes the NBN folks mentioned at
AusNOG, but if you're thinking at this scale, you should be able to
make your solution scale to 100 million, or a billion devices.  Be
creative!

Cheers,
Andrew