I was recently working a project where I was significantly redesigning the WAN infrastructure for a large international organization.  This organization has between 400 and 500 sites, and many different methods of potential connectivity.  The issue that we were solving was that path selection wasn’t always predictable, especially in failure scenarios, and asymmetric flows caused havoc on some of their applications.

Due to the large nature of the change, it was split up into smaller components (phases) and planned to be implemented across a rather large window in a single day.  We had a need to verify that we had a consistent environment at the end of each of each phase, before moving on to the next.  We also needed a tool to rapidly test a multitude of failure scenarios during failover/user acceptance testing.  The other requirement is that this testing happen from many perspectives.  I needed to not only verify connectivity, but that the flow of traffic in both directions was selecting a consistent path.

Because of these requirements, I decided to use TCL (eventually EEM-TCL) and run the script from the routers and switches themselves.  I used this approach, because it wasn’t practical to set up machines at each of these locations to run the script locally.  This ended up working incredibly well for our purposes and since I couldn’t find any good pre-built examples of what I was trying to do, I’ve sanitized the script and posted it for others to use.  The functional information on how it runs is on the GitHub page. Simply put, the script is pre-configured with a bunch of test targets that have ping and traceroute run against them.  Output is simplified so only relevant information is displayed and so you can quickly determine if there is an issue.  For the change referenced above, there was about 20-30 different target IP addresses (internal and external) that gave a sampling of the different types of locations/connectivity types.

In larger organizations I could see this script could also being used as a level 1 troubleshooting tool.  Run the script, and validate against a “known good” output before escalating to an upper tier.  Escalation could have a copy of the output which would potentially point to trouble areas in the network.  You could also take this script and have it run whatever commands you like.  TCL gives you some pretty powerful options for variable handling and output filtering, so this could just be a good starting point on setting up whatever validations you would like to run.

See Script Here

Network Collective

Two weeks ago I was able to announce a side project that a couple of friends and I have been working on called Network Collective.  Network Collective is a live-streamed, video roundtable, where network practitioners of all experience levels talk about the things that interest them, and hopefully the things that interest you.  Rather than explaining what Network Collective is (that’s already been done here), I want to share a bit about why I’m so passionate about it and why I believe we need something like it in the networking community.


Over the past couple years, I have been observing that highly technical conversations are moving from public to private platforms.  Quite a bit of what used to take place in the open on Twitter, and on networking podcasts, had moved to Slack channels and other private chats.  It makes sense since Twitter’s 140 character limit makes communicating complex ideas difficult.  The introduction of Slack, and other communication platforms like it, allow for a more nuanced conversation to occur.  The problem is that you have to be aware that the channel exists, and be invited to participate in it.  For those of us who are established in this community it is great, but it leaves out the up and coming and even the experienced engineers who are just joining this socially minded community of network engineers.

I know that my career trajectory changed when I was introduced to the network engineering community on Twitter (thanks again Jeff).  I didn’t understand half of the acronyms or content when I first started, but you better believe that Google was my friend.  I found inspiration in hearing my peers talk about things that I had never considered, and my perspective became broader and better informed because of it.  This public discussion satiated my innate curiosity and was the fuel that drove me to dig deeper in to this career that I had chosen.  I am the engineer that I am today because of this exposure to other peoples experiences and passions.  It bothers me to think that this environment doesn’t exist, to the same degree as it did for me, for those who are just starting down this path now.

As time went along, my involvement in this community not only exposed me to the collective experience of the network community, but it also afforded me opportunities to start dialog with some of the people I admired most.  One of the most incredible things about the public figures in the network engineering community is just how approachable they are.  From dialog, relationships and friendships have grown, and those relationships make both online, and in-person experiences like conferences, all the more rewarding.


My goal with Network Collective is to bring the technical conversation back into the public square and to build a platform where listeners and contributors alike can start dialog and build stronger relationships within the community. Sure, we’re going to have a lot of fun geeking out about the things we’re passionate about along the way, but I want those conversations to benefit the community at large rather than a smaller subset of it in the back channel.  We would love nothing more than for you to join us.  The more people that participate in the conversation, the more valuable the conversation becomes.  Our fist episode of Network Collective is going to be airing tomorrow, April 11th, at 7:00 PM EDT.  You’ll be able to watch it directly on our website:  http://thenetworkcollective.com.  If you can’t participate in real-time, the video recordings will be posted on our website after the fact.  If you want the video feed delivered directly to you, you can subscribe to our YouTube channel directly as well.  While I believe the video format better enhances the community aspect of the show, we will be releasing an audio-only feed of the episode as well.  Just search for Network Collective in your favorite podcatcher or go to our Subscription Page to find the direct RSS feed.

As a final note, I just want to express how grateful I am for the response of the networking community. When we put this idea out there, the outpouring of support was nothing short of amazing.  I would be remiss to not mention two very important people who are making this vision a reality.  Eyvonne Sharp and Phil Gervasi partnered with me the moment that the idea was introduced and are an integral part of making this happen.  We’re very excited to get this started and very much hope that you will join us along the way.  We would love to have you.

I started this blog with the best of intentions but unfortunately life changes sometimes.  There has been a tremendous amount of significant personal change going on in my life recently and that is the reason for the stagnation of this space.Simply, my daughter has been diagnosed with a conditioned called tuberous sclerosis.  It is a significant, debilitating, disorder that causes benign cancerous growths to grow in many of the bodies primary organs (brain, kidney, heart, skin, eyes, etc…).  My wife and I have started a blog to detail our journey and if you are interested it can be found here:  http://www.lydiablog.com.

If you have been checking here occasionally, as I know some of you have, I apologize for the false start.  I do still intend to get this blog rolling but there just are other things that need my attention at the moment.  Thank you for your support.

Jordan