For over 30 years I’ve been in the playing in the “world of IT” and wow has that world changed a great deal in those years. But through all that change, there has been a thread, for me, that has always remained constant.

Troubleshooting!

The thrill of the “chase” and the challenge of solving the “who done it”.  I’ve learned a lot over the years. What works… what doesn’t work.  What helps… what hinders.

Like any Detective “on the job” for so many years… it would be impossible to pass on to you everything I would really like to.  So let’s go with the tips I think will give you the biggest ROI if you apply them.

Tips from a Network Detective

1. Be Methodical
2. Know What is Normal (Knowledge is Key)
3. Get to the “Crime Scene” as Fast as You Can
4. Have “Crime Scene Maps”  that Help and don’t Hinder
5. Let the Clues and Evidence Guide You
6. Learn and Improve

Tip #1: Be Methodical

Detection is, or ought to be, an exact science and should be treated in the same cold and unemotional manner.              -Sherlock Holmes

This is going to be a hard one.  There is going to be not just the “who done it” you are trying to solve… but all the added stress, pressure, emotions and potential team tensions that will play into all this.  Now, more than ever, you need to be methodical.   You need to already know what is normal in your network. You need to already know where to look for the basic facts and what CLI or GUI you will be using.

My “troubleshooting methodology” is listed below.  A key tip? — Take notes and keep track of what you see

• Gather the Facts
• Collect the Clues
• Follow the Evidence
• Interview the Witnesses
• Question the Suspects

Tip #2: Know What is Normal (Knowledge is Key)

Knowing what is “normal” in your network is absolutely critical!  Why?   If you do not know what is “normal” in your network …. how could you begin to even hope to be able to differentiate a “fact” from a “clue” or “evidence”?

• Fact: The CPU of Router X is 80%.  Is this normal?  If you don’t know what is normal on this box, you do not know if this is a clue or evidence. You may end up wasting precious time and energy “questioning” this Router when this is merely a fact and not clue or evidence.

Related Tips:

• Know how your Traffic Flows in Your Network.
• Know what has Changed in your Network
• Know how your Network is Configured to react/respond to varying Failures in your Network
• Know the “Modus Operandi” of the Devices in your Network.
  • Fact: show proc cpu on an ASR1002-HX is 1%.  Question: Do you know then that this isn’t having CPU issues?  No. You can’t know that or assume that.  Why?  For this device you need to look at the CPU of the QFP.  If you do not know this, you are missing knowledge that is key.

Tip #3: Get to the “Crime Scene” as Fast as You Can

Just like in the world of detectives, Network Detectives benefit from getting on the “crime scene” as soon as possible. While the facts, clues and evidence are still fresh.

If you do not isolate your failure domains…. you may be exposed to experiencing cascading failures. So get on the crime scene as quickly as possible because once the cascading starts you will have multiple suspects to identify and that takes the challenge to a whole other level.

Network monitoring tools can be hugely essential here.  Some crimes you might even be able to avoid because you got called in before the crime when there were activities that your monitoring tools labeled as suspect.

Tip #4: Have “Crime Scene Maps”  that Help and don’t Hinder

Are your network diagrams up to date and accurate? Would they “help” as crime scene maps… or would they “hinder”?  Are your network diagrams easy to understand?

Let’s say we are detectives trying to solve a “who done it” in a 1 story house.  The floor plan (aka “crime scene map”) is as we see it below.  The crime was committed in the Lounge.  All the clues and evidence for motive point to a suspect who was in the Conservatory. No surveillance cameras and no witnesses ever saw the suspect in the hallway.

Question: Based on these facts can we “let loose” on the theory we are holding that he did it?

Answer: In this situation absolutely not.  Why?  Because remember – knowledge is key.  And there is extremely key knowledge missing on this crime scene map.

So what about the above “crime scene map” is a “hindrance” and not a “help”?    Ever play the game of clue?

There are 2 hidden passages that are missing from our map.  There is a secret passage between the Kitchen and the Study… as well as a secret passage between the Conservatory and the Lounge.  This essential fact is missing from the map.  Armed with these new facts we can now go search the secret passage between the Conservatory and the Lounge and see if there are any clues or evidence we can find.

Tip #5: Let the Clues and Evidence Guide You

It is a capital mistake to theorize before one has data.  Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.  – Sherlock Holmes

First opinions are crucial, but if the evidence changes, so must the theory.  – Gil Grissom, CSI LasVegas

This is one of the ones that seems to challenge just about everyone. Why?  Because we are human.

We come up with theories that end up guiding our decision and our troubleshooting…. instead of letting the clues and evidence guide us.    You are going to have theories.  That is just a fact. But hold them loosely.

Don’t let your theory guide you… let the clues and evidence alone be your guide. If you don’t you are going to waste time and miss essential clues and evidence.  You know what else this will do?  It will be the filter you use when you are asking someone else to take a look at something.  Trust me, it will.  I see this ALL the time. You are trying to “help” but when you tell me the facts, clues, and evidence they will be from a biased perspective because you already have a theory you are holding onto so tightly it is coloring everything.

Tip #6: Learn and Improve

Insanity is doing the same thing over and over again and expecting different results.

Every network outage is a chance to improve the network.  So after a network outage, regardless of how small, ask yourself:

• Could this have been Prevented?                   
• What could we do to be better Prepared if something like this were to happen again?
• How could we have troubleshot faster?
• How could we have gotten to “Repaired” faster?

What Now?

One of those 6 tips “resonated” with you…. with your network.  Which one?  What are your thoughts? Your ideas?  You can make a difference.  You can help protect the packets in your network. So go do it!  🙂

1. Be Methodical
2. Know What is Normal (Knowledge is Key)
3. Get to the “Crime Scene” as Fast as You Can
4. Have “Crime Scene Maps”  that Help and don’t Hinder
5. Let the Clues and Evidence Guide You
6. Learn and Improve

*This blog has been updated from the original.

Categories: Network Detective