A colleague and I have been debating over a few months the topic of troubleshooting. My initial stance on the topic was that you CAN teach troubleshooting. However, as time has passed working with particular individuals I have came to a realization that you CANNOT teach troubleshooting. My belief now is that some people simply have a mind that thinks in logical sequences to rule out options and others don’t. While I’m sure this post may cause some heat towards myself the point isn’t to say anything bad about anyone. My point is that some individuals have a very effective troubleshooting skill set, while others simply have a set rubric of tests that if exhausted, results in a complete halt in process. I’ve come up with four items that impact an individuals ability to troubleshoot effectively even after the initial checklist has been exhausted.
Permanent link to this article: https://www.packetpilot.com/you-coach-not-teach-troubleshooting/
I’ve been working on doing some multicast labs lately and am constantly resetting my lab devices to their default configs and starting from scratch. As many of us know, to enable PIM on all of your interfaces you must go into each interface and enable it manually. There is no default command to enable PIM on all interfaces. We know PIM should be enabled 1 to 1 with interfaces involved in routing making this a boon. With that in mind, and the fact that I am rather comfortable with the concept of needing PIM on the interfaces, and likely speak and type this command in my sleep, I decided to make it easier and modify a previous TCL script I had written to enable PIM on every interface that has an IP address assigned to it. With the great “Send to Chat” feature of SecureCRT I can do this across my entire topology on one fell swoop. In a real world environment, you could use a tool like Solarwinds to push this out to your devices.
Permanent link to this article: https://www.packetpilot.com/bulk-enable-pim-via-tcl/
The other night while studying I tweeted out the first six words that came to mind as I thought to myself “How can I retain this better”. The concept is really quite simple and I find myself doing these steps for everything I am trying to learn and retain. While everyone learns *best* a certain way, everyone also learns more by being around the material more. I don’t feel I need to explain how to find what way of learning works best for you. Nor am I going to pretend to have a clue how people learn or retain better in any sort of scientific way. I’m simply going to explain my view on my six word study method.
Read, watch, listen, learn, discuss, do!
It’s a simple mantra that you can think about every time you start studying a topic. In fact, you already learn all of these ways throughout the day. The concept here is to apply all six of them to every topic you are focused on studying.
Permanent link to this article: https://www.packetpilot.com/daily-learning-actions/
Windows for Network Engineers
Part two in my series of apps for network engineers across the three major platforms. I previously did the post for Mac when I first refreshed my laptop and purchased my first new Mac in 8 years. Issued by work, my daily laptop is a Windows machine which is fine with me. I would prefer to use Mac but give me a machine that has the tools I need and I’m fine. So with that in mind, I am going to list my favorite Windows tools for Network Engineers.
Permanent link to this article: https://www.packetpilot.com/apps-for-a-network-engineer-part-ii-windows/
Being the geek I am, I was troubleshooting an issue one day where I wanted to see what dial-peer a certain digit string was hitting on a gateway. I saw something that simply intrigued my mind that took focus away until I figured it out. This “thing” I saw was irrelevant to what I was troubleshooting but once the problem was resolved I simply had to figure it out. I’m sure many of you experience that same thing throughout your days. So, this is what I’ve discovered. *Please note that this is not an official explanation of the item/output, it is simply my interpretation from time spent figuring out what it really meant.
Permanent link to this article: https://www.packetpilot.com/what-is-that-timestamp-dial-peers-last-setupdisconnect-time/
[notice]For this example I use the following IP scheme 192.168.0.12|–O–18.104.22.168—–[INTERNET]——22.214.171.124–O–|10.42.42.42[/notice]
I was approached a few weeks back to assist in creating a VPN Tunnel between two end points. Of course in my naivety I readily assumed it was between to Cisco devices but that turned out not to be the case. The tunnel was to be between a Linux box (in this case Ubuntu on a hosted VPS provider) and an unknown endpoint. This tunnel was going to be host to host as opposed to LAN to LAN. After some quick discovery work, getting access to the Linux box, and seeing the required proposal from the other side I started diving into the unknown of Openswan. Luckily, after doing som research for the configuration and verification things started shaping up and much to my approval, a lot of what you would look for in Cisco verification was the same on the Linux box. The configuration goes as such.
Naturally the first step is to install Openswan. As per usual use your distributions software management to install this. The first thing I configured was the ipsec configuration file. On the Ubuntu box this resided in “/etc/ipsec.conf”. The configuration was as follows.
Permanent link to this article: https://www.packetpilot.com/linux-to-cisco-openswan-ipsec-configuration/
As I began to study MPLS L3VPNs I was excited to start flinging my fingers around the keyboard. However, I ran into a little snafu during my learning. All of the videos and configuration example I was finding didn’t separate the difference between the Route Distinguisher (RD) and the Route Targets. Most of the examples simply matched the RD to the Route Targets and/or used the same Route Targets for both import and export. This left me feeling like I wasn’t really understanding what those commands and numbers were accomplishing. I decided to make a visual representation to make it easier to understand.
To make this concept easier to understand we first need to know that the RD does not dictate what routes a route will import or export into it’s PE-CE routing process. The purpose of the RD so to add an additional label to prefixes so overlaps can be inserted in the BGP table and shared amongst the various PE routers. For example, my RD of 65000:8 indicates any routers in the BGP table from my customer vrf would indicate a prefix of 10.20.30.40 as 65000:8:10.20.30.40. This means if another vrf with a different RD of 4242:42 could also install 10.20.30.40 in the providers BGP table as 4242:42:10.20.30.40.
Now that we are clear on the use of the RD we can move onto the Route Targets. There are two route targets we define in our VRF policy. The import and export targets. Many examples and videos show these as the same (which is a perfectly valid configuration) often times matching the RD. To clarify exactly what they are used for I have used three different Router Targets. I am going to correlate their indicators with colors to make the example easier to visualize.
Routes exported from the headquarters use 30:8 which we will call the “Blue Routes”
Routes exported from Branch 1 will use 10:8 which we will call the “Red Router”
Routes exported from Branch 2 will use 20:8 which we will call the “Green Routes”
This exporting is done by the PE routers connecting to the CE routers. The CE routers in this example our peering via eBGP with the PE routers inside of a VRF. The VRF configuration on the PE routers is what indicates the Router Target identifier to export. At this point we can write a policy of which routers should be allowed into the individual CE routes using the VRF Route Target import. Lets follow a case from the HQ to Branch 1.
HQ CE peers with its PE router which has a VRF policy stating to export its routes as the color Blue. These routes are passed around to the other PE routers. When the Branch 1 PE peer receives the routes it sees that it’s VRF policy is stating to export its routes as the color Red as well as import any routes that are colored Blue. Back at the headquarters we have our VRF policy set to import both the Red and Green routes. Branch 2 does the same as Branch 1 but swapping out Red for Green.
By writing the VRF policies this way we have created a Branch to HQ connection while not passing routes Branch to Branch. In my diagram I show the routes coming into the CE routes as it is the ultimate end goal however, please keep in mind that the VRF configuration is done on the PE routes.
I hope that by using simple colors for the routes it has simplified the reasons we use the RD, and the import and export Route Target. I found it difficult to understand the true use of these configuration when they were using the same value for the RD as well as the import and export Route Targets.
Permanent link to this article: https://www.packetpilot.com/route-targets-explained/
This post is primarily for archived documentation for myself in the event of possible future need. However, I was having a hard time finding any configuration examples for this task. The steps below are those I used to connect a 70inch Infocus Mondopad to our CUCM environment to be able to make video calls including sharing the Mondopads screen over the called Cisco phone.
Permanent link to this article: https://www.packetpilot.com/infocus-mondopad-registered-with-cucm-sip/
The Scenario goes like this: A Synapps – SA Announce paging and messaging server integrated with Cisco’s CUCM hosting around 30 phone to phone paging groups. The paging had been working fine for months and out of no where one of thirty particular groups was putting in multiple trouble tickets over multiple days that the paging isn’t working.
So begins the troubleshooting and diagnosis. My first action was to monitor the paging server as it has a real time display of who is calling a paging group and which group they are calling at in given time. When I was monitoring this I could see multiple people calling multiple groups including the one in question. So this brings up one of those “what gives” questions. Are they just doing something wrong up in the area. Time to take a trip and raise that pedometer count.
I arrive in the area and try and locate and area where I can visually see and hear multiple phones. Easier said than done but in this case I was the only one available to work on the issue and knowing that the paging server will activate the speakerphone and mute lights when a group the phone is a member of is called this was my best bet and understanding what was going on. After making my first test page I can see that lights on the phones I can see are immediately lighting up, however I can’t hear audio. As I stand there dumbfounded with the phone still off hook all of a sudden the audio starts picking up the background noise. However, paging shouldn’t have a 6 second delay before you can start talking. Six seconds is a long time to wait after hitting page to start talking. So whats going on? It’s only one group experiencing this. What is different about their group? Time for a deep dive in the diagnostics world. Enter Wireshark.
Permanent link to this article: https://www.packetpilot.com/it-was-broke-but-the-sharks-got-to-it/
This article is another example of trouble shooting by putting multiple pieces together. While it relies upon existing knowledge of the environment in which the article is based it should prove to be a good example of a trouble shooting process that will hopefully be able to spark some creative thinking the next time you have a problem that needs to be resolved.
The scenario starts out with a user ticket stating that the phone isn’t working. After some fact gathering the below details and possible solutions were outlined.
Permanent link to this article: https://www.packetpilot.com/trouble-shoot-with-tdr/