This article is another example of trouble shooting by putting multiple pieces together. While it relies upon existing knowledge of the environment in which the article is based it should prove to be a good example of a trouble shooting process that will hopefully be able to spark some creative thinking the next time you have a problem that needs to be resolved.
The scenario starts out with a user ticket stating that the phone isn’t working. After some fact gathering the below details and possible solutions were outlined.
Symptoms:
- Phone Powered Up
- Phone states “Ethernet Disconnected”
- Laptop in same port gets IP address
Possible Solutions:
- Bad Network Drop
- Bad Patch Cable(s)
- Bad Port on Phone
The first thing I wanted to look at was the port in question. Luckily, the desk side technician had plugged his laptop into the port to verify it was working correctly. Since ip device tracking was enabled I could find the port he was using.
Switch#show ip device tracking all | i 10.128.26.100 10.128.26.100 6cc2.177e.e4f1 2113 GigabitEthernet2/0/7 30 INACTIVE ARP
At this point I know what port we are dealing with. With the fact gathering prior we know that the port is giving out power as the phones display is lit and it states Ethernet disconnected. I then looked at the power output on various ports including the suspect port.
Switch#show power inline | Module Available Used Remaining (Watts) (Watts) (Watts) ------ --------- -------- --------- 1 1440.0 138.8 1301.2 2 780.0 59.0 721.0 3 1440.0 196.0 1244.0 4 766.0 67.6 698.4 Interface Admin Oper Power Device Class Max (Watts) --------- ------ ---------- ------- ------------------- ----- ---- Gi1/0/1 auto off 0.0 n/a n/a 30.0 Gi1/0/3 auto off 0.0 n/a n/a 30.0 Gi1/0/6 static on 20.0 Ieee PD 0 20.0 Gi1/0/7 static on 20.0 Ieee PD 0 20.0 Gi1/0/8 auto off 0.0 n/a n/a 30.0 Gi1/0/9 auto off 0.0 n/a n/a 30.0 Gi1/0/10 auto off 0.0 n/a n/a 30.0 Gi1/0/12 auto off 0.0 n/a n/a 30.0 Gi1/0/13 auto off 0.0 n/a n/a 30.0 Gi1/0/14 auto off 0.0 n/a n/a 30.0 Gi1/0/15 static on 20.0 Ieee PD 0 20.0 !----Lines Omitted for Brevity Module Available Used Remaining (Watts) (Watts) (Watts) ------ --------- -------- --------- 1 1440.0 138.8 1301.2 2 780.0 59.0 721.0 3 1440.0 196.0 1244.0 4 766.0 67.6 698.4 Interface Admin Oper Power Device Class Max (Watts) --------- ------ ---------- ------- ------------------- ----- ---- Gi2/0/1 auto off 0.0 n/a n/a 30.0 Gi2/0/2 auto off 0.0 n/a n/a 30.0 Gi2/0/3 auto off 0.0 n/a n/a 30.0 Gi2/0/4 auto off 0.0 n/a n/a 30.0 Gi2/0/5 auto off 0.0 n/a n/a 30.0 Gi2/0/6 auto off 0.0 n/a n/a 30.0 Gi2/0/7 auto on 4.0 Ieee PD 1 30.0 Gi2/0/20 auto off 0.0 n/a n/a 30.0 Gi2/0/21 auto off 0.0 n/a n/a 30.0 Gi2/0/22 auto on 3.5 IP Phone 7861 1 30.0 Gi2/0/23 auto on 15.4 AIR-CAP3602I-A-K9 4 30.0 Gi2/0/24 auto off 0.0 n/a n/a 30.0 Gi2/0/25 auto off 0.0 n/a n/a 30.0 Gi2/0/26 auto off 0.0 n/a n/a 30.0
Knowing all of our known POE device are Cisco APs and Phone notice the 2/0/7 Port has an Ieee PD listed. This indicates that the phone is not communicating CDP info to the switch. It is also receiving a similar about of power (4) when compared to a known working phone of the same model (3.5) From here I decided to use a largely unknown diagnostics command on the port called TDR.
Switch#test cable-diagnostics tdr interface gi 2/0/7 TDR test started on interface Gi2/0/7 A TDR test can take a few seconds to run on an interface Use 'show cable-diagnostics tdr' to read the TDR results.
[notice]Mixed documentation and research indicated running this MAY cause interruptions on the device[/notice]
Switch#show cable-diagnostics tdr interface g 2/0/7 TDR test last run on: May 07 14:51:01 Interface Speed Local pair Pair length Remote pair Pair status --------- ----- ---------- ------------------ ----------- -------------------- Gi2/0/7 100M Pair A 79 +/- 1 meters N/A Open Pair B 0 +/- 1 meters N/A Normal Pair C 78 +/- 1 meters N/A Short Pair D 78 +/- 1 meters N/A Short
Notice how the port is listing at 100M for a speed in the TDR diagnostics but if you look at the interface it is set to auto. This plays a part in how we interpret the above diagnostics.
Switch#show int gi2/0/7 GigabitEthernet2/0/7 is down, line protocol is down (notconnect) Hardware is Gigabit Ethernet, address is 6c99.89ee.8007 (bia 6c99.89ee.8007) MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Auto-duplex, Auto-speed, media type is 10/100/1000BaseTX input flow-control is off, output flow-control is unsupported ARP type: ARPA, ARP Timeout 04:00:00 Last input 00:36:14, output never, output hang never Last clearing of "show interface" counters never Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: Class-based queueing Output queue: 0/40 (size/max) 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec 223444607 packets input, 22699473542 bytes, 0 no buffer Received 2044391 broadcasts (1910506 multicasts) 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 watchdog, 1910506 multicast, 0 pause input 0 input packets with dribble condition detected 1216195746 packets output, 224071930018 bytes, 0 underruns 0 output errors, 0 collisions, 1 interface resets 112574 unknown protocol drops 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier, 0 pause output 0 output buffer failures, 0 output buffers swapped out
Putting the facts together:
- Port works with laptop
- Ethernet Disconnected error with phone
- Power lists Ieee PD instead of phone model
- TDR Cable Diagnostics list an Open Pair
- Opens and Shorts are listed at ~79 Meters and one Pair terminating correctly
- Indicated by 0 meters listed on the good pair
- Assuming the opens/shorts all at ~79 meters the End to End cable is 79 meters.
- This concludes that the cable either has a bad termination at the end whether the patch cable to the phone, or the port on the phone itself.
With this in mind we can safely assume that it isn’t the port on the switch, the patch from switch to the patch panel, or the cable in the wall. In this particular case it was a bad port on the phone. However, with a few quick commands we saved time of having to walk across campus to play the trial and error troubleshooting method and could prove the network drop itself was good up to the termination/port on the end device.
Once the phone was swapped out with a working phone the TDR diagnostics were re-run showing the A and B pairs successfully being in a Normal State
Switch#test cable-diagnostics tdr interface g2/0/7 Link state may be affected during TDR test TDR test started on interface Gi2/0/7 A TDR test can take a few seconds to run on an interface Use 'show cable-diagnostics tdr' to read the TDR results. Switch#show cable-diagnostics tdr int g2/0/7 TDR test last run on: May 07 16:11:39 Interface Speed Local pair Pair length Remote pair Pair status --------- ----- ---------- ------------------ ----------- -------------------- Gi2/0/7 100M Pair A 0 +/- 1 meters Pair B Normal Pair B 0 +/- 1 meters Pair A Normal Pair C 78 +/- 1 meters N/A Short Pair D 78 +/- 1 meters N/A Short
These troubleshooting steps require an understanding and inside knowledge of the working environment but can be useful in many other situations. It took some research in the Cisco Documentation and other blogs to piece together how to properly read the TDR Diagnostics output. I have an article here that describes my findings.
