Web View      Printer Friendly

When you print this document, the only thing visible will be the page. You can print to a PDF or printer paper. This is not really a mobile viewing friendly page.

ROUTED BYTES

About

This is your step by step troubleshooting checklist to make sure you can spot the issue, no matter where it is. Knowing the OSI model and how it works, better helps your find issues quicker, and more reliabily.


Table of Contents:

Pg 1 - Intro and simple table.

Pg 2 - OSI Layer 1 [Physical]

Pg 3 - OSI Layer 2 [Data Link]

Pg 5 - OSI Layer 3 [Network]

Pg 6 - OSI Layer 4 [Transport]

Pg 7 - OSI Layer 5 [Session]

Pg 8 - OSI Layer 6 [Presentation]

Pg 9 - OSI Layer 7 [Application]


WWW.ROUTEDBYTES.COM
ROUTED BYTES

1 Physical
bits

Check these items first if you cannnot get the device to connect or if you can't seem to get a signal at the device. This is the layer where physical connections happen.

  • Check ethernet cable. Make sure the connector is plugged in all the way, put a tester on it. Use the Cisco TDR function on the cable.
    Cisco IOS
    switch# test cable-diagnostics tdr interface [interface - ex Tw1/0/1]
                                switch# show cable-diagnostics tdr interface [interface - ex Tw1/0/1]
  • Check the fiber cable. Make sure there's no visible damage. Never look into the fiber cable, as it can damage your eyes.
  • If you have optics with diagnostic capabilities, check to make sure your readings look good on the optics. Make sure you're sending and receiving light in good levels and using supported optics.
    Cisco IOS
    switch# show interface [interface - ex Te1/1/8] transceiver detail
  • Don't mix fiber types (OS1, OS2, OM2, OM3, etc). Don't mix single mode or multi mode optics and fiber.
  • Check for the correct fiber polarity, if you're not using a BIDI optic. There are arrows on the top of the optic for the direction light is travelling.
  • Make sure the interface is active and not disabled, the device has power, and there are no speed or duplex miss-matches.
    Cisco IOS
    switch# show interface [interface] status ! works on Nexus
                                switch# show power inline [interface]
  • Use CDP neighbors if this is a Cisco device to see if it's seen. LLDP for non-cisco devices.
    Cisco IOS / Cisco Nexus
    switch# show cdp neighbors
                                switch# show lldp neighbors

WWW.ROUTEDBYTES.COM
ROUTED BYTES

2 Data Link
frames

Check the following if a device cannot get basic communication on a network.

  • Show the MAC Address table to check for the MAC.
    Cisco IOS / Cisco Nexus
    switch# show mac address-table
  • Check port configuration, on if it should be access or trunk.
    Cisco IOS / Cisco Nexus
    switch# show run interface [interface]
  • Check VLANs for trunks or access. Check native VLAN miss-matches.
  • Check the port-channel or etherchannel if it should be in it or not.
    Cisco IOS
    switch# show etherchannel summary
  • Check Spanning Tree and Root Bridge Election to make sure no VLANs getting blocked.
    Cisco IOS
    switch# show spanning-tree ! shows per vlan view for each ports state
                                switch# show spanning-tree summary ! show a simple spanning tree summary
                                switch# show spanning-tree blockedports ! shows ports blocked by STP
                                switch# show spanning-tree root ! this will show the root host
  • Check BPDU guard.
  • Port security config.
    Cisco IOS
    switch# show port-security
                                switch# show port-security interface
  • MAC filtering or blocking.
  • Is ISE microsegmentation blocking something?
    Cisco IOS
    switch# show auth session int [interface] detail
  • Check ARP tables.
WWW.ROUTEDBYTES.COM
ROUTED BYTES

2 Data Link
frames
  • Check for MTU miss-matches between the links.
    Cisco IOS / Cisco Nexus
    switch# show interface [interface] | i MTU
  • CAPWAP connectivity between AP and WLC.
  • Check Wireless SSID-to-VLAN mappings.
  • Check 802.1X
    Cisco IOS
    switch# show auth session int [interface] detail
WWW.ROUTEDBYTES.COM
ROUTED BYTES

3 Network
packets

Having issuse routing IPs and getting to various subnets? Check the following.

  • Check IPs and Subnet configurations on the interface of the router. Make sure If it's a P2P link that the other IP on the neighbor interface is in the same subnet.
    Cisco IOS / Cisco Nexus
    router# show ip interface brief ! shows interfaces and IPs
                            router# show run interface [interface] ! shows config of a port
  • Check neighbors and routing of various protocols that you may use.
    Cisco IOS / Cisco Nexus
    router# show ip route ! shows the routing table
                            router# show ip bgp ! shows BGP routing instance
                            router# show ip ospf ! shows OSPF routing instance
                            router# show run | section router ! shows routing config
  • If you have multiple routing protocols, check route redistribution.
    Cisco IOS / Cisco Nexus
    router# show run | section router ! shows routing config
  • Check default gateways. Make sure you have one set on your computer or device if needed, or that it's bring advertised.
    Cisco IOS / Cisco Nexus
    router# show ip route ! look for gateway of last resort
  • If working on a client device, ensure you're getting an IP from the DHCP server. 
    Windows -  ipconfig /all
                              MacOS   -  ifconfig
                              Linux   -  ip addr sh
  • Check route filtering to make sure you're not dropping any routes.
  • Check VRF configurations and that you're routing in the correct VRF.
    Cisco IOS / Cisco Nexus
    router# show vrf ! shows all vrf's
                            router# show run int [interface] ! shows port config to look for vrf
  • Check NAT translations if you're natting traffic.
  • Check ACLs on the router, switches, or if you're using ISE, the DACLs.
    Cisco IOS / Cisco Nexus
    router# show run ! parse through the config and look for mappings
WWW.ROUTEDBYTES.COM
ROUTED BYTES

4 Transport
segments

This is how your data gets from point A to Z and what all happens to the packes.

  • Check to make sure the port is open at the device from a remote device.
    Shell
    nmap [device ip] # This will test all ports
                            nmap -p [port] [device ip] # This will test a certain port
  • On the device the service is hosted, check local firewalls, ACLs or anything on the device that may be bocking the traffic.
  • Check the network for packet loss.
  • On the device, ensure the service is running and reachable locally.
  • Check firewall rules for firewalls sitting inbetween the user and the device itself. Not the firewalls on the device, but anything inbetween that may be blocking it like a NGFW.
  • Use Wireshark to do a packet capture at the device to ensure traffic is actually being exchanged, and look for any weird malformed packets.
    Cisco IOS / Cisco Nexus
     ! This filter will reduce a lot of noise.
                            !arp && !lldp && !cdp && !stp
  • Check MTU or fragmentation errors on the device.
    Cisco IOS / Cisco Nexus
    Windows -  ping -f -l [mtu] [device ip]
                            MacOS   -  ping -D -s [mtu] [device ip]
                            Linux   -  ping -M do -s [mtu] [device ip]
  • Confirm MTU settings of the network.
    Cisco IOS / Cisco Nexus
    Switch# show int | i MTU
WWW.ROUTEDBYTES.COM
ROUTED BYTES

5 Session
data

You figured out traffic can make it to point B, but is the session being established and maintained?

  • Wireshark is your best friend. Take a peek to see if you see application/session traffic being exchanged between the computer/user and the host.
    Cisco IOS / Cisco Nexus
     
                            !arp && !lldp && !cdp && !stp # Reduces network noise
                            tcp.port == 80 || tcp.port == 443 # Shows web traffic
                            icmp || dns || dhcp # Shows network traffic
  • Check authentication logs. Are you authenticating correctly, either with username and password, or certificate based authentication.
  • Check VPN or traffic between the point of entry and the application server iteself if there is something inbetween that is having trouble maintaining or establisihing a connection.
  • Check load balancers to make sure traffic is being pointed to the applications correctly.
  • Check CAPWAP tunnel issues.
    Cisco IOS / Cisco Nexus
     
                            AP# sh capwap client configuration ! Shows CAPWAP info
                            AP# sh capwap client led statistics ! Shows LED info
                            AP# sh flexconnect client ! Shows detail client info
                            AP# sh dot11 clients ! Shows simple client info
                            AP# sh dot11 wlan ! Shows broadcasted WLANs
  • Check API logs.
  • Check TLS/SSL negotiations.
WWW.ROUTEDBYTES.COM
ROUTED BYTES

6 Presentation
data

You've confirmed traffic is making it to A to B, and sessions are being established correctly, but is information being formatted and decrypted properly.

  • Check SSL certificates to make sure they're within their validity period. (OPENSSL)
  • Check certificate trust chains. Is the root or signing certificate in a trusted root store on the end user devices.
  • Check encryption and decryption settings for the site to make sure they match and compatible so information can decrypt properly.
  • Use browser developer tools to ensure information is being loaded properly
  • Check and verify TLS versions.
WWW.ROUTEDBYTES.COM
ROUTED BYTES

7 Application
data

Everything has been confirmed as working, except for the application itself. Information is making it to the user, but something isn't functioning correctly on the app.

  • Check backend DNS resolution.
  • Check DNS servers to make sure they are up and sending responses.
    Bash
    nslookup [ip or hostname]
  • Check web server and database availability, and configuration.
  • Check Active Directory and LDAP connections to ensure connectivity.
  • Check application logs.
  • Check the reverse proxy.
  • Check Web Application Firewalls.
WWW.ROUTEDBYTES.COM