General Network Challenges

For some of us, building a network from the ground up is a huge challenge. Understand, once we have that network built we could have some general network challenges that would happen just due to the way the equipment is designed to operate. In this article, I will demonstrate some of the general challenges that networks face today.

Asymmetric Routing

Things might seem perfectly happy and healthy in your network, until you realize the traffic is going out on one path and coming back on the other. This is a common challenge that a network could face after a perfectly redundant design; we call this asymmetric routing. You might wonder why that will be an issue as long as the traffic makes it back. Well, what if there is a firewall we have constructed on one path? stateful firewalls want to dynamically see the return traffic and punch a hole to allow the traffic back. Obviously, asymmetric routing will cause an issue

I was personally challenged many times with asymmetric routing. I used a number of techniques to thwart the issue. The very recent issue I had with asymmetric routing is seen on (image 1) the traffic to 66.211.181.161 is clearly going out but never gets a response back because there is a firewall that is receiving the packet on the other end that doesn’t know what to do with that information. However, when I only use one router on my environment (image 2) the traffic goes out and gets back from source to destination.

Unicast Flooding

Unicast Flooding happens in a scenario where a switch takes in traffic and looks at the destination address that is a layer 2 address in that particular frame and for whatever reason if the switch does not know where to send the packet unicast flooding will occur, I think you already guessed how the switch will respond to that particular frame, It basically will send it out to every single port except the one on which the frame came in on. This is called a Unicast Flood. The reason for flooding might vary. However, it is mainly caused by one of the following reasons.

The first one is, asymmetric routing. If you have a situation where outbound traffic is taking one path and inbound traffic is taking another path, switches along the way will not have their MAC address table populated with a particular device thus the switches will flood traffic when it is destined for those particular devices

Another reason we might see for excessive unicast flooding is with Spanning Tree Protocol (STP) topology changes, particularly with Topology Change Notifications (TCNs). TCNs are designed to correct forwarding tables after forwarding tables has changed or after the forwarding topology has gone through a topology change. In an environment where we have excessive TCNs we might get unicast flooding due to the instability it has with our overall topology thanks to STP. A great solution for this is to eliminate TCNs in our environment by configuring PortFast. Most people uses PortFast to make sure DHCP works properly and to bring the port up as quickly as possible so the system can get the DHCP address, and yes that is one of its function too. But aside from bringing the port up quickly PortFast is used in many network environment to stop unnecessary TCNs when work stations are powered on and off throughout the day.

The third scenario we can run in to with unicast flooding is when there is a full MAC table. When the MAC table of a switch is completely full no new destination MAC addresses can be learnt, we will deal with unicast flooding as a result. But it is pretty rare. It usually happens as a result of some type of security attack. To guard the MAC table we use techniques like port security, blocking a specific MAC addresses from communicating in our environment.

Cisco has responded to some of the unknown unicast and multicast flooding and the problems they may cause, click here to read more.

Microbursts

Microbursts happen for a very short period of time, very likely we are dealing with micro seconds here. Where there is a burst of network traffic that occurs significantly higher than normal as a result it will overwhelm the capacity of that link or that system for that short period of time.

The impact of having microburst in a network varies but mainly it can cause a delayed or dropped packet or frames depending on where it happens, if you are looking at an application that goes from point A to point B, this can be a traffic within virtual environment or travelling across physical or virtual network or both it is the weakest link in that path that will cause a delay. This delay might not just affect that device we can have congestion on downstream systems as well. For example if a device is causing a bottleneck on a system all traffic going to that direction will also be delayed causing the congestion on those downstream systems. It also can mess up any Service Level Agreements (SLA), which is an agreement between business entities to deliver certain types of traffic or a certain processes in a certain amount of time. A microburst can delay that information or transaction from happening and also in some cases it might causes an application failure as well.

Out-of-order packets

These are simply the packets that don’t shows up in a same order they were sent. For example, we have two systems, System A and System B, and they send packets back and forth to each other. Let’s say we have multiple path between them, if System A sends three packets 1, 2, and 3 and System B receives them in 2, 1, and 3 order, we refer to that as out-of-order packets. I am sure you are wondering what can cause something that can result in this order.

The number one reason for out of order is multiple paths, either in layer 2 or layer 3, and if the same paths are not being used each and every time. Depending on the actual throughput and how congested the paths are, as well as the setting of different services of QOS on one router versus the other, perhaps a packet being sent on a one path, based on what type of traffic it is and what QOS is in place, gets delayed and put in queue causing it to get there after one of the earlier packet that was sent that went across a different path. So the big deal about packets arriving out of order to their destinations is that, depending on the applications arriving out of order may cause performance degradation or a complete failure. We have number of solution to tackle this challenge.

Session Based Path Selection, the conversation for a given session is set to always go through the same path on a network. This can be accomplished by tweaking the CEF forwarding table so that the same session can flow over the same path. We also have similar control in etherchannel to control which of the physical links will be used for various packets or sessions as they go across that etherchannel.

TCP Reassembly, if we are using TCP based application, TCP has capability of reassembly built in to it thus TCP can either say through the acknowledgement process “hey send that again” or it can identify the incorrect packets and based on the TCP headers has a potential of reordering them before it passes them up to the protocol stack.

Permit on Intermediate Device, if you are using intermediate devices like firewalls that are doing stateful filtering they are paying attention to those packets that are going through so we might want to tell those devices that we are “ok” with asymmetric routing. That way they don’t have to see every single packet in a session. The same concept holds here for packets that are received out of order.