Switch Security – DHCP Snooping, IP Source Guard and Dynamic ARP Inspection

A few years back, I was designing an Enterprise MDM strategy for iPhone deployments. At some point, I was browsing the Apple Enterprise iOS forum and saw a post about an iOS bug where devices sometimes retain a previously assigned IP (from another network) when joining a new network. In this case, someone was joining a network with a retained IP which coincided with the organization’s e-mail server. When this occurred, it apparently disabled e-mail service throughout the organization. From a network engineering standpoint, that sounds impossible if the network was designed correctly. I was trying to think of how that could possibly happen, and the only thing I could think of is that the email server resides on the same subnet as the clients and the client was responding to ARP requests for the e-mail servers IP… Sounds like a giant mess.

At the same time, I was studying for the CCNP SWITCH exam reading about DHCP Snooping, IP Source Guard and Dynamic ARP Inspection. Although these features were intended to mitigate spoofing attacks, they could also help prevent this supposed iOS bug from affecting network services. Whether malicious or accidental, spoofing is usually not a good thing.

DHCP Snooping

To better understand DHCP Snoopings purpose, it is important to understand what is possible without it. Under normal circumstances, an attacker could easily implement a rogue DHCP server. When other clients in the same VLAN first plug in and send a DHCP Request, they would receive and might accept a DHCP Offer sent by this rogue DHCP server. This rogue server might set the client default gateway to the attacker’s IP, causing all traffic destined for a foreign subnet to be sent to the attacker; a successful Man-in-the-Middle attack.

DHCP Snooping gives a Cisco switch the ability to control where a DHCP Reply can come from. Any DHCP server traffic such as a Reply, ACK or NACK is only permitted from a trusted port. On untrusted ports, only DHCP Requests are permitted. By default, all ports are untrusted therefore trusts must be configured manually. Basically, a trusted port is where a legitimate DHCP server resides or where legitimate DHCP Reply’s would be received from, therefore the traffic should be permitted. If ingress DHCP server traffic is intercepted on an untrusted port, the port is placed into the err-disabled state.

Typically, the trust boundary lies where end-users connect, therefore DHCP Snooping should be enabled at the access layer. Trusts should be configured only on links where a DHCP Reply would be expected such as uplinks to the distribution layer or the port where a local DHCP server resides.

In addition to permitting/denying DHCP server traffic, DHCP Snooping also keeps track of when clients receive a successful DHCP binding. It records information such as the IP address assignment, the lease time, and the requester’s MAC address as well as the port on which the request was received. This information is then stored and can be utilized by other technologies such as IPSG and DAI.

Dynamic ARP Inspection

Dynamic ARP Inspection provides a method to protect the integrity of layer-2 ARP transactions. DAI leverages the DHCP Snooping database to validate the integrity of ARP traffic. ARP is used when a host has an IP address and wants to determine the MAC address. As an example, if a client sends an ARP request for the default gateway, an attacker could potentially reply to the request with its own MAC address, causing the client to send traffic destined for the default gateway IP to the attacker. This type of attack is known as ARP Spoofing/Poisoning and DAI can potentially mitigate it’s threat by validating the integrity of the ARP traffic against known good bindings.

Like DHCP Snooping, DAI also utilizes the concept of trusted ports. On trusted ports, no ARP Inspection is performed. By default, all ports are considered untrusted, therefore trusted ports must be manually configured. Like DHCP Snooping, the trust boundary should lie at host-connected ports, therefore DAI should be enabled at the access layer. Trusts should only be configured on links to other switches and the distribution layer.

DAI inspects ARP traffic and verifies the source MAC and IP against the known trusted values in the Snooping database. For example, if in the database the MAC 111.222.333 is bound to, the host connected to this port will only be allowed to respond to ARP Requests for ARP Reply’s found to be invalid are dropped and logged. Hosts utilizing static IP’s must be manually configured and permitted via an ARP ACL. This ACL must then be added to a DAI filter in order for it to recognize the bindings. Optionally, DAI can be configured to ignore the DHCP Snooping database and utilize only the ARP ACL for increased security. DAI is enabled on a per-VLAN basis.

IP Source Guard

IP Source Guard can help ensure that hosts utilize only their assigned IP addresses. IPSG can leverage the information in the DHCP Snooping database to dynamically create Port ACL’s permitting only Layer-3 IP traffic which has a source matching the port-IP binding in the DHCP Snooping binding database. This prevents a host from transmitting using a source IP differing from that which it was assigned via DHCP. The IPSG PACL also includes a VLAN binding, ensuring that the IP can only be used on its respective VLAN. IPSG can optionally verify the source MAC as well as the source IP for added protection by leveraging the port-security feature. When enabled, this ensures that the source MAC is associated with the source IP in the Snooping database. For statically configured hosts, a static binding must be manually configured to permit traffic flow. IP Source Guard is enabled on a per-port basis.

Lab Scenario


Configuring DHCP Snooping

First, we will enable and configure DHCP Snooping since it makes our lives easier with Dynamic ARP Inspection and IP Source Guard. In addition, we will configure Trust’s for the interfaces on which a DHCP Reply is expected. I have set up DHCP servers in a split-scope fashion on both 2821 and the 1841. This means that the paths to both the 1841 and 2821 must be trusted by DHCP, meaning fa0/48 on both switches as well as their interconnecting port-channel po1 which is a LACP-negotiated EtherChannel.

Remember, configuration of an port-channel interface automatically does the same on all port-channel member interfaces, in this case fa0/1 and fa0/2. It is not necessary to manually enter the trust configuration on individual ports which are members of a port-channel.

Above, you can see that DHCP snooping must be enabled globally and then on a per-VLAN basis. In this case, things are simple and there is only one VLAN. Multiple VLANs can be configured concurrently by using a VLAN list such as 1,5,7-10.

Verification of the DHCP Snooping Configuration

Now I will connect a client and record what happens using “debug ip dhcp snooping event”. The client is not getting an IP address. As you can see below, the DHCP packet is being received, relay information is being added, and then it is being sent out other active ports.


Seeing only the switch side, our understanding of the situation is precarious at best. In the final step, we can see that the packet is in fact being sent out, so let’s check what is getting to the DHCP server.


Aha! Debugging is a beautiful feature. From here, we can see that there is a problem with the relay information. By default, a DHCP Snooping adds Relay Information, or DHCP Option 82. My understanding from the above output is that the switch intercepts DHCP Requests and injects it’s relay information into it prior to forwarding it on. The problem here seems to be that it sets the “giaddress” to a null value, which would normally be the DHCP Relay’s IP address. Under normal circumstances, a DHCP Relay is used when a subnet does not locally contain a DHCP server which could overhear the layer-2 broadcast DHCP Requests. A DHCP relay acts as a relay for subnets without a DHCP server inside the layer-2 boundary, listening for DHCP Requests and relaying them past layer-2 boundaries to a configured DHCP server. When enabling DHCP snooping on a VLAN which has a directly connected DHCP server, the relay process is not necessary and the null giaddr value confuses the DHCP server which thinks the request came from a relay, not directly from the requester.

By default, the relay function is enabled. It is turned off by disabling the information option.

Now lets see what happens.


As you can see, the client now received its IP, and the switch recorded the transaction. It now knows the MAC, IP, Lease time, VLAN and Source Interface of the transaction. This information can now be leveraged by Dynamic ARP Inspection and IP Source Guard to help protect our network.

Configuring IP Source Guard

IP Source Guard leverages the DHCP snooping database in a powerful way. It dynamically creates Port ACL’ s based on the information in the DHCP Snooping database. It references the IP address and the associated interface to create a PACL which permits only traffic from the trusted IP. In this example, IP Source Guard creates a PACL permitting only traffic with a source address of to enter port fa0/34; all other traffic is dropped.

Enable IPSG (Per-Interface)

We can also see what the dynamic PACL contains below.

With ip verify source port-security enabled, the result is the following:

As you can see above, the mac-address field shows permit-all. IPSG relies upon Port-Security to enforce the MAC validation. In order for it to do this, port-security must be enabled on the port using switchport port-security. After adding this command, the result is the following:

As you can see, IP Source Guard is now validating both MAC and IP. It utilizes a PACL to enforce the IP, and Port-Security for MAC. It is important to note that the only reason this worked is because I turned it on AFTER the host already had a MAC and IP binding present in the DHCP Snooping database. You should probably leave the port-security function of IPSG disabled unless you have manually configured bindings. When enabled and a binding is not present, IPSG’s port-security function defaults to a deny-all MAC function making it impossible for the switch to dynamically learn MAC bindings such as via the switchport mac-address sticky function. I would suggest you configure port-security separately unless you have a specific need for this functionality.

Lets see what happens if a host plugs into a port with IPSG enabled and tries to set a static IP. As seen below, if a host plugs in and no static binding or DHCP Snooping entry exists, a deny-all PACL and deny-all MAC port-security function is configured on the port.

This configuration makes it impossible for any communication to take place on the port without manually configuring a MAC for port security. All host traffic will be dropped at the switchport because of the deny-all rules.

IPSG & Static IP’s

In the case of a host with a static IP, a static source binding must be configured, binding a MAC to a specific VLAN and IP on a specific interface. For example:


The result of configuring the static binding entry is shown below.

The ip verify source port-security function plays much nicer with static bindings. Since IPSG is configured on a port-by-port basis, utilize “ip verify source” for DHCP ports and save “ip verify source port-security” functionality for ports with statically configured hosts. This function more or less facilitates automatic port-security MAC configuration so you do not have to enter it in two places if you wish to use the feature.

Configuring Dynamic ARP Inspection

Dynamic ARP Inspection is easy to configure when running DHCP Snooping. DAI inspects ingress ARP Replies and verifies the source-address against the DHCP Snooping database. In this case, DAI would inspect ARP Replies on fa0/34 and ensure that the source information matches the MAC and IP in the DHCP snooping database.

First, lets redact the static binding from the earlier IPSG config and disable the port-security portion of its configuration since I want to demonstrate DAI using the DHCP Snooping database.


Enable DHCP Snooping on the appropriate VLAN(s), in this case, VLAN 1.

Configure trust on our links to other switches, in this case, po1.

Now, lets see the results. What happens when someone sets a static IP? Below I have set the IP of a host to When I brought it online, it cannot even resolve the MAC address of the default gateway,!

When the host attempts to perform an ARP request, DAI blocks the unknown binding since it cannot validate the source address.


Lets pretend that this static host binding is valid and we need to allow it. In order to do so, we must utilize an ARP ACL and add it to the DAI Filter to allow it.

The console cut off the rest of the line. The resulting ACL is shown below.

Now that we have created the ARP ACL, we must add it to the DAI filter.

The static keyword can be added at the end of this line to force DAI to use only the ARP ACL and not utilize the DHCP Snooping bindings. Be careful!

At this point, the statically configured host should be able to successfully perform an ARP Request for the default gateway. Lets see.

Now, lets see what happens if we change the IP address of the host to an ip different than the ARP ACL entry.

Success! The host is blocked again. This switch is blocking communication.


Finally, we can verify DAI operation. As seen below, the ACL Match shows that our STATIC_ARP_ACL is being utilized alongside the DHCP Snooping database for VLAN 1. If the filter was configured to utilize only the ACL and not the DHCP Snooping database, the Static ACL field would say Yes.


After seeing all of these features in action, it should be obvious that using DHCP to assign addresses makes implementation a lot easier. The DHCP Snooping database facilitates the collection of information required for the function of Dynamic ARP Inspection and IP Source Guard. Remember to configure DHCP Snooping Trust on interfaces where a valid DHCP Reply may be received, otherwise the DHCP Server traffic will be blocked.

Remember to configure interface trusts on ports extending to other switches for dynamic ARP inspection. Also, remember to create entries in an ARP ACL and apply it to the DAI filter if you are running hosts with static IP’s.

If you want to use IP Source Guard in combination with static IP hosts, remember to create static IP source bindings for each host otherwise the default PACL will deny all traffic.

Remember that IP Source Guard’s port-security integration feature requires port-security to be enabled on the port, otherwise it uses a permit-all function. Additionally, remember that this integration feature is easier used with static host entries than DHCP hosts, unless I missed something. This is no big deal because IPSG is implemented on a per-port basis anyway.

Important: When implementing Dynamic ARP Inspection to secure Wireless networks, make sure to set a minimum of 90 minute DHCP lease time… Wireless devices in standby mode behave erratically with regard to DHCP lease renewal, especially Apple devices. When first implementing DHCP-Snooping and DAI, definitely keep an eye on your switch logs to mitigate mass end-user complaints.


AWS – Globally Adjusting ELB SSL Policy

A while back, I had to adjust the policy of all Elastic Load Balancers in my organization to disable SSLv3 due to the POODLE exploit. This can be an error-prone task if done by hand, especially if your architecture spans multiple regions and/or more than a handful of ELBs. The nice thing about cloud architecture, is that nearly everything can be automated and/or scripted.  For that reason, I went ahead and wrote a PowerShell script to handle this.

Most other write-ups I have seen do not take into account Stickiness policies, which are also applied to listeners.  If you run the Set-ELBLoadBalancerPolicyOfListener cmdlet with only an SSL policy applied, it will remove any other existing listener policies.  It is important to check the ELB for other policies, and make sure they are reapplied.  There is logic in this script that handles that.

It is important to note that this script makes use of Amazon’s template SSL Negotation Policies, but could be adapted to make use of your own.

As of 4/9/15, you cannot simply set the ELB policy to a newer reference policy, although AWS documentation states otherwise.  For this reason, a new policy must be created which references the AWS Reference-Security-Policy of choice.  You can retrieve a list of available reference policies with the Get-ELBLoadBalancerPolicy cmdlet.

Code also available on my GitHub

# AWS Global ELB SSL Policy
# Brian Dwyer - Intelligent Digital Services - 4/5/15

# Variables
$PolicyName="SSL-POLICY--$(Get-Date -Format yy-MM-ddTHHmmss)"

# Dependencies
Import-Module AWSPowerShell

Write-Host 'Finding AWS Regions containing ELBs...'

$RegionsWithELBs = @{}

ForEach ( $region in (Get-EC2Region).RegionName )
    $ELB_Count = (Get-ELBLoadBalancer -Region $region).count
    if ( $ELB_Count -ge 1 )
        $RegionsWithELBs.Add($region, $ELB_Count)

# Display ELB Regions & Count
$tformat = @{Expression={$_.Name};Label="Region"}, @{Expression={$_.Value};Label="ELB Count"}
$RegionsWithELBs.GetEnumerator() | Sort-Object Value -Descending | Format-Table $tformat -AutoSize

ForEach ( $region in $RegionsWithELBs.Keys )
    # Verify reference policy existence in region
    if ( (Get-ELBLoadBalancerPolicy -Region $region).PolicyName -contains $ELBReferencePolicy )
        Write-Host "`nModifying ELBs in region: '$region' `n"

        # Loop through the ELBs
        ForEach ( $lb in (Get-ELBLoadBalancer -Region $region ).LoadBalancerName )
            # Verify ELB serves HTTPS
            if ( (Get-ELBLoadBalancer -Region $region -LoadBalancerName $lb).ListenerDescriptions.Listener.Protocol -contains 'HTTPS' )

                # Find Existing Policies (App/Cookie Stickiness, etc.)

                $PoliciesToApply = @($PolicyName)

                ForEach ( $currentpolicy in ((Get-ELBLoadBalancer -Region $region -LoadBalancerName $lb).ListenerDescriptions | Where-Object { $_.Listener.Protocol -contains 'HTTPS'}).PolicyNames )
                    if ( (Get-ELBLoadBalancerPolicy -Region $region -LoadBalancerName $lb -PolicyName $currentpolicy).PolicyTypeName -ne 'SSLNegotiationPolicyType' )
                        $PoliciesToApply += @($currentpolicy)

                # Configure SSL Policy
                Write-Host "`nCreating '$PolicyName' from '$ELBReferencePolicy' for $lb"
                New-ELBLoadBalancerPolicy -Region $region -LoadBalancerName $lb -PolicyName $PolicyName `
                  -PolicyTypeName SSLNegotiationPolicyType `
                  -PolicyAttribute @{ AttributeName="Reference-Security-Policy";AttributeValue="$ELBReferencePolicy"} `
                Write-Host "Activating policy '$PolicyName' for ELB: $lb"
                Set-ELBLoadBalancerPolicyOfListener -Region "$region" -LoadBalancerName "$lb" -LoadBalancerPort 443 -PolicyName $PoliciesToApply

                # Cleanup Old Policies
                ForEach ($policy in (Get-ELBLoadBalancerPolicy -Region "$region" -LoadBalancerName "$lb" | Where-Object {$_.PolicyTypeName -eq 'SSLNegotiationPolicyType'}).PolicyName)
                    if ( $policy -ne $PolicyName -and $policy -ne $ELBReferencePolicy )
                        Write-Host "Removing old policy '$policy' from ELB: $lb"
                        Remove-ELBLoadBalancerPolicy -Region "$region" -LoadBalancerName "$lb" -PolicyName $policy -Force
        Write-Host "Region $region does not contain policy $ELBReferencePolicy"

AWS – Highly-Available NAT in VPC


Like most sysadmins, one of my primary responsibilities is ensuring high-availability in our environments. Recently, I’ve been working a lot more with Amazon AWS. Amazon recently began forcing new accounts to make use of VPC. When you create a VPC, an Internet Gateway must be provisioned to route traffic to the Internet. VPC’s utilize subnet constructs for virtual networking. Subnets are assigned a routing table, and in the case of a Public subnet, the default route of this table is pointed at the Internet Gateway. Instances in this public subnet are assigned public, non-RFC1918 Elastic IP addresses. At the moment, only 5 Elastic IP addresses may be requested per account. You can request more via support, but obviously they are trying to ween people away from using them for everything. Consequently, NAT & supporting instances must be in place to facilitate external communication for non-public subnets.

In the case of these subnets, the default-route should be pointed at a NAT instance residing in the Public subnet. This brings about a single point of failure. Should the NAT instance go down, nothing in that subnet can speak to the outside world; the default-route becomes a black-hole. In order to combat this, multiple NAT instances can be provisioned in different availability zones, and with a little magic, configured to take over each others traffic-routing responsibilities on-demand.

Amazon has furnished a document with a workaround for this situation. Essentially, a script running on each NAT instance performs a health check on the other NAT instance, and should the other instance go down, the healthy instance will take over. It does so by adjusting the routing tables via AWS API calls. The script will also attempt to bring the failed instance back online.

UPDATE: The NAT Monitor script outlined by Amazon has a flaw. The ec2-describe-instances call to determine the state of the other NAT instance does not function properly. The documentation references using $5 instead of $4 to set the NAT_STATE variable, however I have found $6 to work best, but test this because your EC2-API-tools version might yield different results. I also highly suggest the –show-empty-fields argument because if the number of fields changes, the awk statement could potentially grab the incorrect field.

NAT_STATE=`/opt/aws/bin/ec2-describe-instances $NAT_ID -U $EC2_URL --show-empty-fields | grep INSTANCE | awk '{print $6;}'`

There is one issue with the configuration outlined by the Amazon document; the IAM roles permissions are too loose. Using the policy defined in the document, the NAT instance is granted permissions to restart every instance belonging to the account. Additionally, the NAT instance could modify any and all routing tables, such as those in other regions, VPC's, etc. You probably don't want your NAT instances in US-West-2 making any modifications whatsoever to US-East-1. The below policy is an attempt to restrict permissions as best as permitted by supported IAM policy conditions. Just substitute/replace the region and VPC information with your own. Also, tag the NAT instances with a 'Type' and 'VPC' field, setting 'Type' to 'NAT' and 'VPC' with the VPC's ID.

Restricted IAM Policy


Branch Office Connections – GRE over IPSec VPN

While studying for the CCNP Route exam, I noticed that GRE Tunneling and IPSec were mentioned as topics, however configuration of the two was never really covered in the certification guide.  In addition, I did not see any labs involving this type of scenario, so I decided to create my own.  Hopefully this provides some insight to others about how to configure GRE Tunnels over IPsec.  Don’t feel bad if this takes you a while, setting up this scenario and writing this up took a whole Saturday afternoon.  The configuration files are at the bottom of the post.

IPSec can secure data in transit and provides authentication that it came from a trusted source.

To route from the branch office to the enterprise core, you can use either static routes or a routing protocol. To secure traffic over the internet, IPSec can be used.  To utilize a routing protocol to connect securely to the core with IPsec, it must be combined with a GRE tunnel which supports the broadcasts/multicasts necessary for proper IGP operation.

GRE Tunnels

  • Act link a point-to-point link from a Layer 3 perspective.
  • Supports many passenger protocols, including IPv4.
  • Encapsulates/forwards broadcasts and multicasts, therefore supporting IPv4 IGP’s.
  • GRE tunnels can run through IPsec tunnels.

Implementing Branch Office GRE over IPSec VPN

Branch Office GRE/IPSec VPN (R2-R4) & Private Serial (R3-R4)

Here, we will utilize split-tunnel at the Branch Office to ensure that only traffic destined for the Core will be encrypted over the GRE IPSec tunnel. A default route is configured on R2 and R4 to ISP1. Additionally, ISP1 has been configured with a loopback to test split-tunnel functionality.

All routers (except ISP1 obviously) will participate in EIGRP AS 1. Each participating router shall be configured with the network and no auto-summary commands.

Configure the GRE Tunnels on R2 and R4

The tunnel source is the internet-facing interface, in this case fa0/0 with the IP of on R2 and fa0/1 with the IP of on R4.
When associating a tunnel source with an interface, if the associated interface goes down, so does the tunnel.  If you have redundant connections, use a loopback as a source instead.

R2 – R4 GRE Tunnel Configuration

First, configure the GRE tunnel. After doing this, you should see your EIGRP neighbors go up.

In a real world situation, you’d probably also want to configure the tunnel with Path MTU Detection.  This will mitigate fragmentation issues that could cause your routers to drop CEF and use process switching.  Enable PMTUD at each tunnel endpoint with tunnel path-mtu-discovery.



R2 GRE Route-Map

Here, we will configure a route-map to prevent a default route from ever being advertised over the GRE tunnel. The only way for R4 to ever learn R2’s default route should be via the serial connection to R3, and it should only be utilized if R4’s fa0/0 interface goes down. We could also utilize this to block other networks from being advertised over the GRE tunnel by modifying the ACL.

Now, you can see that R4 only learns a backup Default-Route through R3. Previously, it learned it through the tunnel as well, which would be silly to utilize and might break split-tunnel if the static default-route was lost for some reason.

Encrypt GRE Tunnel traffic using IPSec

We will now encrypt traffic traversing the GRE tunnel using IPsec. We must first define an ACL to catch the traffic we wish to encrypt, which will be GRE traffic from our GRE source-interface destined for the GRE tunnel destination.

R2 IPsec Policy Creation

ACL (Catch GRE to R4)


ISAKMP Key & IPSec Transform Set (Transport Mode)

Keep in mind that transport mode is used only because we are not traversing NAT, we are going from R2 to R4’s public IP for both IPSec and GRE Tunnels.   Tunnel mode is required if the GRE destination is different from the IPSec destination.  If the endpoints were each behind a NAT router, the situation requires a combination of NAT-T and Tunnel Mode.  Tunnel mode adds an extra 20 bytes to the overall packet size. NAT-T also adds an extra 20 bytes per packet and encapsulates the IPSec traffic into UDP packets over port 4500.

Crypto Map Configuration

Here, we configure the Crypto Map to R4. We will use the previously created GRE catching ACL.

Apply Crypto Map to GRE Tunnel

The Crypto Map should be applied to the physical exit interface. Cisco recommends applying it to the logical tunnel interface as well.

R4 IPsec Policy Creation

We will follow the same procedure utilized on R2 and reverse the IP addresses.

ACL (Catch GRE to R2)


ISAKMP Key & IPSec Transform Set (Transport Mode)

Crypto Map Configuration

Here, we configure the Crypto Map to R4. We will use the previously created GRE catching ACL.

Apply Crypto Map to GRE Tunnel

Verify IPsec and Split-Tunnel Functionality

You can utilize Wireshark to verify only ESP traffic traverses between the GRE endpoints. I found that the Crypto Map did not need to be applied to the logical Tunnel 0 interface to function, but only the physical exit interface. I am not sure why Cisco recommends applying it to both.

You can see that the Crypto traffic for Tunnel 0 and FastEthernet 0/0 is identical. I am not sure if running the crypto map VPN on both the Tunnel 0 and FastEthernet 0/0 is causing excess encryption overhead, or if it only encrypts the traffic once.

After disabling Crypto Map VPN on the Tunnel 0 interface, I confirmed via Wireshark that the GRE traffic continued to be encrypted. A show crypto ipsec sa detail shows that the traffic continues to be encrypted as well.

Verify Split-Tunnel

To test this, I will simply ping a simulated public internet address of from the inside interface of R4.


We have implemented remote branch office with redundant connectivity to the core; a leased line and a secured backup utilizing GRE over IPsec via the public Internet. Our branch office is now able to participate in the EIGRP routing process over either link.  If you wanted to use the IPsec VPN as your primary connection to HQ, modify the bandwidth value of the Tunnel 0 interfaces to make the EIGRP metric more preferable than the serial route.  By default, tunnel interfaces seem to be given a bandwidth value of only 9kbps.  The default delay value of tunnel interfaces is high as well.  Tweak the bandwidth and delay values to achieve desired results.

Normal Conditions

Downed Serial Link to HQ

Configuration Files

ISP1 Configuration

R1 Configuration

R2 Configuration

R3 Configuration

R4 Configuration

VMware ESXi & Intel SR1550AL Servers

I recently encountered an issue when running an ESXi 4.1 environment where I was getting a Purple Screen of Death at right about the 24 hour uptime mark.  Like most technologists, I first hit Google with the error code and had no luck.  I found a similar error code which suggested that it had something to do with a storage I/O issue.  The issue did not seem driver related, so I began to look into firmware and BIOS updates for the server.

After some investigation, I realized that there were four BIOS/firmware’s in my SR1550 which needed updating.  After trying to do them all manually, I found a nifty utility called the Intel Deployment Assistant.  The Intel DA utility is a bootable CD that allows you to configure many different features, including the ability connect to the Internet to check and download the latest BIOS/firmware updates and install them automatically.  This worked great and I found out that the system had two year old BIOS and firmwares and the current build numbers were much newer.  So, I let the utility do its thing, but one update kept failing.  I tried it on all three of the SR1550’s and had the same issue on all of them; the HSC/backplane firmware would not update.

At this point, I started looking for older revisions of the Hot Swap Controller firmware figuring that the jump from revision 1.41 to 2.15 was too great.  There were many revisions between 1.41 and 2.15, but they were not available on Intel’s website.  Finally, after a lot of searching, I found an Intel document that confirmed my suspicion that I need to upgrade to a different older update prior to jumping to 2.15.  Alas, I decided I had to get into contact with Intel about this issue.

At this point, I’m sitting in the server room and its already after six and my stomach is growling.  I had no phone reception in the server room so I tried a button to chat with an Intel support representative.  Much to my surprise, after about 10 minutes, the guy I was in contact had a solution.  Although he could not locate the firmware revision I needed, he located an internal document that stated if I disconnected the RMM2 and then proceeded with the upgrade to 2.15, it should work.  So I popped open the server, disconnected the RMM2 module, booted up, manually launched the update from a DR-DOS boot-disk and much to my surprise, it worked!

For those of you who do not know what the Intel RMM is, it is a Remote Management Module that allows you to have out-of-band control of the server.  As a matter of fact, it offers an IP-based KVM over SSL through a Java applet, and it even allows you to remotely mount and boot the server to ISO images… How awesome is that!   After that little bugger gave me all that headache, I decided to give it a try and it is great.  You use a little utility called psetup to configure IP and login settings, and then your good to go!

Anyway, yes, after updating all of those firmwares, I now have over 180 days uptime on the ESXi boxes.  I hope this helps any other SR15xx, SR25xx users out there that may be encountering stability issues with ESXi.