In this post, we are going to go over troubleshooting our VPN using debug commands. This is particularly useful for the folks out there reading this that only have access to only one side of the VPN or have a VPN to a 3rd party. I wanted this to remain a separate post from my ASA and IOS site-to-site VPN configuration posts because troubleshooting this is almost entirely identity on both a router or an ASA so I wanted to combine the troubleshooting to a single post.
Walking through Successful IPSec VPN Creation
I'm going to start with the debug crypto isakmp command and walk through a successful ISAKMP SA creation. This is after I issue the clear crypto session command and ping a host from one side to the other side.
From the beginning, we see the the initiator start to prepare to establish the SA to the other peer (2.2.2.1). The output states that the source/destination port will be 500 (UDP as we know) and that it can't start Aggressive Mode since it's not configured to so it's going to use Main Mode. It next states that it's found a preshared key configured locally for the peer (crypto isakmp key cisco123 peer 2.2.2.1). At this point, Main Mode has NOT started,
After that, we'll see Main Mode begin. This is where the parameter negotiation will begin:
Then MM#3 and #4 are processed as part of the Diffie-Hellman exchange and if you'll notice from the highlighted portion, NAT-T did it's job in detecting that there was no NAT between peers.
Main mode will wrap up with MM#5 and #6 where the pre-shared keys are being used to authenticate each other, the Send ID s are shared, etc
Phase 1 has now completed and Phase 2 will begin. The output will let you know that Quick Mode is starting. You can see the first Quick Mode message sent from the initiator with the IPSec proposals (crypto ipsec transform-set tset esp-aes 256 esp-sha512-hmac).
The peer will send back a reply with chosen proposal and the Proxy ID.
The initiator will then send the final Quick Mode message as a final acknowledgement. At this point, the debug output will indicate that Phase 2 has completed.
Makes sense, right? Since the name of this post has "troubleshooting" in it, let's break some stuff to see what it looks like.
Note: When troubleshooting site-to-site VPNs, there's always a side that sends the first packet. This process is started by the first side that needs to send traffic to the other side. This peer is referred to as the initiator. The responder always gets a bit more detail in regards to what is going wrong during the IKE process. If you need to troubleshoot why a VPN won't come up, a good exercise might be to clear the crypto session and then let the other side initiate the traffic if you find yourself the initiator. For educational purposes, I'm going to walk you through what it looks like when VPN failing from both sides.
TROUBLESHOOTING PHASE 1
For this section, I'm going to make some changes to the ISAKMP policy on the remote peer and clear the crypto session by issuing the clear crypto session command. When we do the debug after we clear the session, the changes I made should be reflected.
ISAKMP Policy Troubleshooting
From the initator, this is what it looks like when the initial ISAKMP policy parameter negotiation has failed:
As one can see from the above output, it never makes it past the MM#1 and #2 exchange and the ISAKMP policy is rejected. At this point, one could probably bank on it failing for one of the following reasons:
- Encryption mismatch
- Hash mismatch
- Diffie-Hellman Group mismatch
- Authentication type mismatch
If this is all you can see and you can't get the other side to troubleshoot it with you or have them initiate traffic so you can view the output as a responder, then I would have the other side verify the above. If your side is the responder, then let's dig into what it looks like for the conditions it could be.
On the responder side, the debug output will actually specify what exactly was wrong. Here are the following outputs for various configurations I broke:
- Mismatch Encryption in the ISAKMP policy
- Mismatch Hash algorithm in the ISAKMP policy
- Mismatch Diffie-Hellman Group in ISAKMP policy
- Mismatch Authentication type in ISAKMP policy
- Mismatch Authentication type in ISAKMP policy
Now let's take a look at what happens when the ISAKMP policy is matching and the ISAKMP peers are defined but the pre-shared keys are wrong.
From the initator side, everything will look correct until you get to MM#5 where the peers are authenticating and it will fail. From the initiator side, you will see the initator prepare to send MM#5 which will authenticate itself to the peer and it will clearly fail and start retransmitting until it times out.
If it fails at this point, it's extremely likely there is a key mismatch in the crypto isakmp key <key> address <peer-address> configuration. This command had to exist in the configuration in order to get past the initial MM#1 and MM#2 messages but since MM#5 and MM#6 is where both the peers use that key to authenticate to each other, that's where a mismatched key would fail.
On the responder side, the debug is clearer and shows that the key exchange failed after MM#4 finished and MM#5 was supposed to begin due to MM_KEY_EXCH
TROUBLESHOOTING PHASE 2
Now we're going to jump into Phase 2 troubleshooting.
I'm going to alter my IPSec transform set to let it fail on Phase 2. By changing the transform set, I should see the Main Mode exchange complete and Phase 2 start. From the intiator, you should see Quick Mode fail on QM#2 where no proposal is chosen:
On the responder side, we can see a bit more detail. In this example, I changed the responder transform-set to esp-3des instead of esp-aes like the peer is configured to use and I'm using debug crypto ipsec to generate the following output.
That's very intuitive is it? Let's try something different. Let's turn off ISAKMP debugging and use debug crypto ipsec and try to do this again:
Much better, right? This data is gleamed from the responder and unfortunately won't show in the debug of the initiator but as stated above, you can easily flip the script on who the initiator is when troubleshooting site-to-site IPSec VPN.
If I change the encryption back to what it was before and change the hash to esp-md5-hmac on the responder, you can again see that hmac-sha512 is being offered by the initator and being rejected by the responder:
The same goes for when I configure the responder with mode transport and the other peer is mode tunnel for IPSec. As you can see, the initiator is offering tunnel and the responder rejects it in this debug from the responder:
Now I'm going to fat finger the peer IP address in the crypto map and see what happens:
As you can see able, the peer address is not found in the crypto map on the responder and it is rejected.
Now lets say I mess up and put the wrong ACL in the crypto map or don't put one that matches traffic for the traffic going from one side to the other. On the responder, it will clearly state that the proxy IDs are what were not supported:
With that, I'm going to go ahead and wrap up this blog post!