Posted on by patrickmarcpreuss

Fortigate – IPSec Troubleshooting – VPN Analyse

Force VPN Tunnels to the CPU
First you need to force the Encrypted traffic over the CPU.

Take packet captures from the GUI

It is somehow counter intuitive but we need to take care that the IPSec session is not offloaded into Hardware. This basically the Opposite as described in Ensuring IPSec traffic is offloaded for improved throughput.

Check the configuration

Ensure the NPU Offload is disabled on phase1

config vpn ipsec phase1-interface
    edit phase-1-name
        set npu-offload disable
end

Check also the firewall policies related to the VPN tunnel are not to be offloaded.

config firewall policy
    edit <policy_id>
        set auto-asic-offload disable
end

related commands

diagnose vpn ipsec status

diagnose vpn ipsec status
All ipsec crypto devices in use:
NP6_0:
    Encryption (encrypted/decrypted)
        ...
        aes              : 32687140864      22498409408

        ...
    Integrity (generated/validated)
        ...
        sha256           : 32687140864      22498409408
        ...

NP6_1:
    Encryption (encrypted/decrypted)
        ...
        aes              : 0                2636628416
        ...
    Integrity (generated/validated)
        ...
        sha256           : 0                2636628416
        ...
...

NPU Host Offloading:
    Encryption (encrypted/decrypted)
        ...
        aes              : 3070486752       0
        ...
    Integrity (generated/validated)
        ...
        sha256           : 3070508591       0
        ...

CP9:
    Encryption (encrypted/decrypted)
        ...
        aes              : 10822            186613515
        ...
    Integrity (generated/validated)
        ...
        sha256           : 10824            186613518
        ...

SOFTWARE:
    Encryption (encrypted/decrypted)
        null             : 0                0
        des              : 0                0
        3des             : 0                0
        aes              : 0                0
        aes-gcm          : 0                0
        aria             : 0                0
        seed             : 0                0
        chacha20poly1305 : 0                0
    Integrity (generated/validated)
        null             : 0                0
        md5              : 0                0
        sha1             : 0                0
        sha256           : 0                0
        sha384           : 0                0
        sha512           : 0                0

diagnose vpn tunnel list

name=IPSEC-TUNNEL ver=2 serial=5 192.0.2.101:0->198.51.100.20:0 dst_mtu=1500
bound_if=15 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/520 options[0208]=npu frag-rfc  run_state=0 role=sync-primary accept_traffic=1 overlay_id=0

proxyid_num=1 child_num=0 refcnt=312 ilast=0 olast=0 ad=/0
stat: rxp=208918627 txp=159912503 rxb=202810145273 txb=78937393736
dpd: mode=on-idle on=1 idle=20000ms retry=3 count=0 seqno=1
natt: mode=none draft=0 interval=0 remote_port=0
proxyid=RD-NAFTA_2f_0 proto=0 sa=1 ref=2101 serial=1
  src: 0:0.0.0.0/0.0.0.0:0
  dst: 0:0.0.0.0/0.0.0.0:0
  SA:  ref=6 options=10225 type=00 soft=0 mtu=1438 expire=169/0B replaywin=0
       seqno=495cd esn=0 replaywin_lastseq=00000000 qat=0 rekey=0 hash_search_len=1
  life: type=01 bytes=0/0 timeout=1767/1800
  dec: spi=ff3f19ff esp=aes key=32 redacted
       ah=sha256 key=32 redacted
  enc: spi=ffe7b5ff esp=aes key=32 redacted
       ah=sha256 key=32 redacted

  dec:pkts/bytes=517158/459351384, enc:pkts/bytes=404152/75330823
  npu_flag=03 npu_rgwy=198.51.100.20 npu_lgwy=192.0.2.101 npu_selid=3 dec_npuid=1 enc_npuid=1
run_tally=1

npu_flag=00 Means that ingress & egress ESP packets are not offloaded.
npu_flag=01 Means only egress ESP packets can be offloaded, ingress ESP packets will be handled by the kernel.
npu_flag=02 Means only ingress ESP packets can be offloaded, and egress ESP packets will be handled by the kernel.
npu_flag=03 Means that both ingress & egress ESP packets will be offloaded.

Analyse

Extracting ESP Sequence Numbers

To analyze VPN traffic, we ‘ll first need to extract ESP (Encapsulating Security Payload) sequence numbers from capture files taken on both SIDE A and SIDE B of the VPN gateways. This will allow us to examine the sequence numbers sent and received in each direction.

Use the following commands to filter and extract ESP sequence numbers based on source and destination IP addresses:

tshark -r CAPTURE-TAKEN-ON-SIDE-A.pcap -Tfields -e esp.sequence -Y "ip.src==<GATEWAY IPADDRESS OF SIDE A>" > SIDE-A-sender-is-SIDE-A.csv
tshark -r CAPTURE-TAKEN-ON-SIDE-B.pcap -Tfields -e esp.sequence -Y "ip.src==<GATEWAY IPADDRESS OF SIDE A>" > SIDE-B-sender-is-SIDE-A.csv
tshark -r CAPTURE-TAKEN-ON-SIDE-A.pcap -Tfields -e esp.sequence -Y "ip.dst==<GATEWAY IPADDRESS OF SIDE A>" > SIDE-A-sender-is-SIDE-B.csv
tshark -r CAPTURE-TAKEN-ON-SIDE-B.pcap -Tfields -e esp.sequence -Y "ip.dst==<GATEWAY IPADDRESS OF SIDE A>" > SIDE-B-sender-is-SIDE-B.csv

Comparing ESP Flow in Each Direction

Next, you’ll compare the extracted ESP sequence numbers for each direction to identify any potential issues, such as late or missing packets. The diff command is used for this purpose. Lines with a pipe character “|” indicate where the numbers differ between SIDE A and SIDE B.

Example:


diff -y  SIDE-A-sender-is-SIDE-A.csv SIDE-B-sender-is-SIDE-A.csv 
1963269                                                         1963269
1963270                                                       | 1897734
1963271                                                         1963271

Analyzing SPI Number Differences

You can also calculate the difference between SPI (Security Parameter Index) numbers to gain insights into the packet flow. For instance:

# Calculate the SPI number difference
# echo 1963269-1897734 | bc
65535

This result suggests that the SPI numbers are 16 bits apart, which can provide valuable information about the security associations.

You may want to check for IP fragmentation, RFC compliant mode is Fragmentation after encryption.

For me it is not an option to do Fragmentation before encryption, this will most likely break some applications.

Best course in this case is implement TCP MSS Clamping on the Gateway to a reasonable Size. My Personal preference on VPN Tunnels is to do clamping to an MSS of 1350, this account for IPSec ESP Tunnel Mode Headers, Trailers, Padding it leaves also Headroom for Internet Connections with PPPoE, GRE Tunnels.

On Server Systems you can also implement per destination routes with MTU, this should lower this for all Traffic on the server. Third look into the configuration of the Application to lower the MTU or MSS.

Links

Photo by Mika Baumeister on Unsplash

Blog at WordPress.com.