What Is IPsec (Internet Protocol Security)?

Internet Protocol Security (IPsec) is a suite of protocols for protecting data over untrusted networks. It does this by delivering three core controls at the network layer:

• Encryption. Every IP packet is made unreadable to anyone who is not authorized.
• Endpoint authentication. Both sides prove who they are before a tunnel is established.
• Integrity checking. Cryptographic hashes ensure nothing is altered on the way.

IPsec is used for securing almost any IP-based service without modifying the application itself. This flexibility makes it the de facto standard for site-to-site and remote access Virtual Private Networks (VPNs), cloud VPN gateways, and granular network segmentation across hybrid and multi-cloud environments.

PCI DSS requires strong cryptography to protect sensitive information during transit. Protocols such as IPsec, Transport Layer Security (TLS), and Secure Shell (SSH) are examples of secure technologies that can be used to meet this requirement.

How Does IPsec Work?

Every IPsec session begins with a control-plane handshake and ends with data packets being protected end-to-end.

The network layer (Layer 3) handles this entire process, allowing any IP-based protocol (such as HTTP, VoIP, database traffic, or custom apps) to be secured without touching the application code. At the same time, the data remains private, authenticated, and protected against replay attacks.

Let’s look at the flow of a standard IPsec setup:

1. IKE negotiates and authenticates

The two devices run the Internet Key Exchange (IKE v2) handshake. Each side proves its identity using a certificate, a pre-shared key (PSK), or an Extensible Authentication Protocol (EAP) method, and creates fresh encryption keys. No keys travel over the network.

IPsec uses keys for encryption, which are random character strings that lock and unlock (encrypt and decrypt) messages. These keys are exchanged between connected devices, enabling mutual decryption of communications.

2. Child Security Associations (SAs) are created

All information sent across a network is broken into packets, which are small bundles containing both the actual data and instructions for computers. IPsec secures packets by attaching extra information, called headers, at the start, and trailers at the end.

3. Packets are wrapped with ESP header/trailer

IPsec checks each data packet to make sure it’s authentic. This way, you know the packets came from a trusted source.

4. Packets cross the network

IPsec keeps your data private by scrambling the information inside each packet, including its address details. Everything you send using IPsec stays confidential.

5. The receiver unlocks and delivers

Encrypted IPsec packets travel across networks using a delivery method. They often use the User Datagram Protocol (UDP), a faster protocol that doesn’t establish dedicated connections like Transmission Control Protocol (TCP) does. This allows them to pass through firewalls more easily.

6. Decryption

The receiving end decrypts the packets, allowing applications to access the delivered data. All cryptographic agility (such as algorithm changes, rekey intervals, and SA lifetimes) is handled centrally through IKE and the Security Association Database, so your IT or security team can enforce confidentiality, integrity, and peer authentication in one place.

IPsec Protocols

An IPsec deployment uses two data protection protocols: Authentication Header (AH) and Encapsulating Security Payload (ESP). It relies on one key management protocol, Internet Key Exchange (IKE). It also depends on a core construct called the Security Association (SA). AH and ESP protect packets while IKE negotiates keys and parameters. SA is the peer-to-peer direction rule set that both peers install.

Below, we explain each IPsec protocol and element, and how they are used in deployments.

1. Authentication Header (AH)

AH provides integrity and sender authentication and can block replayed packets. It does not encrypt data. It is rarely used today because ESP can provide integrity without encryption through ESP-NULL. Legacy environments may still run AH for integrity-only traffic.

2. Encapsulating Security Payload (ESP)

ESP is the core IPsec protocol, providing confidentiality, integrity, authentication, and anti-replay protection. For this reason, the NIST Guide to IPsec VPNs (2020) recommends using ESP for all new VPNs and host-to-host tunnels. It states that “null encryption with the ESP protocol be used instead of the AH protocol when encryption is not desired.” This is because ESP’s integrity-only mode (ESP-NULL) makes AH redundant.

3. Security Association (SA)

An SA is a one-way agreement that defines how traffic is protected. It includes the chosen protocol, algorithms, keys, lifetimes, transport or tunnel mode, and the Security Parameter Index that identifies it on the wire. Each direction uses a separate SA.

4. Internet Key Exchange (IKE)

IKE is the protocol that establishes the conversation before any protected traffic flows. It establishes a secure control channel, authenticates peers, and generates shared keys using the Diffie-Hellman key exchange. The key management protocol then installs the SAs, enabling ESP to protect your data.

IKEv2, the standard used in current deployments rather than the legacy IKEv1, uses fewer messages, recovers from outages more effectively, supports mobility and multihoming, and provides more precise error handling.

IPsec Modes of Operation

IPsec operates either in transport or tunnel mode. In transport mode, the focus is on protecting the payload of the original packet while keeping the original IP header visible for routing. Tunnel mode, on the other hand, encapsulates the entire original packet, adding a new outer IP header that hides the original source and destination addresses during transmission.

1. Transport mode

Transport mode applies ESP or AH directly to the data payload. While the actual content is encrypted or authenticated, the original IP header, including source and destination addresses, stays intact so routers can forward the packet as usual. Transport mode is typically used in server-to-server or application-specific tunnels where both endpoints run IPsec.

Here are a few common scenarios where you can use transport mode:

• Protect database queries from an app server to a database server on the internal network.
• Secure API calls between microservices that operate on different hosts.
• Restrict management access from an admin workstation to a specific server.

2. Tunnel mode

Tunnel mode wraps the original packet inside a new packet with a new outer IP header, masking the source and destination addresses. Tunnel mode is used for both site-to-site and remote-access VPNs because it effectively conceals internal addressing and separates internal routing from the public network.

Some examples of when to use tunnel mode:

• Allow remote employees to access internal applications from their laptops through a VPN gateway.
• Link a Virtual Private Cloud (VPC) to an on-premises data center for hybrid workloads.
• Backhaul traffic from a small site to a regional hub where security tools are located.

The table below provides a comparison of how transport and tunnel modes behave in practice.

How IPsec Ensures Data Security

IPsec ensures data security by keeping data confidential, unaltered, and tied to the right sender. It encrypts data for confidentiality, checks that packets were not altered, confirms they came from an authenticated peer, and blocks replayed packets.

Let’s look at how each control works in practice to keep your data secure.

1. Confidentiality with ESP

IPsec uses ESP to encrypt data so it is unreadable to anyone who intercepts it. The peers negotiate the cipher and key length. Deployments today use AEAD ciphers such as AES-GCM or ChaCha20-Poly1305, which provide both encryption and integrity.

2. Integrity with ESP or AH

ESP or AH adds an integrity check to confirm your data has not been tampered with. With AEAD ciphers, integrity is part of the encryption. With non-AEAD ciphers, integrity is provided by a keyed hash such as HMAC with SHA-2. The sender computes an Integrity Check Value, and the receiver recomputes it on arrival.

3. Peer authentication in IKE

IPsec checks that traffic comes from the device you agreed to trust. During IKE, the two peers prove their identities using a pre-shared key or a digital certificate, then establish a shared tunnel key.

This process ties the traffic to an authenticated device or gateway. If you need to verify a specific user, add user authentication on top, for example, EAP in IKEv2 or application-layer login.

4. Replay protection with sequence numbers

IPsec uses sequence numbers in each packet header and keeps a moving window of recent numbers. If a packet arrives with a number that is already in use or too old for the window, IPsec drops it. This prevents attackers from replaying captured packets to trigger actions or confuse systems. The window size can be tuned to tolerate normal packet reordering on busy networks.

IPsec vs. SSL VPN

IPsec and SSL/TLS create encrypted tunnels. The differences are the layer they operate on, how much traffic they can protect by default, and their ability to traverse networks easily. Both options support modern ciphers and strong authentication. The better choice depends on whether you are connecting networks and devices or enabling user-centric access to applications.

Choose IPsec: When uptime-critical systems need full-network encryption across fixed sites or managed devices

Choose SSL/TLS VPN: When users require quick, firewall-friendly access through a web browser or lightweight agent.

The table below compares the two across key factors.

Advantages of Using IPsec

IPsec offers significant advantages because it operates at the network layer, uses modern cryptography, supports many deployment patterns, adheres to open standards for interoperability, and helps meet encryption-in-transit requirements when paired with current cipher suites and logging.

1. Network layer transparency

IPsec protects IP packets without requiring changes to applications or protocols. It consistently covers most traffic, reducing per-app configuration and drift, making it easier to deploy across mixed environments and legacy systems.

2. Strong cryptographic capabilities

ESP uses modern ciphers that provide confidentiality and integrity simultaneously. IKEv2 establishes keys and rotates them on a schedule while also supporting Perfect Forward Secrecy. Anti-replay checks block packet reuse attacks.

3. Flexible deployment

IPsec adapts to site-to-site, user-to-gateway, cloud-to-data center, and host-to-host use cases. Transport mode reduces overhead between specific endpoints, while tunnel mode cleanly connects networks and hides internal addressing.

4. Interoperability

Being a widely accepted open standard allows IPsec to operate across vendors, increasing choice in the IT systems in your organization. This also eases migrations and multi-vendor designs. However, this interoperability can also become a challenge during configuration when each vendor interprets or prioritizes RFC (Request for Comments) options differently, or default settings don’t align.

5. Regulatory compliance

Many regulatory and security frameworks, including ISO/IEC 27001, NIST SP 800-53, GDPR, PCI DSS, and HIPAA, require or recommend encryption in transit for sensitive or regulated data. With modern cipher suites and proper logging, IPsec helps your organization meet these controls and provides auditable records of key events and policy enforcement.

Challenges and Limitations

Deploying IPsec comes with specific challenges, notably its performance overhead, reliance on stable network transport, and inherent visibility constraints. Other limitations include firewall and Network Address Translation (NAT) compatibility, as well as lifecycle management. We’ll discuss these in more detail below.

1. Configuration complexity

IPsec configurations can be tricky, consisting of many components that must be properly aligned on both ends. These include IKE versions, cipher suites, Diffie–Hellman groups, lifetimes, identities, modes, and traffic selectors. Even minor discrepancies can disrupt handshakes, create unstable tunnels, or compromise security if outdated defaults are unintentionally included.

2. Firewall and NAT compatibility

Firewalls frequently obstruct IPsec traffic, demanding precise configuration to ensure proper operation. NAT devices can also disrupt traditional IPsec tunnels by rewriting IP headers. This invalidates ESP’s integrity check and may block essential protocol 50 traffic. To maintain secure connectivity, both endpoints must implement NAT Traversal (NAT-T) during the IKE negotiation phase.

3. Performance overhead

Encrypting every packet adds extra bytes and CPU work, which can slow traffic on older or heavily loaded links. You can reduce this performance overhead by turning on hardware encryption support (AES-NI), using network cards that handle IPsec in hardware, and lowering the Maximum Transmission Unit (MTU) or Maximum Segment Size (MSS) to avoid fragmentation. You should also keep an eye on tunnel performance to prevent bottlenecks on busy links.

4. Ongoing lifecycle management

IPsec deployments depend on shared secrets or certificates that must be issued, rotated, and revoked. Even with automation, administrators must track expirations, enforce strong entropy, and audit key usage across every tunnel to keep the environment secure.

5. Reliance on transport performance

IPsec cannot compensate for a congested or unstable network. Regardless of how robust the IPsec configuration is, high latency, packet loss, or asymmetric routing on the underlying transport can slow traffic, trigger frequent rekeys, or even cause the tunnel to drop.

6. Visibility constraints

Since IPsec encrypts the payload (and sometimes headers), traditional NIDS (Network Intrusion Detection System) tools can’t parse or match signatures. This visibility gap is why analysts often pair IPsec with network traffic analysis (NTA) tools that analyze flow metadata rather than payloads.

Rather than decrypting data, NTA identifies threats by analyzing packet size, timing patterns, destinations, and connection frequency. It can spot malware sending beacon signals at precise intervals, even without decrypting the traffic.

Integrating IPsec into Zero Trust Architectures

Zero Trust security replaces “trust but verify” with “never trust, always verify,” validating every user, device, and workflow regardless of location.

While IPsec does not deliver a complete Zero Trust solution on its own, its native capabilities, such as encryption and authentication, provide a strong foundational layer for Zero Trust principles. Here’s how:

1. Micro-segmentation at Layer 3

IPsec segments network traffic into tightly controlled, policy-driven zones by creating SAs for specific IP ranges. Only permitted IP pairs within each SA can communicate, significantly reducing lateral movement in the event of a breach.

2. Mutual cryptographic authentication

IPsec uses mutual authentication during the IKEv2 handshake, requiring peers to exchange and validate pre-shared keys or digital certificates before data transmission begins. Integrating this process with an enterprise certificate authority or public key infrastructure (PKI) ensures each session adheres to the “always verify” principle at every connection attempt.

3. End-to-end encryption

ESP encrypts traffic by default, either the payload only (transport mode) or the entire packet (tunnel mode), utilizing strong ciphers such as AES-256-GCM. This guarantees data confidentiality across public networks, remote branches, or cloud environments, aligning with Zero Trust standards.

4. Device posture and compliance enforcement

When paired with endpoint detection and response (EDR) solutions or network access control (NAC), IPsec validates patch levels, antivirus status, and encryption status before allowing tunnel establishment. Non-compliant or compromised devices are blocked from network access until they meet security requirements.

When integrated with identity access management (IAM) policies and continuous device posture validation, IPsec helps organizations achieve comprehensive Zero Trust security without modifying their existing applications. However, Gartner projects that 70% of new remote-access deployments will rely on Zero Trust Network Access (ZTNA) rather than VPN solutions.

How Group-IB Helps Organizations Secure IPsec Deployments

Even a well-configured IPsec tunnel can be compromised by vulnerabilities elsewhere in your network. Building a strong security posture begins with a layered defense strategy.

The first step is to identify your entire attack surface, including all IPsec-enabled hosts, vulnerable endpoints, and misconfigurations.

This comprehensive visibility allows you to monitor IPsec deployments and prioritize remediation efforts proactively. Group IB Attack Surface Management helps you achieve this by:

• Continuously scanning the entire Internet to uncover exposed VPN gateways, outdated IKE versions, and weak cipher suites associated with your digital footprint.
• Mapping relationships between IPsec endpoints, domains, and certificates so you can spot shadow IT tunnels and verify that every encrypted connection complies with corporate policy.
• Assessing risks or misconfigurations using real-time threat intelligence data and prioritizing it by business impact, enabling your team to patch or remediate vulnerabilities.

Complementing ASM, Managed XDR analyzes IKE and ESP flow metadata in real time. The platform can detect brute force attempts, cipher downgrade attacks, and abnormal tunnel behavior. It then correlates these findings with endpoint and log telemetry to quarantine compromised endpoints until verified by your SOC.

Get in touch with Group-IB experts today to learn how our solutions provide continuous confidence in the security and compliance of your IPsec deployments.