VPN is a virual private network which creates a secure tunnel over a public network. It can be

– client to firewall,

– router to router,

– firewall to firewall.

VPN uses the internet as the public backbone to access a secure private network. For example, remote employees can access their office network

VPN Protocols can be listed as follow,

– PPTP ( point to point tunneling protocol ) developed by Microsoft

– L2F ( layer 2 forwarding protocol ) developed by Cisco

– L2TP ( layer 2 tunneing protocol ), IETF combined Microsoft + Cisco, work in data link layer

– IPSec ( Internet Protocol Security ), IETF open standard vpn, work in network layer, generally done in routers.

So, IPSec provides Layer 3 security (RFC 2401) and works in tunnel mode or transparent mode.- The components of IPSec are security association (SA), authentication header(AH), Encapsulating Security Payload( ESP), and Internet Key Exchange (IKE), ISAKMP ( Internet Security Association Key Management Protocol ) is a framework for IKE and authentication.

We need IPSec because IP (Internet Protocol) is not secure, IP protocol was designed in the early stages of the internet where security was not an issue. And there are possible security issues like

  1.  source spoofing
  2. replay packets
  3. no data integrity or confidentiality

IPSec standards are defined in following RFCs, RFC 4301 IP Security Architecture, 4302 AH, 4303 ESP, 2408 ISAKMP, 5996 IKE v2, 4835 cryptographic algorithm implementation for ESP and AH.

The main benefits of implementing IPSec are,

1. Confidentiality – by encrypting data

2. Integrity – routers at each end of a tunnel calculates the checksum or hash value of the data

3. Authentication – signatures and certificates, all these while still maintaining the ability to route through existing IP networks

4. data integrity and source authentication

5. anti-replay protection

6. key management

IPSec works in two modes viz,

– tunnel mode:

  1. entire IP packet is encrypted and becomes the data component of a  new ( and larger ) IP packet.
  2. frequently used in an IPSec site to site VPN

– transparent mode:

  1.  IPSec header is inserted into the IP packet
  2. no new packet is created
  3. works well in networks where increasing a packet size could cause an issue
  4. frequently used for remote-access VPNs

IPSec Architecture


Security Associations and ISAKMP

  1. SA is a collection of parameters required to establish a secure session
  2. SA is uniquely identified as SPI, IP destination address, security protocol ( AH or ESP) identifier
  3. SA is unidirectional, two SA required for a bidirectional communication
  4. a single SA can be used for AH or ESP, but not both, must create two or more SA for each direction if using both AH and ESP
  5. Security Association Database ( SAD) ,  Security Policy Database (SPD)
  6. ISAKMP is internet security association and key management protocol defined by RFC 2408
  7. used for establishing security associations and cryptographic keys
  8. only provides the framework for authentication and key exchange, but key exchange independent


How to set up an SA

  1. Manually – manual keying
  2. Automatically – using IKE (internet key exchange)

Authentication Header (AH)

  1. provides source authentication and data integrity, protection against source spoofing and replay attacks
  2. If both AH and ESP are applied to a packet, AH follows ESP
  3. AH and ESP used both for authentication and integrity, but AH does not support encryption and ESP does support encryption.
  4. AH operates on top of IP using protocol 51

Encapsulating Security Payload (ESP)

  1. uses IP protocol 50
  2. provides offered by AH , plus data confidentiality, uses symmetric key encryption
  3. must encryption and or authenticate in each packet, encryption occurs before authentication
  4. authentication is applied to data in the IPSec header as well as the data contained as payload


Internet Key Exchange (IKE)

  1. used for mutual authentication and establishing and maintaining security associations, RFC 5996
  2. used for establishing IPSec sessions
  3. a key exchange mechanism
  4. five variation of an IKE negotiation, two modes (aggressive and main modes), three authentication methods (pre-shared, public key encryption and public key signature)
  5. uses UDP port 500

Phase I

  1.  establish a secure channel ( ISAKMP SA)
  2. using either main mode or aggressive mode
  3. authenticate computer identity using certificates or pre-shared secret
  4. It’s a three two-way exchanges between the initiator and receiver.
    • First exchange—The algorithms and hashes used are agreed upon in matching IKE SAs in each peer.
    • Second exchange—This exchange uses a Diffie-Hellman exchange to agree upon a shared key. Diffie-Hellman is not an encryption algorithm but an algorithm for establishing a shared key over an unsecured medium.
    • Third exchange—This exchange verifies the other side’s identity. The identity value is the IPSec peer’s IP address in encrypted form.

Phase II

  1. establish a secure channel between computers intended for the transmission of data (IPSec SA)
  2. using quick mode

IPSec Best Practices

  1. use ESP option
  2. use  3DES and AES instead of DES
  3. use SHA instead of MD5 as a hashing algorithm
  4. enable perfect forward secrecy to reduce lifetime of SA for highly sensitive data

That’s it. Cheers!!