IPSEC outline. RFC2401
IPSEC is a set of extensions to the IP protocol family. It works pretty much the same for both IPv4 and IPv6. It provides two basic services, and a large number of variants on them.
The two services are:
Authentification means that you can be sure that the data came from who it says it came from. Verification means that you can be sure that it hasn't been altered.
Confidentiality means that the contents are not visible to third parties, even if they have access to the data in transit.
These two services are considered distinct, but the IPSEC framework supports them in a uniform manner.
On the wire format.
Authentication is achieved by the addition of an Authentication Header (AH) which comes after the basic IP header and contains cryptographically secured Hashes of the data and Identification information. The hashes can also cover the invariant parts of the IP header itself. There are several differnt RFCs giving a choice of actual algorithms to use in the AH, however they all must follow the guidlines specified in RFC2402.
Confidentiality is achieved through the addition of an Encapsulating Security Payload (ESP) header, and the possible rewriting of the payload in encrypted form. The ESP header does not consider the fields of the IP header before it and therefore amkes no guarantees about anything except the payload. The various types of ESP applicable must follow RFC2406. An ESP header can also provide Authentication for the payload, (but not the outer header).
An orthogonal (mostly) division of IPSEC functionality is applied depending on whether the endpoint doing the IPSEC encapsulation is the original source of the data or a gateway:
IPSEC secured links are defined in terms of Security Associations (SAs). Each SA is defined for a single unidirectional flow of data, and usually (ignoring multicast) from one single point to another, covering trafic distiguishable by some unique selector. All traffic flowing over a single SA is treated the same. Some traffic may be subject to several SAs, each of which applies some transform. Groups of SAs are called an SA Bundle. Incoming packets can be assigned to a particular SA by the three defining fields, (Destination IPaddess, Security Paramiter Index, Security protocol) SPI can be considered a cookie that is handed out by the reciever of the SA when the parameters of the connection are negotiated. and the protocol must be either AH or ESP. Since the IP address of the reciever is part of the tripple, this is a guaranteed unique value. They can be found from the outer IP header and the first security header (which contains the SPI and the protocol).
An example of a tunnel mode AH packet is:
[IPHDR][IPoptions][AH][IPHDR2][IPoptions][TCP][data]
An example of a transport mode AH packet is:
[IPHDR][IPoptions][AH][TCP][data]
Because an ESP header cannot authenticate the outer IP header, it is useful to combine an AH and an ESP header to get the following:
[IPHDR][IPoptions][AH][ESP][TCP][data]
This is called Transport Adjacency. The tunnelling version would look like:
[IPHDR][IPoptions][AH][ESP][IPHDR2][IPoptions][TCP][data]
However it is not specifically mentionned in the RFC. As with Transport adjacency, this would authenticate the entire packet except a few headers in the IP header and also encrypt the payload, (seen in italics). When an AH and an ESP header are diectly applied together like this, the order of the headers should be as shown. It is possible in tunnel mode, to do arbitrary recursive encapsulation so that order is not specified.
Administrative Interface. (Sample implementation)
How the IPSEC systems and gateways are configured is to some extent left to the designer, however the RFC has some strong recomendations as to how this should be implemented, so as to minimise confusion.
There are two administrative entities that control what happens to a packet. One is the Security Association Batabase (SAD) and the other is the Security Policy Database (SPD). Each interface supporting IPSEC should have a logically separate SAD and SPD.
They are similar in that given a number of selectors that describe some traffic, they will deliver an entry that describes the processing needed. however the SPD is two steps removed from the actual processing. The SPD is used for outgoing packets, to decide what SAD entries should be used, and the SAD entries in turn describe the actual process and the parameters for it. The SPD entries specify the already created SAD entries to use (if it's a bundle there can be more than 1), but if there is not already a suitable one, it is used to create new ones. The fields of the SA being created can be taken either from the SPD entry or from the packet that initialted the creation.
Outgoing packets go from the SPD entry to the specific SA to get encoding parameters. Incoming packets get to the correct SA directly using the SPI/DEST/Proto tripple, and from there get to the SPD entry.
The SPD can also specify what traffic should bypass IPSEC and what should be dropped, so it must also be consulted for incoming non-IPSEC traffic. SPD entries must be explicitly ordered as several might match a particular packet, and the processing must be reproducible.
The SPD can be though of as similar to a packet filter where the actions decided upon are the activation of SA processes. Selectors can include src and dest address, port numbers if relevant, application and user IDs if available (only on host based transport SAs), hostnames, security sensitivity levels, protocols etc.
An SAD entry would include:
An SPD entry would contain:
Each SA can define one ESP header and one AH header. so an IPSEC session must have one or the other or both, but cannot be defined with neither, otherwise there would be no headers to specify the SPI to look up the SA. The RFC doesn't say what would happen if the AH and ESP headers dissagree about the SPI value. One would presume this would imply multiple SA's in a bundle.
These values and SAD entries should be addable by hand by some (implementation dependent) interface, however the IETF has also defined automatic mechanisms for initialisation of sessions and such things as key exchange. These are defined in RFC2407, RFC2408, and RFC2409
IKE
IKE is an out-of-band mechanism for distributing Key information between IPSEC endpoints. As such it is not part of the IPSEC specification. It is specified in RFC2409 .
For newer information see RFC7296