libtins v4.0 release

libtins is a high-level, multiplatform C++ network packet sniffing and crafting library. In this post I’ll be covering the major changes that were made in the v4.0 release of the library.

This was a major release that fixed several features that either were wrong or could have been implemented in a better way. These changes led to considerable performance improvements on several protocol’s parsing code, making libtins slightly faster than it already was.

This version broke the ABI compatibility, which means you should rebuild your project when updating your libtins reference. The API, on the other hand, wasn’t broken except on a couple of spots which hopefully won’t affect anyone.

In the following sections we’ll look into what was changed and the reasoning behind those changes.

Performance improvements

One of the changes that I’m most happy about in this release are the ones that led to performance gains. Several small changes provided a considerable speed improvement when parsing packets.

The changes that were made and helped achieve this were the following:

Ban std::list as the storage for options

Up until this versiom, std::list was used to store options in PDUs like TCP, IP and DNS. This container has now been replaced with std::vector, which proved to make parsing those protocols considerably faster. I’m not completely sure why std::list was used initially as it’s normally a poor choice of a container except for very specific cases. When there are options to be parsed, inserting elements into a std::vector outperforms std::list as it doesn’t need to allocate memory every time a new option is added.

The TCP + options benchmarks is a good test case for this. When comparing the latest version to the previous release, I noticed the former performs about 25% faster than the previous one.

Note that this shouldn’t break your code unless you were explicitly doing something like:

const TCP tcp = ...;

// Ooops, this will fail
const list<TCP::option>& oops_options = tcp.options();

But this is easily fixable by using TCP::options_type:

// Fixed!
const TCP::options_type& yay_options = tcp.options();

Total option size tracking

The total option size tracking in IP and TCP was completely removed and now both the size and required amount of padding are only calculated when serializing a packet. These were basically wasting space and CPU time as there is really no reason for themn to be pre-calculated. Given that there’s only at most a handful of options in every packet, it’s really not worth the “optimization”.

The TCP benchmark performed about 13% better when using the latest version. This should be all attributed to this removal as there’s nothing else that was changed that could have caused a performance improvement.

PDU, I am your parent

Another change that was introduced is the notion of a parent for every PDU. So far there was only a notion of the “next” layer in a packet but there was no way to get the previous one in case it existed. This was okay for the most part except on some cases when while serializing a packet, a layer eeds to know what its parent is in order to compute some value. An example of this is TCP, which uses the packet’s IPv4/IPv6 addresses while computing its checksum.

As a consequence of this, a pointer to the parent PDU was being passed along while serializing a packet which made for a weird API. Now there’s no need to have that argument as each PDU knows at all times what its parent is. Note that this probably added a slight performance penalty due to the fact that the base size for every PDU is now 8 bytes (in x64) larger but it is probably negligible compared to the other optimization improvements performed.

After adding this I figured it wouldn’t be a bad idea to implement a PDUIterator class that represents a bidirectional iterator over every PDU present in a packet. This makes it slightly easier to go through each of them without having to deal with pointers at all:

const PDU packet = ...;

// Let's iterate all PDUs in this packet
for (const PDU& pdu : iterate_pdus(packet)) {
    // Print the type of this PDU as a string
    cout << Tins::Utils::to_string(pdu.pdu_type()) << "\n";
}

Compilation time improvements

A significant amount of effort was put into making libtins build faster, both while building library itself and any application that uses it.

In order to achieve this, some headers like internals.h and utils.h were split into multiple files. These had started just containing a couple of functions and classes but grew too much over time and ended up having a considerable amount of code and pulling in several header files.

Comparing version 4.0 with the previous one, 3.5, I can see the latest one builds 17% faster in my local machine. This number is fairly relative but any user of the library should see some sort of build time improvement when building their project.

RadioTap parsing/serializing

The parsing and serializing of RadioTap was initially implemented in a really awful way. The format of his protocol is tricky, which makes it non-trivial to parse. The initial approach involved basically hardcoding the parsing and storing of every individual field, keeping every one of them on a separate member using the storage type for them. This meant that every time the parsing of a new field was added, it would break the ABI as the member in which the field was stored would be added on demand.

In version 4.0 this was re-written from scratch using a generic parser (RadioTapParser) and writer (RadioTapWriter). The entire portion of the RadioTap header that contains the options is stored in a std::vector and only parsed on demand. This means that if you don’t care about using RadioTap options at all then you won’t be paying for the price to parse them.

This also allows iterating and processing every option in an easy way:

RadioTap radio = ...;

// Construct a parser around the buffer where the options are stored
RadioTapParser parser(radio.options_payload());

// Now iterate through all fields
while (parser.has_fields()) {
    cout << "Option type: " << static_cast<int>(parser.current_field()) << "\n";

    // Fetch the current option
    const RadioTap::option option = parser.current_option();

    // Now we can convert this option to the right type
    // using RadioTap::option::to<T>, which will obviously
    // will depend on the type of option it is

    // Advance to the next field
    parser.advance_field();
}

Note that as a consequence of the current design, it is pretty inefficient to fetch fields individually as each of them will involve an O(N) search over the entire options buffer. If you need to fetch multiple fields, just iterate through the payload like shown above and process them.

Removing HWAddress storage template parameter

HWAddress had a second template parameter (besides the one used to store the length of the address) that indicated which storage type to use for each element stored (defaulting to uint8_t). This is not really useful as I’ve yet to found a case where I’d like to store a hardware address where each element is larger than a byte. This was also requiring to include some “heavy” headers in hwaddress.h which slowed down compilation times considerably for any translation unit that included it.

This template parameter has now been removed and the storage is always uint8_t. This will break your code if you were forward declaring the class. If this is the case, I’m sorry but that had to go away for good.