Dynamic compression rate over TCP connections

Advisors

–* Henry-Joseph Audéoud, LIG / POLARIS — henry-joseph.audeoud@univ-grenoble-alpes.fr
–* Etienne Dublé, LIG / PIMLIG — etienne.duble@imag.fr

Context

Compression is a well-known technique allowing to increase the available bandwidth on slow networks.

While working on the implementation of a Network File System, we have designed a preliminary algorithm allowing to modulate the compression level given the available network bandwidth:

• in case of a low bandwidth, it makes sense to compress data to increase applicative bandwidth;
• in case of a high bandwidth, it does not make sense to spend CPU time for compression;
• in case of intermediate bandwidth value, the compression level should be fine tuned.
• in case of high CPU load (either from other instances of such connections or from another independent CPU load), the compression level may be carefully decreased.

The current algorithm is based on zstd compression. It tries to select an appropriate compression level by considering the trade-off between CPU and network usage. Compressing more makes sense when network is the bottleneck, and compressing less makes sense when CPU is the bottleneck.

Research objective

The objective of this internship is to develop an algorithm of dynamic compression (probably based on current development), to explore the gain we can have on exploiting such a technique. The tests and results should compare to the state of the art, define performance objectives and context parameters. Ideally, the intern would also study and improve the algorithm regarding the impact of its internal parameters, possibly find a way to auto-tune them, and study the impact of simultaneous connections.

Work items

The work will involve:

1. exploring the state of the art on static & dynamic compression over TCP connections
2. according to this exploration, define a simple experimental test bed to test the efficiency of a compression over TCP connections
   • define the parameters of the links on which the tests has to be done (real bandwidth, latency...),
   • define type of data to test transmission on,
   • define performance criterion (CPU load, payload bandwidth...),
   • a simple Proof-of-Concept program to dynamically adjust the compression rate to the available bandwidth,
3. analyse the results and compare them to the state of the art.

The work may then extend on some related topics, to discover where it may be improved:

4. How may the algorithm adapts to CPU load (to deal with many links compressing together)?
5. Can we fine-tune the internal parameters of the algorithm (such as fast adaptation to a change in available bandwidth)?

References

• Discussions about adapting zstd compression to bandwidth:
  • https://github.com/facebook/zstd/issues/75
  • https://github.com/facebook/zstd/pull/1327