↓Skip to main content
Internships

Dynamic compression rate over TCP connections

Advisors #

Context #

Compression is a well-known technique for increasing the available bandwidth on slow networks.

While working on the implementation of a Network File System, we have designed a preliminary algorithm that modulates the compression level given the available network bandwidth:

  • when bandwidth is low, it makes sense to compress data to increase application-level throughput;
  • when bandwidth is high, it does not make sense to spend CPU time on compression;
  • at intermediate bandwidths, the compression level should be fine-tuned;
  • under high CPU load (from other such connections, or from an unrelated workload), the compression level may be carefully decreased.

The current algorithm is based on zstd compression. It tries to select an appropriate compression level by considering the trade-off between CPU and network usage. Compressing more makes sense when the network is the bottleneck, and compressing less when the CPU is.

Research objective #

The objective of this internship is to develop a dynamic compression algorithm, probably building on the current implementation, and to measure what such a technique actually gains. The results should be compared with the state of the art, and should establish performance objectives and the parameters of the context they hold in. Ideally the intern would also study how the algorithm’s internal parameters affect it, possibly find a way to auto-tune them, and study the effect of simultaneous connections.

Work items #

The work will involve:

  1. explore the state of the art on static and dynamic compression over TCP connections;
  2. from that, define a simple experimental testbed to measure how effective compression over TCP is:
    • define the parameters of the links the tests are to run on (real bandwidth, latency, and so on),
    • define the types of data to test transmission on,
    • define the performance criteria (CPU load, payload bandwidth),
    • write a simple proof-of-concept program that adjusts the compression rate to the available bandwidth;
  3. analyze the results and compare them with the state of the art.

The work may then extend to related topics, to find where it can be improved. How can the algorithm adapt to CPU load, so as to handle many links compressing at once? Can its internal parameters be fine-tuned — for instance to adapt quickly to a change in available bandwidth?

References #