Beyond matchings: Dynamic multi-hop topology for demand-aware datacenters

State-of-the-art topologies for datacenters (DC) and high-performance computing (HPC) networks are demand-oblivious and static. Therefore, such network topologies are optimized for the worst-case traffic scenarios. However, recent optical circuit-switching technologies enable real-time dynamic topologies that change in microseconds or less. This capability opens the door for the design of self-adjusting networks: networks with demand-aware and reconfigurable topologies in which links can be re-adjusted online and in response to evolving traffic patterns. In this paper, we study self-adjusting networks using a recently proposed model of reconfigurable networks and present a novel algorithm, GreedyEgoTrees (GET), that dynamically changes the network topology. While previous algorithms used a local perspective, GET takes a global view and greedily builds ego-trees for nodes in the network, where nodes cooperate to help each other. In contrast to recent proposals, GET is optimized for multi-hop routing, and we show that it has nice theoretical guarantees as a function of the demand's entropy. Empirical results also show that GET outperforms recently proposed algorithms (like static expander and greedy dynamic matching) and can significantly improve the average path length by up-to 65% for real DC, HPC, and other communication traces.