Symmetric Spaces for Machine Learning

Representation learning embeds data, often a graph, into a geometric space where the distances mean something. Euclidean space is good at grids, because it has families of equidistant lines, and bad at trees, because its volume grows only polynomially. Hyperbolic space is exactly the reverse. Real data has both kinds of structure at once, and by 2020 the machine learning community had begun stacking Euclidean and hyperbolic factors together to cope. In a collaboration with Anna Wienhard’s group in Heidelberg and Michael Strube’s group at HITS, the idea was to use a body of classical geometry in which Euclidean and hyperbolic pieces already live side by side: Riemannian symmetric spaces.

Symmetric spaces contain flat subspaces and hyperbolic subspaces side by side, and their enormous symmetry groups make them computable. Our ICML 2021 paper proposed using them systematically and took two tools from their structure theory that had never been used for embeddings. The first is Finsler metrics. In a graph there are usually many shortest paths between two vertices, and the $\ell^1$ metric on a flat, which a symmetric space supports alongside its Riemannian metric, has the same property. We optimize with the Riemannian structure and measure with the Finsler one. The second is the vector-valued distance. In Euclidean or hyperbolic space the only invariant of a pair of points is a number. In a symmetric space of rank $n$ it is a vector in a Weyl chamber, and that extra information can be read back out of a trained embedding: coloring edges by the angle of their distance vector separates the tree-like parts of a graph from the grid-like parts, though the model was never told which was which. We implemented all of this in Siegel spaces, which had not appeared in geometric deep learning before, and beat the constant-curvature baselines on graph reconstruction.

The NeurIPS 2021 paper does the same for the space of symmetric positive definite matrices, which is cheaper to compute in. Here the vector-valued distance between $P$ and $Q$ is nothing more than the logarithms of the eigenvalues of $P^{-1}Q$,

$$d_{vv}(P,Q) = \bigl(\log \lambda_1(P^{-1}Q),\ \dots,\ \log \lambda_n(P^{-1}Q)\bigr),$$

and the Riemannian distance is its Euclidean norm, the Finsler distances its other norms. We also developed a gyrocalculus on the space, closed-form analogs of addition, scalar multiplication, rotation and reflection, so that architectures written for Euclidean space can be translated more or less symbol for symbol. The models outperform their Euclidean and hyperbolic counterparts on knowledge graph completion, recommendation, and question answering. The ECML 2023 paper, led by Wei Zhao, builds full graph neural networks in this space and shows large gains on graphs with mixed structure, with a library that trains five standard architectures in Euclidean, hyperbolic and SPD geometry.