ShockHash: Towards OptimalSpace Minimal Perfect Hashing Beyond BruteForce

Author(s):
HansPeter Lehmann, Peter Sanders, Stefan Walzer
 Source:
 Date: August 2023

A minimal perfect hash function (MPHF) maps a set S of n keys to the first n integers without collisions. There is a lower bound of n*log(e)O(log(n)) bits of space needed to represent an MPHF. A matching upper bound is obtained using the bruteforce algorithm that tries random hash functions until stumbling on an MPHF and stores that function’s seed. In expectation, e^n*poly(n) seeds need to be tested. The most spaceefficient previous algorithms for constructing MPHFs all use such a bruteforce approach as a basic building block.
In this paper, we introduce ShockHash – Small, heavily overloaded cuckoo Hash tables. ShockHash uses two hash functions h_0 and h_1, hoping for the existence of a function f : S > {0, 1} such that x > h_{f(x)}(x) is an MPHF on S. In graph terminology, ShockHash generates nedge random graphs until stumbling on a pseudoforest – a graph where each component contains as many edges as nodes. Using cuckoo hashing, ShockHash then derives an MPHF from the pseudoforest in linear time. It uses a 1bit retrieval data structure to store f using n + o(n) bits.
By carefully analyzing the probability that a random graph is a pseudoforest, we show that ShockHash needs to try only (e/2)^n*poly(n) hash function seeds in expectation, reducing the space for storing the seed by roughly n bits. This makes ShockHash almost a factor 2^n faster than bruteforce, while maintaining the asymptotically optimal space consumption. An implementation within the RecSplit framework yields the currently most space efficient MPHFs, i.e., competing approaches need about two orders of magnitude more work to achieve the same space.