The Berry curvature is a geometrical property of an energy band which acts as a momentum space magnetic field in the effective Hamiltonian describing single-particle quantum dynamics. We show how this perspective may be exploited to study systems directly relevant to ultracold gases and photonics. Given the exchanged roles of momentum and position, we demonstrate that the global topology of momentum space is crucially important. We propose an experiment to study the Harper-Hofstadter Hamiltonian with a harmonic trap that will illustrate the advantages of this approach and that will also constitute the first realization of magnetism on a torus. 