We show how the weakly trapped Harper-Hofstadter model can be mapped onto a Harper-Hofstadter model in momentum space. In this momentum-space model, the band dispersion plays the role of the periodic potential, the Berry curvature plays the role of an effective magnetic field, the real-space harmonic trap provides the momentum-space kinetic energy responsible for the hopping, and the trap position sets the boundary conditions around the magnetic Brillouin zone. Spatially local interactions translate into nonlocal interactions in momentum space: within a mean-field approximation, we show that increasing interparticle interactions leads to a structural change of the ground state, from a single rotationally symmetric ground state to degenerate ground states that spontaneously break rotational symmetry. 