Theoretical framework for higher-order quantum theory