Fixed-t subtracted dispersion relations are presented for Compton scattering off the nucleon at energies Eγ<~500 MeV, as a formalism to extract the nucleon polarizabilities with a minimum of model dependence. The subtracted dispersion integrals are mainly saturated by πN intermediate states in the s channel, γN⃗πN⃗γN, and ππ intermediate states in the t channel, γγ⃗ππ⃗NN̅ . For the subprocess γγ⃗ππ, we construct a unitarized amplitude and find a good description of the available data. We show results for Compton scattering using the subtracted dispersion relations and display the sensitivity on the scalar polarizability difference α-β and the backward spin polarizability γπ, which enter directly as fit parameters in the present formalism. Double polarization observables are shown to have a unique potential for measuring the spin polarizabilities of the nucleon.