The former work and pensions secretary said the party had ‘substantially lost the trust of the British people’.