The party is losing votes to both the left and the right.