Well either the enderman or the skeleton would have to hit each other for a fight to start, however the skeleton would never be able to hit the enderman and therefore start the fight. Also enderman are passive mobs which only become aggressive if the player looks at them I think, although I'm not sure how they would react if the Wither were to attack it, as the wither is aggressive to every mob automatically.
