Is Oliver worth making a 5* and building up

If you absolutely need a 5* blue mage then go for it. Otherwise, there are plenty of great blue mages in the game.
Honestly, I don't see myself using this unit. Just atk ploy fodder and maybe mirror strike fodder.

He’s too beautiful to kill. Build him and +10 him in time.