Would this be broken?

nope
skills like bonfire, glacies etc which adds damage based on stats will tear through the def and res effortlessly

Well, yeah... If *both* stats are at such a point that the only way to breach it is to have a hard counter (so high attack unit w/ Atk boon + TA), then unless the character has Odin/Henry level attack (and even then if they're practically impossible to land a killing blow they could have a use in a comp with Ignis/Glacies) they'd be stupid, given that - assuming they're following standard BST ruling - all those left over points of HP are going to go into Attack and Speed. At the end of the day this is all hypothetical though, given that the unit will never exist.

edit: Or, alternatively, the enemy can just use flat-damage against you and kill you regardless, as the person above me pointed out. I'm a dumbass.

Or better, no stat anywhere except HP.
Then you equip Reprisal. Massiv retaliation damage based on missing HP :o
Zephiel's reprisal is such a pain !

Moonbow/Luna/Black Luna and this unit is dead

My understanding is that Wo Dao and Wrath deal true damage, so that's another counter.