Refined weapon on 4 star?

If I put a weapon on a 4 star unit that can be refined do I have to wait until the unit itself is 5 star or can I refine the weapon immediately?