phy: Fix u-boot coruption when fixed-phy is used