We study the formation and evolution of domain walls with initial inflationary fluctuations by numerical lattice calculations, correctly taking into account correlations on superhorizon scales. We find that, contrary to the widely-held claim, the domain wall network exhibits remarkable stability even when the initial distribution is largely biased toward one of the minima. This is due to the fact that the domain wall network retains information about initial conditions on superhorizon scales, and the scaling solution is not a local attractor in this sense. Applying this result to the axion-like particle domain wall, we show that it not only explains the isotropic cosmic birefringence suggested by the recent analysis but also predicts anisotropic cosmic birefringence that is nearly scale-invariant on large scales and can be probed by future CMB observations.