They may not have much money left over after downsizing, however. In markets like California the value is almost entirely from the land. The comparative value of the house (even a 5 bed) is comparatively negligible. So a 5 bed on a small plot would cost almost the same as a 1 bed on the same plot. In Silicon Valley it's really common to just buy a house and knock it down and rebuild, since the cost of building a new house is much less than the cost of the land.
It's a genuine issue that the liquidity of the real estate market is impacted by this.
I mean, the coasts aren't really that densely populated. If we build nicer cities there would be plenty of potential space.
Instead we build shitty suburbs and sprawl, which will always lead to awful, expensive cities