Thinking about it some more I don't think we would need to abandon the whole TTY to get a good set of the features. What is basically required for a lot of the features is more communication between the shell and the terminal. There is already some communication for basic things - like raw mode and alternative buffers, colors and even images. These are how TUI programs like vim or screen/tmux function and how you can exit them without losing what was previously in the buffer.
I wonder if markers for the prompt and start/end of command output would probably enable a lot smarter virtual terminals with only some minor additions to the VT100 protocols. Possibly some extra data could be sent as well - like optional tooltip data maybe or even supporting actions that when the user clicks something it can send a response back to the shell. Maybe like a retry button on previous commands for example.
There is quite a lot that could be done if the terminal and shells had better protocals to communicate between each other. I dont think these will change overnight though so seeing terminal emulators try out these features to find what people like/want to use IMO is a good thing to see where we can take things in the longer term.
Would we have to abandon SSH or always X forward
No we would not. At the end of the day a TTY is just a input and output pipe between the terminal and the program running on the shell with a specific protocal VT100 (or some bastardized version of that - looking at you xterm). This is what network protocals are as well - just with different protocals in play. So you can do a lot over that connection with changes to the protocals. No need for xforwarding at all.
Thinking about it some more I am not sure that we would have to go that far. Well not in the longer term - short term we might for experimenting with ideas. I think one of the bigger problems ATM is the terminal does not understand what a shell is or its component parts. Terminals just display characters and move the cursor around the screen and send keyboard and now mouse input back to the command they run. They are also aware of alternative buffers and raw input mode and know about echoing characters back the the screen.
If we extended the terminals to also understand some shell concepts like the prompt, commands being typed and output from the commands and gave the shell some markers it could send along with these (like we do with color information ATM) the terminal would be able to use these to change how it displays each part and would open up a lot of new an interesting features. Could even add things like tooltip support or actions on clicking some bits of the text.
I am starting to see these terminals as experimenting on what we features could be enabled if we were not stuck on the current VT100 protocals. Though if we ever get wider adoption and generalisation of these ideas backed back into the protocals will be another question to consider.