I've got a few raspberry pi's kicking around. The newest one - the Pi 4 - is actually a really good machine! I'm currently building a robot car with a Pi3 with my son. I think using a Pi 3 for a 'desktop' would be possible, but maybe a little clunky.
I agree that using a phone for day to day activities is tough. The newer Samsung phones have the "Samsung Dex" interface which, when plugged into an HDMI monitor, can be used as a full-fledged desktop. It's neat and can be used in a pinch if it's available to you.