You could buy the same modem that is in the pinephone in mini pcie format and use a usb adaptor to connect it to a pi.
I think a project that uses the pi cm4 format and adds a modem, screen, buttons, speaker and bms to be a compelling open-source phone.
Have a look at lvra.gitlab.io. It should be possible to get the rift s mostly working.