Don't ignore the fact that an independent firmware like GrapheneOS is also based on Android's modified base kernel. So yeah it's clean from all that Google proprietary oversight but that doesn't mean there's some kind of magic that makes your phone impervious to malware. Keep in mind that there are a continuous number of revelations about newly discovered exploits and compromises for both software (operating system and apps) and hardware vulnerabilities. These could involve vulnerabilities are relatively new, or some going back for years. So some of these could be something that's been used against us for years and only now discovered. The point is, there are a lot of potential weak points -- there are literally millions of lines of code that make up the Android operating system and all it takes is one string of that code to interact with a string of code in an app in a certain way to open up an exploit. Or there's some vulnerability in a commonly used processor chip that affects millions of phone models and requires a firmware patch (software). There are any number of variables involved so it's mind-boggling to sort any of them out.
Basically, there are any number of compromises involved with any online interaction. You can dismiss security and privacy issues as irrelevant and just keep using your Pixel 3 as is, and odds are you'll be fine. Just keep being vigilant and use common sense, that's apparently working for you. But don't be too surprised if one day you find you are a victim of identity theft or you find your phone is one of the bots in a DDoS attack, it's a matter of possibility or probability.