OEMs have quite a lot of extra steps before releasing any build to the public.
They have to pass xTS, the set of test suites required before getting certified by Google, possibly carrier certification, regulatory requirements and more depending on where the build will be released.
There are "quicker" release channels for security fixes, but I don't think it's common for OEMs to only ship those without any other change to the system.
I don't think Graphene does anything of sort, they take what's already certified in the Pixel builds and uses it.
Not like they could do much aside testing on the public part of xTS.
> OEMs have quite a lot of extra steps before releasing any build to the public.
AIUI updates are less stringent and burdensome than initial certification. Regardless much of the process is automated. Graphene has CI too. 3PL's taking 4 weeks to run automated tests is also absurd. There are some "manual steps" to run CTS-V but they shouldn't be weeks level burdensome either. This is the point, this is an industry problem.
The reason that the OEMs even have to deal with this 3PL test mess is for GMS certification, so again this is a policy decision that enforces a poor process. The bad properties of the process are not inherent to the problem space of validating builds against requirements. An industry problem.
> There are "quicker" release channels for security fixes, but I don't think it's common for OEMs to only ship those without any other change to the system.
Seems like a decision that is not user-centric.
> I don't think Graphene does anything of sort, they take what's already certified in the Pixel builds and uses it. Not like they could do much aside testing on the public part of xTS.
Private test suites for software are a toxic idea, it's in the same box as "SSO tax", and other such "pay for security" models. Given the software industry can't be trusted not to do this, I'm almost keen to see legislation to explicitly ban this practice.
Android CTS and VTS are open source so we can and do use those. They're filled with flaky and badly made tests along with enforcing anti-privacy and anti-security design decisions though, so not everything is supposed to pass. Google likes to enforce that OEMs aren't allowed to make certain kinds of privacy and security improvements which could impact app compatibility until Google decides to do it themselves in new major Android versions with new API levels forcing app developers to deal with it.
They don't allow adding our Network and Sensors toggles which are detected as modifications to the permission model. They don't detect Contact Scopes and Storage Scopes but they might be considered Compatibility Definition Document violations. We don't worry about this, our focus is passing the tests which are actually relevant including the ones we've added for duress PIN, hardened_malloc, our more advanced hardware memory tagging integration that's always on, etc.
If we wanted to get access to the proprietary GTS for Google Mobile Services to see how much sandboxed Google Play passes, we could, but we focus on real world app compatibility.
> AIUI updates are less stringent and burdensome than initial certification
That's true having dealt with some of it, nonetheless I haven't found that much of a difference due to having to use 3PL.
There's more manual steps on top of CTSV for camera and GMS, but that's all there is to it.
The only real difference I've seen is on Google's side to actually say "ok" before it getting approved.
Carriers and regulations are better on that side, but assume you have a security fix in the modem, for some carriers you're supposed (emphasis here) to redo it...
> Seems like a decision that is not user-centric.
I can see how having two release channels one solely for security and a bigger one might be a burden on some.
But you hardly want to only fix security issues when you have a real bugfix you want to also release, so it makes sense to me the channels have to be merged.
> Private test suites for software are a toxic idea
To be fair on android side they're quite fine.
One is specifically for GMS compliance, one for camera verification, and one for security patches verification.
The latter is janky and not as updated as you'd think, so unless you really forget to apply patches it'll pass.
With that said, the amount of people running those test suites not for certification can probably be counted on a single hand, I think that's the least of the problems.
We'll have the same update pace for security updates and major releases with the devices we're working on with our OEM partner. That's not specific to Pixels. It will in fact be easier to support the devices with the OEM partner due to them planning on doing most of the device support work including getting MTE working properly. For Pixels, we have to do a lot of work on device support, while for non-Pixels that work is going to be done for us. Our OEM partner is actively getting what's needed from Qualcomm including getting them to fix things. We're in direct contact with Qualcomm ourselves and plan to deploy new security features they've developed which are not yet available elsewhere.
Samsung and Google ship a small subset of the security preview patches early while we're shipping all of them. We're doing a lot of work to integrate and test those. We also have to port them from Android 16 to Android 16 QPR1 and now Android 16 QPR2. It seems they might start providing them for Android 16 QPR2 themselves but for now we had to port them for our QPR2 releases.
We also have to test and fix all the issues caused by us having much more advanced exploit protections including full system hardware memory tagging with a more advanced implementation. We uncover MANY upstream memory corruption bugs we need to fix. Features like Contact Scopes, Storage Scopes, 2-factor fingerprint authentication, etc. are not always easy to port to new versions. We still don't have early access to upcoming quarterly and yearly releases but we'll get it and then we can have day 1 updates for those instead of it taking days for an experimental release and around 1-2 weeks before it reaches the Stable channel. We intend to do much better than we are now, we just need the same early access OEMs have but don't actually use to make day 1 releases for major OS updates.
Hopefully you don't mind me asking this question, but didn't you work with people who managed to do exactly what you are suggesting with a fairly small team at Essential for a few years?
Yep. And GrapheneOS's changes to the kernels of devices they ship are laughably small, 20-30 commits at most. I don't think they even do any basic CVE checks on any of the source code.
Fuzzing, actual security analysis - all those things are done by Google.
Their contributions upstream go way back, I think someone could misread this comment that they've not contributed, and that would be an unfortunate misunderstanding.
GrapheneOS has made substantial upstream contributions to the Linux kernel and Pixel drivers including vulnerability reports. Many of our kernel changes are for the out-of-tree drivers needed for Pixels which are in a separate repository from the Generic Kernel Image code from the upstream Linux kernel. We make important downstream changes including enabling many more of the upstream security features and adding important protections not yet available there. We worked with multiple upstream Linux kernel developers to get many of the changes we used to have upstream and therefore no longer need them. We have major kernel security improvements in development including more security-focused integration of hardware memory tagging, but indefinitely maintaining those downstream is not the way we try to do things.
We use much newer Generic Kernel Images than the stock Pixel OS as the base. Android 16 QPR2 was released this month and they finally shipped 6.1.145 from July 2025 for the Pixel 6 through Pixel 9 compared to us being on 6.1.158 which was the latest until yesterday (6.1.159) which will be incorporated soon. It's similar for our 6.6 and 6.12 branches compared to theirs. 6.6 is the current Pixel 10 and near future Pixel 6 through Pixel 9 branch. They only update the kernel revision every 3 months in quarterly/yearly releases so this is the smallest the delay gets right after a quarterly release. They'll still be on 6.1.145 until the next major release in March 2026 so the current delay of having the July 2025 kernel in December 2025 is not representative but rather is the small side of the delay. Shipping the newer LTS revisions is not easy due to frequent regressions both in the upstream code and to a much lesser extent in the out-of-tree drivers needed for Pixels which often need small changes to adapt them to the new LTS revisions.
GrapheneOS does a lot of deep security analysis and has proposed firmware, kernel and userspace exploit protections adopted by Google. We helped them get a bunch of vulnerabilities being exploited in the wild blocked off as whole classes of vulnerabilities including perf events, reset attacks on fastboot mode and much more. GrapheneOS is focused on addressing classes of vulnerabilities rather than individual bugs. Google puts a decent amount of resources into finding and fixing individual bugs and that isn't our focus. We get the bug fixes from the upstream project many months earlier and the Pixel driver fixes from them other than cases we fix them early due to finding them with hardware memory tagging which they don't use for the kernel even in Advanced Protection mode (or most of the base OS processes either, while we always use it for both with a much better implementation in userspace).
Most of our changes are in userspace where we don't try to collaborate with upstream developers as much as we do with the Linux kernel. Most of userspace is not developed as openly in a way we can properly collaborate.
OEMs have quite a lot of extra steps before releasing any build to the public.
They have to pass xTS, the set of test suites required before getting certified by Google, possibly carrier certification, regulatory requirements and more depending on where the build will be released.
There are "quicker" release channels for security fixes, but I don't think it's common for OEMs to only ship those without any other change to the system.
I don't think Graphene does anything of sort, they take what's already certified in the Pixel builds and uses it. Not like they could do much aside testing on the public part of xTS.