Demystifying the processes behind Android's Project Mainline updates.Whether you're navigating spontaneous device behaviors or simply curious about the mechanics of updates, a concise comprehensive guide to understanding the unseen activities that ensure our devices function optimally.
In my previous blog posts, we delved deep into the intriguing topic of why an Android Enterprise device seems to have a mind of its own and reboots unexpectedly. We also ventured into the practical side of things, equipping you with the know-how to export logs, and providing a clearer lens to understand the root causes of these seemingly inexplicable reboots. However, after frequenting various online communities and forums, it came to my attention that this peculiar behavior persists for many. A number of users continue to grapple with their devices' unpredictable reboot patterns.
Today, we are going to further unravel this mystery. Leveraging Project Mainline logs, we will embark on a journey to chronologically dissect the events that lead to these soft reboots, aiming to bring clarity to this enduring Android enigma.
Before diving deep into the intricacies of Project Mainline, I recommend familiarizing yourself with my previous blog posts that delve into this mysterious reboot behavior.
Intune - In Real Life
Intune - In Real Life
Android is an ever-evolving mobile operating system, and over the years, Google has introduced various mechanisms to streamline and expedite the process of delivering updates. One of the most pivotal of these mechanisms is Project Mainline.
Introduced with Android 10, Project Mainline enables Google to update specific internal components of the Android OS directly via the Play Store, without needing full system updates. This approach offers several benefits:
It is designed to distribute new Android features to users of various Android versions without requiring them to perform a complete OS update. To delve deeper into Project Mainline, let's first revisit its forerunner, 'Project Treble'
Android's extensive reach also brings with it some intricate challenges. Each manufacturer, despite rooting their systems in the AOSP source, often feels compelled to introduce their own unique alterations. This could be to ensure hardware congruence or to offer standout UI experiences and system applications. Some even delve into adjusting the system partition's contents or altering the AOSP sources directly. Such practices birthed a persistent Android challenge termed 'fragmentation'.
Fragmentation is the consequence when changes to AOSP are so profound that the universal AOSP build isn't compatible with specific devices. This places the onus on manufacturers to keep pace with AOSP updates, a daunting task considering the regular evolutionary strides Android takes annually. For end-users, this often translates to their devices becoming archaic, sometimes a mere 18 months post-purchase. Custom ROM developers often grapple with challenges, especially when they encounter unexpected hardware inconsistencies after tweaking the AOSP for specific devices. The ripple effect of this is most devices getting sporadic updates or, worse, none at all, leading to magnified security risks and delayed adoption of fresher Android versions.
Recognizing this, Google introduced "Project Treble" in Android 8.0 Oreo as a remedy. Its primary essence was rejuvenating Android's Hardware Abstraction Layer to bridge the gap between generic AOSP code and device-specific hardware. A salient aspect was drawing a clear demarcation in roles through an established partitioning blueprint:
AOSP code starts by looking for device-specific files, followed by brand-specific ones, then by the SoC, and finally resorts to its default files, which are universally compatible with all devices.
At its core, Project Mainline breaks down the essential components of the Android OS into distinct modules. Each module encapsulates a specific fragment of the system, paving the way for them to be updated individually without a full system overhaul.
One of the standout features of Project Mainline is the mechanism of Google Play System Updates. Gone are the days when users would be at the mercy of device manufacturers for timely system updates. Instead, with Mainline, these modular updates are funneled directly through the Google Play Store, akin to regular app updates. This not only accelerates the pace at which users receive these updates but also ensures they are consistent and regular.
Security, a cornerstone of any operating system, receives a notable boost with Mainline. By facilitating rapid deployment of security patches and other imperative updates, the window of vulnerability that often arises due to manufacturer-induced delays is considerably minimized.
A longstanding challenge that Android has grappled with is fragmentation. This refers to the disparate versions of the OS and updates across the myriad of Android devices. Furthermore, a hallmark of Project Mainline is its staunch commitment to compatibility. The modular design ensures that even if devices have divergent versions of a module, the user experience remains unaffected, with apps operating seamlessly across the board. The link below lists all the modules, their package name, formats (APK or APEX) when they were introduced, and their features:
Android Open Source Project
I won't delve deeply into the details of Project Mainline for this blog. Instead, we'll center our attention on the delivery process of these updates and their impact on the device.
The complex process begins with the modularization of Android. The operating system's components are compartmentalized into distinct modules, each responsible for a specific system function. These modules are either encapsulated as APKs (Java-based components) or as APEX (native components). When a particular Mainline module necessitates an update, it is meticulously packaged in its requisite format, be it APEX or APK, and subsequently propagated to devices through the Google Play Store's distribution framework.
As a workaround, Android users are advised to go to Settings > Security > Google Play system update. This initiates the Mainline module update process in Google Play, enabling users to look for and install recent Mainline updates.
When a Google Play System Update is deployed, the device may need to reboot to accommodate new APEX or APK files. From Android 11 onwards, devices can employ "soft reboots" that refresh system elements.
Google sends Mainline module updates as bundled packages. Upon their deployment, the ModuleMetadata app's version number is updated, which informs the Google Play System Update version in device settings. For safeguarding against flawed updates, Android's RollbackManager reverts them to their previous stable versions under specific circumstances like system errors or network issues. Advanced users can leverage command-line tools to manage and inspect APEX module installations and their cryptographic validity.
Let's delve deeper and analyze the logs event-wise to provide a chronological insight into what transpired with the device.
Package Session Verification: The system is verifying a package session which often refers to the installation or update of a package.
Starting preRebootVerification for session 291950126
APEX Package Information: APEX packages are used in newer versions of Android for system components. The log indicates a session associated with an APEX package related to Google's mainline module.
Session 291950126 has following APEX packages: [com.google.mainline.primary.libs]
Broadcast Intent: A broadcast intent was sent by the system regarding a package installation.
pkg=com.android.vending.INTENT_PACKAGE_INSTALL_COMMIT.mainline_train_primary
Shutdown Initiation: The system is initiating a shutdown with a specified reason. Here, the reason seems related to an enterprise update.
reboot reason : enterprise,mainline_update, confirm : false
This log captures the moments just before a device reboot, initiated due to a system update or change (particularly related to mainline modules).
The log entry for Rollback relates to Android's RollbackManager and the feature in newer Android versions that allows for system updates or app updates to be rolled back to a previous version if there are issues. This is particularly useful in scenarios where an update might introduce problems or regressions and is required to revert to a more stable version.
The specific package mentioned, com.google.mainline.primary.libs, suggests this is related to Android's Mainline Modules. As mentioned earlier, Mainline Modules are a part of Android's Project Mainline and allow Google to deliver security fixes, privacy enhancements, and other improvements directly to users without needing to rely on device manufacturers to deliver a full system update.
The log entry is indicating that a rollback capability was enabled for the update of com.google.mainline.primary.libs. This means that if there's an issue with this update, the system can roll back to the previous version of this module.
At this point, it's evident that the device underwent a soft reboot. However, there's another log file that pinpoints the precise causes for such shutdowns or reboots. Let's delve into that as well.
In Android devices, system events, errors, or unusual behaviors can generate log files for debugging and diagnostic purposes. One such file is the "Power off reset reasons" file. This file essentially captures the reasons or causes for unexpected power-offs or reboots that the device might have experienced.
The main purpose of this file is to aid developers, device manufacturers, and sometimes IT professionals in diagnosing why a device may have unexpectedly shut down or restarted. By understanding the root cause, they can then work towards resolving any underlying issues. The exact location of this file can vary depending on the device manufacturer and the version of Android being used. However, it's often found in the device's system logs or /proc/last_kmsg directory, especially if the device experienced a kernel panic.
This file is typically saved in a system directory, and accessing it might require root permissions or ADB (Android Debug Bridge). So, what does our log file say?
23/07/24 18:55:22
reason : enterprise,mainline_update
java.lang.Exception: It is not an exception!! just save the trace for process which called shutdown thread!! ShutdownThread.shutdown
at com.android.server.power.ShutdownThread.shutdownInner(ShutdownThread.java:344)
at com.android.server.power.ShutdownThread.reboot(ShutdownThread.java:444)
at com.android.server.power.PowerManagerService$2.run(PowerManagerService.java:6078)
at android.os.Handler.handleCallback(Handler.java:942)
at android.os.Handler.dispatchMessage(Handler.java:99)
at android.os.Looper.loopOnce(Looper.java:226)
at android.os.Looper.loop(Looper.java:313)
at android.os.HandlerThread.run(HandlerThread.java:67)
at com.android.server.ServiceThread.run(ServiceThread.java:44)
at com.android.server.UiThread.run(UiThread.java:52)
18:55:30 2023-07-24 18:55:30+0100 | REBOOT | | REASON: enterprise,mainline_update
18:55:30 terminating init service start
18:55:31 terminating init service end
18:55:31 volume shutdown start
18:55:31 volume shutdown end
2023-07-24 18:57:02+0100 | ON | RP | / OFFSRC: PWRHOLD / ONSRC: ACOKB / RSTSTAT: SWRESET [72]/ apex_updated Reboot Reason:
reason : enterprise,mainline_update: This denotes the reasons for the reboot. The term enterprise suggests the device is managed by an enterprise policy, indicating it's part of an organization's mobile device management system. The term mainline_update suggests the device is undergoing a Project Mainline update.
Reboot Action:
2023-07-24 18:55:30+0100 | REBOOT |...: This is a timestamped record of the actual reboot action, reiterating the reasons (enterprise,mainline_update).
System Restart Info:
2023-07-24 18:57:02+0100 | ON |: This timestamped entry signifies the device turned back on. RSTSTAT: SWRESET [72] indicates the device underwent a software reset, and apex_updated confirms a Project Mainline module was updated during this reboot.
The log clearly records a deliberate system reboot, likely to apply a Project Mainline update, on an enterprise-managed Android device.
The presence of the term "enterprise" in the logs is indicative of the device being managed under Android Enterprise for enhanced management and security features for Android devices in corporate environments.
When a device is enrolled in Android Enterprise through an Enterprise Mobility Management solution like Microsoft Intune, it is subject to policies and configurations set by the enterprise IT department. This includes the ability to push software updates, control the installation or removal of apps, enforce security policies, and much more.
The log that I analyzed indicated a request to PowerManagerService to initiate a shutdown/reboot. The reason provided was, enterprise,mainline_update, suggesting a mainline module update. Mainline modules are a part of Android's Project Mainline, which allows core OS components to be updated via the Google Play Store (represented here by com.android.vending). So, the reason for the restart seems to be related to an update of a core OS component (mainline module).
Furthermore, the log for rollback manager also indicates that a rollback capability was enabled for the update of com.google.mainline.primary.libs. In the context of device restart behavior, it seems that the device has been initiating an update related to Project Mainline and went through a soft reboot.
However, it's essential to note that such mechanisms are designed with user security and device stability in mind. These reboots ensure that the device runs the most recent and secure software modules.
I trust this article has shed light on the mechanics behind updates and offered a clearer understanding of the log processes. If you found this information valuable, please consider sharing the post. Should you have any questions or need further clarification, don't hesitate to reach out!
