Last week, just after we covered the release of Big Sur, many macOS users around the world experienced something unprecedented on the platform: a widespread outage of an obscure Apple service caused users worldwide to be unable to launch 3rd party applications. Already being dubbed the “Apple Apocalypse” or “OCSP Apocalypse”, the cause was down to a little-known but essential service called “Online Certificate Status Protocol”. In this post, we look at what OCSP is, explain why it affected Macs so severely, and discuss some of the implications that have arisen in light of this unusual event.
Why Your Mac Couldn’t Launch 3rd Party Applications
On Thursday, users on Twitter and other social media platforms began complaining that their Mac computers were becoming unresponsive, hanging and unable to launch or install many 3rd party applications.
Some quick detective work soon pinned the blame on a system daemon called trustd.
As the somewhat sparse man page tells us, trustd is a service that evaluates trust in certificates for all processes on the system.
As it goes about its business, trustd makes a network call to a service called “ocsp” – Online Certificate Status Protocol.
The purpose of the OCSP call is to check whether a piece of software being launched has had its developer certificate revoked. Revoking developer certificates is one way that Apple deal with known malware. By using an OCSP responder service, Apple hope to prevent any software whose certificate has been revoked from launching on pretty much all Macs anywhere within minutes.
As was well-documented over the weekend, trustd employs a “fail-soft” call to Apple’s OCSP service: If the service is unavailable or the device itself is offline, trustd (to put it simply) goes ahead and “trusts” the app. After all, people aren’t always connected to the internet, and – as we find out from time to time when there’s a service outage – neither are Apple!
However, in this particular case, ocsp.apple.com wasn’t in fact offline. The trustd service was able to reach the server, but the server was experiencing a slowdown. And now, rather than failing softly, trustd just kept hanging around for an answer…and users’ Macs just kept, well, hanging.
Reactions to Apple’s OCSP Apocalypse
While the problem persisted for only a few hours before Apple got on top of the server slowdown, the fallout has been going on throughout the weekend. For many users, it came as quite a shock that their usually reliable Macs appeared to have a single point of failure. Ironically but understandably, the failure of trustd – a service designed to improve security – led some to believe that their Macs had been infected with malware, which must have been an unpleasant experience, to say the least.
In the immediate aftermath, some commentators began to draw more sinister conclusions: if macOS sends data about every app you launch to Apple, then presumably Apple can track exactly what users are doing, when and where. As one writer pointed out, OCSP uses the insecure, plain-text HTTP protocol. That appears to suggest that not only Apple but also “anyone with a traffic analyzer on your network could eavesdrop every app you open and when you open it.”
That very idea was taken to its logical, but not entirely accurate, extreme in a blog post that dramatically claimed “your computer isn’t yours”:
The author went on to worry about just how many others might have this same data, from your ISP to the NSA to a MITM on your local network.
Such dramatic conclusions led some to suggest that users should block OCSP either in their firewall software or by editing their /etc/hosts file. (Spoiler: SentinelOne do not recommend doing either. Read on to find out why).
Why You Should Not Block Calls to OCSP Responder Services
So on top of the OCSP apocalypse, we seem to now have a privacy apocalypse…except there’s a few important rebuttals to bear in mind.
First, OCSP doesn’t in fact send hashes of applications over the wire; it sends some obfuscated information about developer certificates belonging to those apps.
Even more precisely, in the case where a developer has more than one Apple app signed with that certificate (most developers sign all their apps with the same certificate), it doesn’t even expose which app from that developer was launched. For anyone familiar with Apple’s developer IDs, this would be immediately obvious. Apple’s certificate revocation, when deployed, doesn’t work on a per-app basis, it works on a per-developer basis.
A second point to bear in mind here was ably explained by Phil Vachon. If you want to be confident that software you’re running has been securely signed and is trusted by the certificate issuer, there’s only a few options available, and each has trade offs.
On the one hand, Apple could periodically dump a list of all revoked app certificates to every user’s Mac. Vachon points out two problems with certificate revocation lists (CRLs): storage (they can be large) and update frequency. Revoked apps could easily fall through the cracks. Another option is a variant of the OCSP model Apple use called OCSP stapling, which has the benefit of ensuring anonymity, but actual implementations to date have been somewhat unreliable. The path Apple have chosen, real-time OCSP checking, does have both known privacy (yes, there’s some data leakage) and security (e.g., the fail-soft design is vulnerable to an attacker blocking the device’s connection to the OCSP server) implications.
However, the privacy implications are neither peculiar to Apple (we are all in the same boat with every online service we use) nor more severe than we face with other providers. Unlike many online services we use, Apple have no business trading user data to 3rd parties or advertisers, and privacy – even if not always well implemented – is a core part of their business model.
Third, and most importantly, from a threat model perspective, users that follow the misguided advice to block calls to OCSP increase the risk to their devices and network. Malware is a far more prevalent and immediate threat than the weak data leakage represented by OCSP responder services. Although SentinelOne protects user devices from macOS malware without relying on certificate revocation services, removing any layer of defence is never a good idea unless that layer poses more risk than it offers security. There is absolutely no evidence that this is the case with Apple’s OCSP service.
Apple’s Response to the OCSP Apocalypse
As we said above, using an OCSP responder service for checking certification status allows Apple to prevent unwanted software from launching on a Mac within minutes, notwithstanding either malicious or accidental network outages.
How many minutes? It’s been suggested that typically OCSP checks are cached on a device for as little as 5 minutes, meaning that if you launch an application a few times rapidly, your Mac doesn’t waste time re-checking that software’s status until at least five minutes since the last launch. If correct, that’s a pretty aggressive timeout, and Apple may have made changes server-side to cache responses for up to 12 hours instead in light of last week’s problems (note: I could not reproduce the same findings on my devices).
Moreover, Apple made a rare implicit acknowledgement of the problem, stating that in the near future they intend to introduce stronger protections against server failure, a possible opt-out of security checks for users, and a new encrypted protocol for Developer ID certificate revocation checks.
Did Apple’s Outage Affect SentinelOne’s Agent?
For SentinelOne customers already running Agents on their devices, the problems with Apple’s servers would not have caused any loss of protection as the service is constantly running and not subject to certification status checks after initial launch.
Customers may have been temporarily affected by the failure of ocsp.apple.com if they tried to launch the SentinelOne Installer to deploy the agent to a new device during Thursday 12th or early hours of Friday 13th (depending on location).
In the unlikely event that Apple have a repeat of the problem with their ocsp.apple.com service, Mac users are advised to temporarily disconnect from the internet to launch any stalled or unresponsive applications, and re-connect once the application has finished launching.
Conclusion
Apple’s OCSP problem caused some major disruption for many of their customers and provided a perhaps unwelcome introduction to network security plumbing for some. Safely and reliably querying certificates for revocation status is a known hard nut to crack, and the privacy and reliability issues with OCSP were well-known to network security engineers long before last week’s meltdown. It’s not as if Apple have been caught doing something that nobody was aware of, even if it’s true that many non-security folk, and some popular bloggers, had never heard of the problem of establishing online revoked certificate status before Thursday’s unwelcome outage. Despite the acknowledged privacy and security drawbacks with OCSP responder services in general, their benefits far outweigh their drawbacks.
If you’d like to learn more about SentinelOne’s protection for macOS, contact us today or request a free demo.
