Recently, we shared details on how effectiveness is measured for Office 365 Exchange Online Protection (EOP) and Advanced Threat Protection (ATP). We also followed up with a comprehensive update on Office 365’s improved ability to stop phishing emails from impacting users. These reports highlighted:
Today, we’ll cover recent research on a testing methodology—email journaling—which is often used but can lead to misinterpreted results.
Email journaling (Figure 1) is when an organization enables recording of emails for retention or archiving. With growing regulatory requirements, organizations increasingly must maintain records of communications between employees performing daily business tasks. Journaling helps organizations respond to legal, regulatory, and organizational compliance requirements by recording inbound and outbound email communications. Exchange Online offers in-depth journaling capabilities. Microsoft provides extensive and up-to-date recommendations on how organizations can manage and configure journaling requirements.
Figure 1. Email journaling mail flow.
Sometimes we receive inquiries from customers seeking guidance on whether journaled email can be used to measure the effectiveness of our security solution. Sometimes, third-party vendors use journaled emails to assess effectiveness; however, this can lead to inaccurate results because there is a:
In this scenario, third-party vendor(s) recommended customers create a journaling rule routing emails to the vendor’s testing cluster. The vendor(s) then evaluated the emails with their advanced filters to determine which emails Office 365 ATP “missed.” However, in Office 365, ATP protection—which includes Safe Attachments (file/URL detonation) and Safe Links (time-of-click protection)—comes after the journaling rules. As a result, emails routed to a journaling archive have not yet been scanned by ATP’s Safe Attachments or Safe Links policies (Figure 2). Our stack is designed so that journaling comes after the standard EOP anti-virus scans—but before ATP scans—so known malicious emails are not archived. A potential best practice is to rescan emails released from an archive to help ensure they’re not malicious.
Figure 2. The top graphic shows the entire mail flow and security stack in EOP/ATP, while the bottom graphic is a blowup of the section that shows where the journaling rule takes effect and how it is before our Safe Attachments/URL sandboxing policy, which is part of Office 365 ATP.
When journaled mail is used to measure effectiveness, it’s important to remember that the emails are not scanned by ATP—since journaling happens before ATP. Figure 3 shows how measuring journaled emails for effectiveness can cause a misinterpreted analysis because most emails thought to have been missed were in fact blocked by ATP and how a large percentage of emails not detected by ATP were false positives.
Figure 3. The emails which were characterized as “misses” never went through the ATP filters. When we ran them through ATP, we found ATP blocked most of the emails and also did not block many emails that were false positives.
There have been situations where customers were advised by third parties to use journaled emails to identify emails missed by Office 365 ATP, which is impossible due to the architecture of Office 365’s mail flow. As with any service, Office 365 ATP also misses some emails. No service is 100 percent secure, but the best services are able to enhance and evolve quickly to address emerging threats. This ability to quickly enhance our services is one of our strengths and is manifest in the rapid evolution of Office 365 ATP into the optimal security service for Office 365.
Interestingly, customers often provide us with email samples already scanned by a third-party vendor’s advanced filters to determine how Office ATP would perform on the same set of emails. Figure 4 shows the unique catch of ATP versus a third-party vendor in one such inquiry. In that inquiry, Office 365 ATP found 18 times more unique malicious emails than the third-party vendor. Also, with phishing being a predominate form of attack, we saw that the third-party vendor missed several hundred phishing emails.
Figure 4. Office 365 ATP’s unique catch rate is 18 times greater than a third-party vendor, from a recent comparison of data shared with Microsoft by a customer.
We don’t look for gaps in third-party services unless a customer asks us to investigate. Our focus is on enhancing our service to help provide maximum security for our customers. We don’t claim to catch everything; however, we are confident that no other service will secure you better in Office 365 than Office 365 ATP. Put us to the test with a trial. We have previously suggested that bifurcating real mail flow will provide a side-by-side analysis on effectiveness. This is the most powerful and informative test.
Begin an Office 365 E5 trial today to experience best-in-class security for your Office 365 environment and contact your rep to help you test us against any competitor. The results will serve as the ultimate validation that Office 365 ATP provides the best security for your Office 365 environment.
The main components of the Microsoft Intune Suite are now generally available. Read about how consolidated endpoint management adds value and functionality for security teams.
To confidently secure identity and access at your organization, here are five areas Microsoft recommends prioritizing in the new year.
Matt Suiche of Magnet Forensics talks about top security threats for organizations and strategies for effective incident response.
Identity-based attacks are on the rise, making identity protection more important than ever. Explore our blog post to learn how Microsoft’s Identity Threat Detection and Response can help.
