By Henry Stamerjohann and Éric Falconnier
During WWDC 2016, Apple announced the new Unified Logging system.
One common, efficient logging mechanism for both user and in kernel mode
This system can be found on iOS, tvOS, watchOS and macOS. On macOS, since Sierra, it supersedes Apple System Logger (ASL) and the Syslog APIs. Log events are stored in a database, both in memory and on the filesystem, following a binary format. The events generated with the legacy APIs (NSLog, asl_log_message, syslog…) are redirected to this new system.
Only messages above a configured level or explicitly configured to be persisted are kept on disk. The rest are discarded as new events come in. This is a good move in an ecosystem where most of the devices run on batteries and have SSDs. But it is incompatible with the standard Unix toolbox (tail, grep, less, …) that Mac admins have been using for a long time to manually inspect the logs or build support and forensic tools.
The following command doesn’t work in Sierra:
grep “Failed to authenticate user” /private/var/log/system.log
Therefore a new Console app and a command line tool have been introduced. The command line tool (log) can be used in place of less (log show) or in place of tail (log stream). It is possible to specify predicates to filter the huge number of events.
The log command to find failed authentications in Sierra
log show – -predicate ‘eventMessage contains “Failed to authenticate user”‘
Log shipping could be implemented with the new log command, but until an API is supported to programmatically access the event data store, the solutions will stay brittle. This is the main concern and something that Apple needs to address soon. Another scenario that has become less trivial is asking users to ship the logs manually. Copying and sending files is not enough anymore. You need to help them to use log.
But other sources of information can still be easily centralized. Many organizations use Jamf Pro or Munki to manage their fleets. These tools have agents that connect directly to a server. Popular dashboards like Sal or MunkireportPHP can complement Munki and offer you even better insights into what is happening on your devices.
To go beyond what is possible with these well-established solutions, two new open source tools have emerged in the last two years.
The first one is Osquery. It was started internally at Facebook. It exposes the state of a system (Linux, macOS, Windows) in the form of a relational database, with different tables corresponding to different subsystems. There is a table for the users, a table for the running processes, and many more. The beauty of this architecture is that you can query a device with SQL, and JOIN tables like you would in Sqlite. This is for example ideal to find the artifacts indicating the presence of malware and query the installed certificates, all from just one tool. Queries can be scheduled to run at regular intervals, and only the differences in the results will be logged. You could use it to track the installation of a LaunchDaemon for example.
The other tool is Santa, released by Google. The main component of Santa is a kernel extension that monitors the execution of binaries. It allows them to run or not run following a set of configured rules. The decisions are based on the checksum of the binary itself or the checksum of one of the certificates used to sign it. You could block a specific version of ‘Transmission.app’ for example, or block all the binaries signed by ‘Digital Ignition LLC’. Santa has 2 modes: MONITOR and LOCKDOWN. In MONITOR mode, Santa will allow all binaries to run except those blacklisted by a rule. In LOCKDOWN mode, Santa will block by default and only allow the whitelisted binaries to run. In both case, all decisions will be logged, which is a good way of knowing what apps are used across all your devices.
Both Osquery and Santa can be managed remotely and can send logs to a server with HTTP requests over a TLS connection. This is where the idea of Zentral comes from.
At its core, Zentral is a framework to gather, process, and monitor system events and link them to an inventory. Modules are available to aggregate the information of existing inventory providers (Jamf Pro, Watchman, …), to act as the configuration and log server for Osquery and Santa, and to integrate with Munki in the post-flight process.
Events created by the changes in the inventories, or posted by Osquery, Santa, Munki…, or coming from other modules are normalized and enriched before they are stored in Elasticsearch.
The Zentral configuration is organized in ‘Probes’. A Probe bundles together filters, actions and optional configuration items necessary for one task. The filters are applied to the event metadata (event type, device OS, device business unit, …) or the event payload. If an event matches, the actions will be triggered. Actions can create tickets, send event notifications as email, text or Slack messages. A machine can be moved to a group in Jamf Pro to trigger more complicated workflows. Finally, the configuration items are specific to some probe types. An Osquery Probe has Osquery queries. It will trigger a Zentral event for every results change on every device. A Santa Probe has Santa rules to blacklist or whitelist some binaries and trigger events for every decision.
This modular approach makes it easy to build Probes corresponding to your own scenarios. You can edit the probes directly in the Zentral interface, tweak the filters, view the events that match, and jump to Kibana where you can build dashboards or drill further down.
USB storage events with an Osquery Probe. Combination of an inventory filter to restrict the probe to a business unit, an Osquery query to be scheduled on the devices of this business unit, and an action to post event notifications in Slack.
The Zentral events filtered by the probe.
The same events in Kibana aggregated by vendor name in a pie chart.
With this example, we have only scratched the surface of what is possible with Zentral. Check out our YouTube playlist where we post our video tutorials.
About Zentral, Éric Falconnier and Henry Stamerjohann
Zentral was open-sourced in Fall 2015. Based in Hamburg, Germany, Éric Falconnier and Henry Stamerjohann lead its development and offer support for it. It is already deployed at Apfelwerk, a German MSP, where it is used in production on a large and geographically distributed client base.
Henry Stamerjohann will address the MacADUK conference at 16:15 on Tuesday 7th February 2017.