Active Directory: A canary under your hat part 3

Foreword:

This three-part blog article series focuses on some research work on how to detect effectively Active Directory enumeration in a SOC environment. To help you through this quite long journey, grab a cup of hot beverage of your liking, and use this short reference:

Part 1: about detecting AD enumeration

This first part explains what Active Directory enumeration is, how it is leveraged by adversaries during their offensive operations, and digs into how detecting such activities. It aims at highlighting issues on both efficiency and cost of common detection primitives that detects AD enumeration.

Part 2: AD Canaries & DACL backdoors

This second part will first quickly explain what are DACL backdoors – “An ACE Up the Sleeve” 1 published in 2017 by already famous Specter Ops researchers Andy Robbins, Lee Christensen and Will Schroeder. And will then detail the detection primitive AD Canaries mechanism, along with
some lab implementation and results.

Part 3: Making it useable – improvements and in-production deployment results

This last part will cover some exploratory work that aimed at making this detection primitive useable for SOC operations; improving enrichment and identifying how to effectively exclude on recurrent false positives. It will then analyze results and observations on some in-production deployment we conducted.

Part 3: Making it useable – improvements and in-production deployment results

In the previous sections of this blog we covered Active Directory enumeration detection issues and introduced a new primitive for detection: AD Canaries. Analysis of the detection capabilities and requirements indicated that AD Canaries should solve some identified issues for efficient, and low-cost, AD enumeration detection.

Nonetheless, these were only laboratory tests, and no conclusions on the efficiency of this primitive can be done without a proper deployment in production and an assessment in a real environment. This will be our topic for the last section of this blog, focusing on improvements in both detection, investigation and false positive management, and on the result feedback of a deployment in production of AD canaries.

Improvements (and failed attempts) for production viability

As a proper deployment in production would require capabilities to exclude normal behaviors and identify the exact enumeration behavior, some more testing in lab needed to be conducted.

Among those tests, many were unsuccessful attempts to improve the detection primitive. I will not cover all of them in detail, but here are some of these attempts and what I learned from them. 

Failed attempt #1: detect specific enumeration through ADCanaries specific attributes

Side note: The first two improvement attempts are failures I faced, and found interesting to share and highlight AD canaries’ limitations and lessons I might have learnt from them. If you do not wish to dig in some arguably bad attempts to improve detection, investigation capabilities or false positive management, you can skip directly to the dedicated section:  DACL deployment improvements 

A first approach to identify the exact behavior was to get insights on what attributes were accessed. Similarly to what honeypot accounts would do, I asked myself if it would be possible to deploy specific Canaries with tempting attributes and properties. If so, it would be particularly interesting to design canaries as in Sean Metcalf’s “honeypots accounts”, but detecting specific attempt to enumerate weaknesses in the environment rather than the interaction with the honeypot. The idea here was to design high fidelity AD canaries that would trigger when specific malicious enumeration patterns would be perform to identify known weaknesses. Doing so would probably reduce drastically the number of false positives to face in SOC operations.

For example, when adversaries perform AS-REP roasting attack to attempt offline password cracking on vulnerable accounts, they can enumerate vulnerable accounts via LDAP. Indeed, accounts that are vulnerable to this attack have the “do not require pre-auth” set in their UserAccountControl attribute. In this case, the question was if deploying a canary configured to be vulnerable to AS-REP roasting (but still hardened by DACL and disabled account) would trigger when enumerating for vulnerable accounts.

Upon testing, I observed that all deployed user canaries would trigger (not just the specifically designed one) upon specific enumeration aiming at identifying AS-REP roastable accounts. This meant something was quite wrong in either the design of the canaries or in my understanding of Active Directory specific enumeration. And it was the latter: I had misunderstood (or forgot) how LDAP filters would match and trigger on accesses on the AD Canary objects.

Figure 1 – LDAP request capture in Wireshark from artisanetworks.force.com article

Because LDAP filters apply when accessing the attribute or property of the object – the UserAccountControl in the case of AS-REP roasting – enumeration filters do not matter when using Canaries. Indeed, access to said properties or attributes would always be denied and trigger all ADCanaries whatever the value of their attribute or properties. Which is exactly what was observed when creating such canaries in my lab.

A workaround would have been to design Canaries with specific DENY on those attributes and “regular” ACEs on the rest of the object. However, such object would not be as stealthy as the previous canary version, and impact on AD attack surface would be greater and difficult to assess. 

In other words, this was a dead end, but it highlighted some by-design limitations of Active Directory canaries.

Failed attempt #2: ADCanaries matching groups existing within AD

Side note: The first two improvement attempts are failures I faced, and found interesting to share and highlight AD canaries’ limitations and lessons I might have learnt from them. If you do not wish to dig in some arguably bad attempts to improve detection, investigation capabilities or false positive management, you can skip directly to the dedicated section: DACL deployment improvements

Another failed improvement attempt was the idea of deploying per-ADGroup canary objects, aiming at getting the list of groups that were enumerated. This came during an investigation where we found an adversary using built-in “net” commands to enumerate specific groups in the domain. 

This type of targeted enumeration of AD group members would not trigger the default AD canaries as they were only members of “Domain Users” as in the early version of the deployment script. So, I got back head to the grindstone and developed this functionality in the script whenever I could.

This was a bad idea for multiple reasons: 

Lowering AD canaries’ stealth capabilities

Adding these canary objects to existing AD groups would allow offensive operators visibility on said objects through the Members property of the group.

Figure 2 – AD canary added to HelpDesk group – referenced in Members property

As we can see here, even from an administrative account the canary cannot be directly accessed, but its distinguished name is still present in the “Members” property of the group it was added to! Giving adversaries opportunities to identify our Canary when doing targeted enumeration.

Lowering effective control on the attack surface induced by AD canaries

Adding the canary objects to existing group would create additional control relationships on the canary object, making it harder to assess the impact on the attack surface of this deployment.

Indeed, by placing AD canaries outside of their isolated bubble we cannot effectively assess that no control relationship misconfiguration will apply to it, nor that the audited deployment will be unaffected by the production environment. 

This was one of the more interesting capabilities of AD canaries, an isolated and hardened deployment inside of Active Directory.

Increasing the number of generated events 

Increasing the number of canaries and their trigger surface would increase drastically the number of logs generated by this deployment in production. Canaries would also constantly be triggered by any application using LDAP authentication in the environment, as they would be part of existing groups and assets that should be listed quite often.

DACL deployment improvements 

You might have noticed in the previous section, I mentioned that AD canary objects were members of the default group “Domain Users” when deployed with the first script version. I discovered that this was due to the default primary group affectation of User and Computer Objects with respectively “Domain Users” and “Domain Computers” groups.

As my previous failure highlighted that having canaries in existing group was not a good idea, it was time to harden the DACLs deployed on these objects and increase their stealth: 

• Implementation of full “Hiding the principal” primitive, deploying a dedicated OU for the deployed objects and removing all access to this OU
• Disabling inheritance on the OU and on all deployed objects to limit control relationships over the deployed canaries
• Deploying a dedicated group for canary objects in order to replace the default primary group and remove canaries for all existing groups in the environment

In this final version, the deployment requires only 5 objects to be deployed:

The same results can of course be obtained in a manual deployment of these 5 AD objects.

Enrichment: properties schemaIDGUID lookup generator

At this point, canaries were hardened and deployment script was improved, but I could not figure a good way to finely distinguish normal AD enumeration behavior occurring on daily basis in production environments. 

To effectively do so, information on what objects were enumerated and what properties or attributes were accessed is required. Such information would allow defenders both fine tuning of the detection rule, and quick identification of what the opponent might be looking for when facing targeted enumeration.

In the first part of this blog I introduced briefly the Security events 4662, but was focusing on the detection issues related to its audit in production. I made the mistake to not dig into all the event’s fields during the conception of AD canaries, so let’s have a second look at it:

<quote>

<EventData>

 <Data Name=”SubjectUserSid”>S-1-5-21-3457937927-2839227994-823803824-1104</Data> 

 <Data Name=”SubjectUserName”>dadmin</Data> 

 <Data Name=”SubjectDomainName”>CONTOSO</Data> 

 <Data Name=”SubjectLogonId”>0x35867</Data> 

 <Data Name=”ObjectServer”>DS</Data> 

 <Data Name=”ObjectType”>%{bf967a86-0de6-11d0-a285-00aa003049e2}</Data> 

 <Data Name=”ObjectName”>%{38b3d2e6-9948-4dc1-ae90-1605d5eab9a2}</Data> 

 <Data Name=”OperationType”>Object Access</Data> 

 <Data Name=”HandleId”>0x0</Data> 

 <Data Name=”AccessList”>%%1537</Data> 

 <Data Name=”AccessMask”>0x10000</Data> 

 <Data Name=”Properties”>%%1537 {bf967a86-0de6-11d0-a285-00aa003049e2}</Data> 

 <Data Name=”AdditionalInfo”>-</Data> 

 <Data Name=”AdditionalInfo2″ /> 

</EventData>

</quote>

I previously focused on Subject user context which provides information about the initiator of the detected enumeration:

• • SubjectUserName
  • SubjectUserDomain

• SubjectLogonId – which can be used to pivot on Logon events (4624) that should be of LogonType 3 (Network) and give the IPAddress of the endpoint from which enumeration was performed.

However, I completely missed that what I was looking for was already present in this very event used for detection, but in a hardly readable format: the “Properties” and “AccessMask” fields! 

<quote>

• Accesses [Type = UnicodeString]: the type of access used for the operation. See “Table 9. Active Directory Access Codes and Rights.” for more information.
• Access Mask [Type = HexInt32]: hexadecimal mask for the type of access used for the operation. See “Table 9. Active Directory Access Codes and Rights.” for more information.

[…]

• Properties [Type = UnicodeString]: first part is the type of access that was used. Typically has the same value as Accesses field.

[…]

• Base DN: CN=Schema,CN=Configuration,DC=XXX,DC=XXX
• Filter: (&(objectClass=*)(schemaIDGUID=GUID))

</quote>

In plain, the accessed properties and kind of access are already logged in the generated events, we only need to translate this information to make it usable. So, I added a new option to the script to crawl the available Properties for the user/computer/group types of object and automatically create a CSV lookup (PropertyName <-> schemaIDGUID) that would be deployed in the SIEM solution.

For the AccessMask, simply copy and paste the 20 line-long table present in MS documentation and format it to whichever format suits you for a lookup in your SIEM.

Combining all these enrichments into the previous detection query, gives us the following KQL query  (as I used Sentinel SIEM this time):

SecurityEvent

| where EventID == 4662 and ObjectServer == “DS”

| extend Object = tostring(split(split(ObjectName, “{“)[1], “}”)[0]),

         AccessedProperties = extract_all(@”\{([a-fA-F\d]{8}-[a-fA-F\d]{4}-[a-fA-F\d]{4}-[a-fA-F\d]{4}-[a-fA-F\d]{12})\}”, Properties)

// ADCanaries.csv generated when deploying via the aforementioned script

| lookup kind=inner _GetWatchlist(“ADCanaries”) on $left.Object==$right.SearchKey

| mv-expand AccessedProperties

| extend Prop = tostring(AccessedProperties)

// PropertiesGUIDs.csv generated when deploying via the aforementioned script

| lookup kind=leftouter _GetWatchlist(“PropertiesGUIDs”) on $left.Prop==$right.SearchKey

// Simple DIY lookup table using MS documentation for event 4662

| lookup kind=leftouter _GetWatchlist(“AccessTypes”) on $left.AccessMask==$right.SearchKey

| join kind = leftouter (

    SecurityEvent

    | where EventID == 4624 and LogonType == 3

) on $left.Computer==$right.Computer and $left.SubjectLogonId==$right.TargetLogonId and $left.SubjectUserName==$right.TargetUserName

// Some aggregation here, not optimal 

| summarize Count=count(), AccessedProps=makeset(ldapDisplayName), LogonIds=makeset(SubjectLogonId), Accesses=makeset(Access), IPs=makeset(IpAddress1), Devices=makeset(WorkstationName1)

by bin(TimeGenerated, 1m), SubjectUserName, SubjectDomainName, CanaryName

// Don’t forget to add fine tuned exclusions on normal behavior here 

Now, let’s test this again in our lab and validate improvements on enrichment and investigation capabilities! 

Enumerating all users, groups and computers using PywerView – A (partial) Python rewriting of PowerSploit’s PowerView from an attacker machine:

Figure 3 – Pywerview non specific enumeration [Kali machine] #1

Enumerating specific weaknesses in the targeted Active Directory environment from the attacker machine:

• Users with Kerberos unconstrained delegation set up, leading to delegation abuse and potential domain privilege elevation:

• Users that aren’t protected from delegation (not “sensitive and not allowed for delegation”):

Side note: Useful resources about Kerberos delegation abuses (non-exhaustive).

• Users with attribute “Do not require Kerberos preauthentication” leading to AS-REPRoasting attack path:

• Privileged users with adminCount attribute set up:

Figure 4 – PywerView specific enumeration [Kali machine] #2

Enumerating specific weaknesses in the targeted Active Directory environment from a compromised endpoint using Rubeus:

• Listing ASREP-Roast-able accounts (do not require preauth):

• Listing Kerberoast-able accounts (ServicePrincipalName set for the account):

Figure 5 – Rubeus specific enumeration + exploitation ASREP-Roasting & Kerberoasting [Windows machine] #3

• Listing all users using built-in “net.exe” executable:

Figure 6 – net built-in enumeration – AD canary not showing (Jan 6, 4:33PM) [Windows machine] #4

Observed results:

Figure 7 – Detection results

We can see here that with some exclusions for the domain controller account and the admin account used for deployment, we did not get a single false positive event. Additionally, we are able to identify patterns for specific enumeration across different tools. Those patterns inform us on the type of information the opponent was trying to get. 

We can also track the malicious user who changed location from a remote Kali machine named “PulseSecure01” to a compromised local asset.

This implementation gives SOC analysts means to identify and finely tune exclusions on legitimate behaviors observed in their environments. This helps in keeping control on the accepted blind spots induced by exclusion creations:

For example, using this new implementation, SOC analysts might want to exclude on a hypothetical service account – let’s say “SvcShare” that performs enumeration of all users with accessed Properties “SAMAccountType” from an identified server on which related software is running.

 The provided enrichment allows us to effectively deploy such exclusion! And should this account be compromised and used to enumerate in another way for example accessing property “servicePrincipalName” related to Kerberoasting attack, AD canaries would still trigger and alert us.

Even though we had to keep in mind that this test was still in a lab environment, this seemed finally ready for production testing! 

Production deployment & results 

The first deployment of this detection primitive was done on one of our SOC clients with about 1.5k endpoints (workstations + servers). 

Deployment did not go through on the first attempt (even though – of course – it perfectly worked in multiple lab environments) multiple changes and adjustments needed to be made on the deployment script. But the client stood still and helped a lot – once again many thanks. 

After some changes and code cleanup, second attempt was the charm, as we did manage to deploy the canaries. In short:

• SACLs were verified on the environment
• Audit policy was verified on the domain controllers
• Canaries were deployed (revert was tested and validated), GUIDs and CanaryNames lookup was deployed in the dedicated SIEM
• SchemaIDGUIDs were retrieved and deployed as a lookup in the dedicated SIEM

Exclusions, automation and operational feedback

Slowly but surely, nominal AD enumeration behaviors were observed on the environment and excluded in the SIEM solution. In a 10 days’ timeframe about 15 recurrent enumeration mechanisms were excluded based on:

• The accessed canaries GUIDs
• The access mode
• The list of accessed properties

After having identified and excluded on almost all behaviors, the detection rule was deployed and incidents were raised to validate BUILD and PRE-RUN phases. During this period only 5 incidents were raised.

Figure 8 – Incidents raised in the SIEM solution after tuning

Among these incidents only one was a false positive that needed to be excluded on as it was not user initiated. Other incidents were false positives generated by administrative actions, legitimated easily – using automated pivots explained below.

As expected during lab experiments, impact on log volumetry was negligible:

On an almost 2-month long period after deployment, the observed impact on event volumetry (i.e. 4662 windows events generated when accesses are attempted on canaries) was stable with a maximum of 120 event/day:

SELECT DATEFORMAT(starttime, ‘YYYY-MM-dd’) as Date, COUNT(*), QIDNAME(qid), Logsourcename(logsourceid), User, REFERENCEMAP(‘<redacted>’,object) as CanaryName, 

object, object_server, “object_access_mode”

FROM events 

WHERE event_id=4662 AND object_server=’DS’ AND QIDNAME(qid)=’Failure Audit: An operation was performed on an object’ AND REFERENCEMAP(‘<redacted>’,object) != NULL

GROUP BY Date

ORDER BY Date ASC

START ‘2023-01-01 00:00:00’ STOP ‘2023-02-28 00:00:00’

Figure 9 – Extracted event count per day – detailed graph result

This amount of generated events is ridiculous compared to the previous 40000 events per day (<link>Part 1: about detecting AD enumeration) on a perimeter with fewer endpoints !

Automation of first investigation pivots were pretty straight forward, focusing on identification of the source of the enumeration. Then pivots were automated to check:

• Kerberos activity of the account coming from the identified source IP (events 4768, 4769) – looking at which services were accessed and potential traces of Kerberos abuse.
• Network activity of the source IP – looking for anomalous behavior that would occur because of enumeration (not restricted to Active Directory) initiated by the potential adversary
• If available, processes executions of the identified user throughout the perimeter (EDR, 4688 events, Sysmon, …) and more particularly on the identified source endpoint (source IP).

And because I really enjoy creating diagrams, here is one to visualize enrichments and pivots we identified for qualifying AD Canaries incidents:

These pivots, along with manual investigation proved to be efficient in identifying the root cause of the enumeration when used during both the tuning, pre-run and run periods.

To sum up this operational feedback, I decided to evaluate the “effort level” of the different tasks involved in this first deployment in production:

Figure 32 – Effort level per deployment task – operational feedback

This is of course a subjective and non-exhaustive notation of my own work, but I hope it will give you some insights on what to expect if you decide to deploy this detection primitive on your SOC perimeters.

Part 3: Conclusion

Detection engineering is a tedious, yet very interesting work. Whereas conceiving new detection techniques might require research methodology and creativity, in-production deployment gives us a second wave of problems to solve in our own SOC environments. As defenders, we need to pick our fights and select what detection mechanism we choose to invest time and efforts in. I hope AD canaries was a good illustration of this approach.

I really enjoyed this (quite long) journey, and I hope this blog will help other defenders tackle AD enumeration detection problem. 

Thanks and shout outs:

Specter Ops researchers Andy Robbins, Lee Christensen and Will Schroeder who published “An ACE Up the Sleeve”

Marie Mion – who was an amazing internship tutor and help me during my research in 2021.

Samir Meghraoua – who helped me a lot while deploying AD Canaries in production in 2022-2023.

Alexandra Toussaint – who pushed me into writing this and helped me throughout the whole process.

Vincent Le Bastard, Nicolas Audiot and all Airbus Protect SOC team.