I have to say that after 25 years of messing around with Group Policy, I am constantly amazed by the new things I still learn about the technology and it’s behavior. Especially when those things are substantial and weird. Such was my experience last week when a colleague indicated that GP processing was failing for one domain in a forest, when a child domain in that forest was completely down. In that case, there was a GPO from the down forest linked to an OU in the functional domain. That got me thinking about why that would break GP processing outright. So I got to testing…
Normally, when a cross-domain-linked GPO is deleted from it’s source domain, or becomes unavailable for whatever reason, the computer or user processing that link in their remote container doesn’t blink. It will register that the linked GPO is unavailable and move on with GP processing. But this was a different situation. In this case, the whole domain hosting the cross-linked GPO was unavailable. That’s not something I had encountered before. But surely GP processing would behave the same way in this case as it does in the orphaned GPO case….right?…right?????
Um, no.
I set about trying to re-create this issue. Because I didn’t have a child domain at hand that I could render unavailable, I decided to just fake it. I took an OU in my test domain that had a GPO linked to it, and I manually added a new link on the gpLink attribute of the OU, to a fictitious GPO in a fictitious child domain, as shown here (shown in bold below):
[LDAP://cn={2600fe1e-07b9-40b1-acd6-d64405d5e6bd},cn=policies,cn=system,DC=testdom,DC=net;0][LDAP://CN={34B2F340-016D-11D2-945F-00C04FB984F9},CN=Policies,CN=System,DC=west,DC=testdom,DC=net;0]
(see this article for more information on how GPO links are formed in AD)
As you can see from the above link list, I simply added a reference to a non-existent GPO GUID, in a non-existent child domain called “west”. The effect, as I’m about to show, was complete and utter destruction of GP Processing.
Then I fired up the event viewer and gpsvc logging on a computer that processed links on this OU and went to work.
The first thing I did was run gpupdate /target:computer to trigger a GP refresh on the computer object (NOTE: everything I’m writing here works the same if you are targeting a user object)
Sure enough, I saw this familiar message from gpupdate:
Seeing this, I expected to see some errors in the Group Policy Operations Log in Event Viewer. Curiously, I didn’t find any error events. What I did see is one information event with this useful bit of information:
What was amazing about this error was the lack of information. I dug a bit deeper into the event and did find a Win32 error code of “58”, which corresponds to “The specified server cannot perform the requested operation.” No shit…
The other amazing part about this was that GP processing failed completely. Like, no policy was processed…at all…nothing. This meant that “core” processing* failed completely.
* For more information about the different phases of GP Processing, check out this article
I decided to dig in a little deeper, and I fired up the resulting gpsvc trace log. (check out this article for more information on gpsvc logging). What I found in that log was only slightly more illuminating. Namely, that file showed the following errors in GP processing, right after the GP engine tries to resolve each linked GPO in it’s hierarchy:
GPSVC(490.930) 16:47:06:828 EvaluateDeferredGPOs: Doing an ldap bind to cross-domain <west.testdom.net> GPSVC(490.930) 16:47:06:828 EvaluateDeferredGPOs: Failed to connect with 81 GPSVC(490.930) 16:47:06:828 GetGPOInfo: EvaluateDeferredGPOs failed. Exiting GPSVC(490.930) 16:47:06:828 GetGPOInfo: Leaving with 0 GPSVC(490.930) 16:47:06:828 GetGPOInfo: ******************************** GPSVC(490.930) 16:47:06:828 ProcessGPOs(Machine): GetGPOInfo failed.
It seems fair to assume that the GP engine simple can’t handle a GPO that exists in an uncontact-able domain, and instead of skipping it, just completely barfs during core processing. Then I started thinking that it would be incredibly easy for anyone with write permissions on gpLink on an OU anywhere in the hierarchy of GP processing, to break all GP processing simply by throwing it one of these unexpected links. Not good.
If there is any silver lining on this, it’s that when GP processing fails spectacularly like this, it doesn’t have any impact on any settings that are already applied to the computer or user. They remain as-is. But of course any new settings that one is trying to deliver, would never get there. And as I’ve shown, there aren’t many red flags or warning sirens to tell you why. All the more reason to keep tight control over who can link GPOs within your environment!