JustAnotherAdmin

Logo

This is my site for sharing back with the IT world.

View the Project on GitHub soccershoe/JustAnotherAdmin

29 October 2019

Windows Event Logging ...Part 2

by This Guy

Let’s start Configuring Logging All The Things! …Part 2

Please refer to Part 1 from my previous post for some intro into this topic.

Here’s where we start doing things.

Configuration File Locations

Here is where I have placed all my necessary files for building the WEC server. They may be referenced in the scripts or scheduled tasks, so update cautiously if you would like a different location. Go ahead and create these.

Configure your Auditing GPO

Client Auditing is configured via a GPO. I’ve created a GPO specific to Auditing for clients in my environment. Do what’s appropriate for yours. Update the GPO as necessary to increase what is being audited by default from Microsoft.

You can find examples in the links below for what to audit and how best to configure your settings. I’ve included in my files here a copy of my auditing settings. Clients pick which WEF server to grab their configuration from via settings defined in the GPO. GPO’s are pretty flexible if you use Group Policy Preferences (GPP) and use things like AD Site or IP address range if you suspect your environment is larger than what one WEF server could handle.

For redundancy, I’ve created the GPO such that clients will send their events to two different WEC servers at the same time.

Deploying the actual WEC Server (assuming GPO’s already deployed)

This is a quick overview/abbreviated version if you are already familiar with how to build your WEF Server. I’ve included this to provide a logical overview for those who prefer.

  1. Copy WEC-Build, WEC-Scripts, WEC-Subscriptions to C:\ from zip download or existing WEF server.
  2. Import all Scheduled Tasks located in WEC-Scripts.
  3. xcopy C:\WEC-Scripts\WUInstallation\*.* C:\Windows\System32\WindowsPowerShell\v1.0\Modules /E
  4. Create D:\WEC-EventLogs.
  5. Deploy Subscriptions using .\WEC-Deploy-Subscriptions.ps1.
  6. Deploy the Channels.
    • wevtutil um C:\windows\system32\CustomEventChannels.man (only needed if CustomEventChannels.man has been previously loaded)
    • xcopy C:\WEC-Build\CustomEventChannels.* C:\windows\system32
    • wevtutil im C:\windows\system32\CustomEventChannels.man
  7. Restart Server
  8. Set EventLog size using .\WEC-Set-EventlogSize.ps1.
  9. Move EventLogs using .\WEC-Move-Eventlogs.ps1.

the end

Configure the WEC Collectors

This is the actual deployment steps here, unlike the previous section. Most of the work is done via GPO. The Site-Linked GPO’s will be linked to client sites that will define which WEC server they connect to to get their subscriptions. Don’t use Site-Linked GPO’s if you don’t need too. The rest of the configuration is completed in the following sections.

Let’s Deploy some Scheduled Maintenance tasks.

Definitions of the files we are using

  1. WEC-HTTPErr-Grooming.xml - clears the logs in the httperr folder as they fill up the C: drive
  2. WEC-Registry-Grooming.xml - clears client subscriptions in the registry to prevent registry bloat and bad performance
  3. Weekly Reboot.xml - clear any memory leaks
  4. Daily Windows Update - optional: automatically apply windows updates to get ahead of security or other Windows issues. Module, Script and Scheduled task are located in my example files. I wouldn’t call these servers needing any sort of 99.999% uptime. So reboot them and update them often.
  5. System_Microsoft-Windows-Resource-Exhaustion-Detector_2004 - This task will restart the server if event id 2004 shows up in the System event log. 2004 warns of resource exhaustion. It’s happened to me, so I made this.

Deploy the Channels and Move the Eventlogs

Once the DLL has been created using the directions previous (I have some basic directions below, but the link previously provided can help if you want to make some customizations not included here), or using the one I provided, use the following steps to deploy the DLL. This must be executed on each Subscription Manager (WEC server):

  1. Stop the Windows Event Collector Service: net stop Wecsvc
  2. Disable all current WEF subscriptions (if there are any yet). Right-click them in the EventViewer GUI to disable.
  3. Unload the existing manifest via command line: wevtutil um C:\windows\system32\CustomEventChannels.man (it may not exist if you haven’t yet built the server).
  4. Copy your newly created CustomEventChannels.man and CustomEventChannels.dll files into c:\windows\system32. These files are preconfigured using the existing documented paths and located in C:\WEC-Build.
  5. Import the new manifest via command line: wevtutil im C:\windows\system32\CustomEventChannels.man. This creates the defined channels and log files on the WEC servers.
  6. Restart the server.
  7. I also recommend increasing the size of each channel to 2GB. Run the example Powershell below or WEC-Set-EventlogSize.ps1 located in C:\WEC-Scripts.
$xml = wevtutil el | select-string -pattern "WEC"
foreach ($subscription in $xml) {
wevtutil sl $subscription /ms:2194304000
}

Move the event logs to the D: drive to be more flexible with disk size and performance. Disk performance could be a bottleneck (probably not these days but at the least it keeps things straight in my head having the custom logs on another separate disk). Run the below in Powershell or use the WEC-Move-Eventlogs.ps1 located in C:\WEC-Scripts.

Stop-Service wecsvc
$xml = wevtutil el | select-string -pattern "WEC"
foreach ($subscription in $xml) {
wevtutil sl $subscription /lfn:D:\WEC-EventLogs\$subscription.evtx
}
Start-Service wecsvc

Troubleshooting

Ok. This section needs a bit more info. But here’s what I have for now.

  1. WinRM connection issues get logged here on the server side: C:\Windows\System32\LogFiles\HTTPERR. Might be able to find some crumbs to follow when clients are unable to connect.
  2. Check your firewall logs that you are allowing for WinRM traffic, and that your GPO’s have WinRM configured as well.

Maintenance Notes

  1. Registry Pruning
    • Delete keys under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\EventCollector\Subscriptions\SubscriptionName for PCs that are no longer active. A script with a scheduled task is deployed on each WEC server that just looks for the last heartbeat time and has intelligence to remove keys here that with HeartBeat times older than a specific threshold.
  2. HTTPErr log pruning
    • Delete files under C:\Windows\System32\Logfiles\HTTPErr The files record errors for the WinRM connections. WinRM uses HTTP.SYS, the same driver as IIS, which means it logs certain things by default.
  3. Keep each WEF server under 2000 subscriptions if you prefer to keep the GUI handy. This should be a manageable size while still being able to use the MMC.exe GUI usable.
  4. A weekly scheduled reboot task is created to keep the server healthy from memory leaks. Windows may be a little leaky.

Appendix

Here’s the link to all the files I’m using for my deployment.

Building DLL Overview Bonus Section

This only needs to be completed if the WEC Subscriptions need to be changed from the build already supplied from the .dll and .man. I’m not a developer and can hardly say I have a grasp on what I’m doing here.

Prereq: Windows 10 SDK from Creators Edition (I think you need this specific version because MS moved some stuff around and moved, or removed, the ecmangen.exe file from other versions)

Editing

Launch the Manifest Generator: “C:\Program Files (x86)\Windows Kits\10\bin\x64\ecmangen.exe” Load the CustomEventChannels.man file. Make any changes to the file. Ensure the following settings are observed:

Compiling

To compile, perform the following from a cmd.exe shell:

Deployment

For each WEF server you need to deploy this to, perform the following:

  1. Disable the Windows Event Collector Service: net stop Wecsvc
  2. Disable all current WEF subscriptions.

Unload the current Event Channel file:

  1. wevtutil um C:\windows\system32\CustomEventChannels.man
  2. Copy (and replace) the following files to each WEF server under C:\Windows\system32:
    • CustomEventChannels.dll
    • CustomEventChannels.man
  3. Load the new Event Channel file:
    • wevtutil im C:\windows\system32\CustomEventChannels.man
  4. Resize the log files: 
    $xml = wevtutil el | select-string -pattern "WEC"
foreach ($subscription in $xml) {
wevtutil sl $subscription /ms:2194304000
}
  5. Re-enable the WEF subscriptions.
  6. Re-enable the Windows Event Collector service

Sources

tags:

comments powered by Disqus