The Problem

ISE PSNs connect to AD domain controllers to authenticate users on the network. Let’s say there are 4 ISE PSNs and 3 AD Domain Controllers. Seemingly out of no where ISE “backlists” all 3 of our DCs. You only know of this because of the ISE Alarm “joined domain is unavailable” – Hint: Check the AD Connector Report in ISE. The Active Directory Integration Guide for 1.3 will tell you if “AD connector cannot communicate with it [DC] for some reason” it will blacklist that domain controller.

In my case, there were two issues: one our Firewall was randomly dropping traffic causing all 3 DCs to get blacklisted, and two our DCs are sending TCP RST packets causing randomly a PSN to blacklist 1 DC (not service affecting, ).

ISE blacklists Domain Controllers, if there is an unrecoverable network or server error (the idea is to prevent ISE from using a bad DC) that DC is blacklisted and the DC discovery process is kicked off to find a better one.

There are two types for blacklists

  1. Network blacklists
    • DC stays in blacklist from 10 seconds
    • The same failed or blacklisted DC can be selected in if it responds to CLDAPs in discovery process
    • Network blacklist reasons
      1. The DC sends a TCP RST in the middle of a conversation
      2. Loss of network connectivity / authentication traffic LDAP, RPC, SMB, or Kerberos
      • Network timeouts are 3-5 seconds
  2. Server (DC) blacklist
    • DC stays in blacklist for 5 minutes
    • Server blacklist reasons
      • If DC is in the broken replication state or not decommissioned properly

Detailed Blacklist Reasons

To debug this process turn on debugging (Administration -> Logging -> Debug Log Configuration. Pick the PSN. Componet Name: Active Directory Log Level: Debug.)

i.e – Logs I see for blacklisting due to seeing TCP rests

01/01/0000 15:25:26,VERBOSE,139663245661952,RdrSocketRead(0x7f04b7d5b0b0, 40): errno=104(ECONNRESET),lwio/server/rdr/socket.c:2055
01/01/0000 15:25:26,WARNING,139663245661952,Add to black list: Domain=DOMAIN.com DC=DC.DOMAIN.com addr=1.1.1.1 status=c000020d,lwio/server/rdr/socket.c:979 
- (C000020D STATUS_CONNECTION_RESET http://www.tenox.net/links/cache/ntstatus.html)
01/01/0000 15:25:26,WARNING,139663245661952,Added to black list: Domain=DOMAIN.com DC=DC.DOMAIN.com addr=1.1.1.1 TTL=15:25:36 reason=Network,lwadvapi/threaded/dcmanager.cpp:234 
- (TTL is 10 Seconds from now -- Reason is Network)