Intermittent: The Server is not operational

Question

The below error is shown when running TEST Connection in AD Production / AD Stage and AD Test.

The user server for ou=someusers, ou=anOU,dc=more,dc=morestuff,dc=stuff: The server is not operational. . HRESULT:.(0x8007203AJ”, “Failed to connect to the server for ou=someusers,ou=anOUusers,dc=more,dc=more, dc=stuff: The server is not operational. . HRESULT:[0x8007203A)” [ InvalidConfigurationException ] [ Possible suggestions ] Ensure that: a) SearchDN is valid. b) The user is active. c) The user is not locked. d) Domain certificate is available in trusted root folder on IQService machine if Domain Configuration TLS is enabled. [ Error details ]

The IQService is installed on a windows server on the same network as the Domain controllers. We have TLS turned on in IQService, and looking at the Wireshark captures; there appears to be a pause (approx 15-20 seconds)  before the error comes back. The window server is 2019, and the operational level is 2012R2. Identity IQ is version: 8.2p1. We have turned off the domain controllers (everything but Primary) to only have one domain controller (in the lower environment)- to ensure there isn't an issue with clustering. There doesn't appear to be a heavy load on the servers; we have checked the usage, and it's always been low (less than 80% of usage, memory, and CPU). The IQService server is not being overwhelmed either as it sees maybe a request every few hours. We have stressed the lower environment to see about trying to capture more issues and have put out (test requests) at about one a min. These test requests are not write requests, only read. (As a note, we see this coming across in write as well). Thoughts on things to test and how to run those tests?

I can validate:

1. The SearchDN is valid (had three pairs of eyes validate, I supposed there could be a white space character etc I will double check that...)
2. The user is not locked else none of the requests would work
3. The user is active
4. Would have to validate the domain certificate being in the proper place, but would assume that all requests would fail if it wasn't.

One thought that was presented is that there might be an issue with ephemeral ports (1024-65535) between IQService and the Domain Controllers. Any suggestions on an approach to testing and validating that theory? thanks!!

UPDATE:

Have opened the validated the ports with the following commands:

netsh int ipv4 set dynamicport tcp start=1024 num=64511
netsh int ipv4 show dynamicport TCP

Update: adding partial application.xml

  <entry key="domainSettings">
    <value>
      <List>
        <Map>
          <entry key="authenticationType" value="simple"/>
          <entry key="authorizationType" value="simple"/>
          <entry key="domainDN" value="%%AD_DOMAIN_DN%%"/>
          <entry key="domainIterateSearchFilter"/>
          <entry key="domainNetBiosName"/>
          <entry key="forestName" value="%%AD_FOREST%%"/>
          <entry key="password" value="%%AD_PASSWORD%%"/>
          <entry key="port" value="636"/>
          <entry key="servers">
            <value>
              <List>
                <String>%%AD_DOMAIN_SERVER%%</String>
              </List>
            </value>
          </entry>
          <entry key="useSSL">
            <value>
              <Boolean>%%AD_IQSERVICE_TLS%%</Boolean>
            </value>
          </entry>
          <entry key="user" value="%%AD_USER%%"/>
        </Map>
      </List>
    </value>
  </entry>

Some additional logs from IQService:

03/04/2022 11:40:19 : RpcHandler [ Thread-11 ] DEBUG : "Initiating the serviceState for 122"
03/04/2022 11:40:19 : RpcHandler [ Thread-11 ] INFO : "Calling Service [ADConnector] and method[testConfiguration] "
03/04/2022 11:40:19 : Impersonator [ Thread-11 ] DEBUG : "Authenticating as User [svcAccount] domain [dom]"
03/04/2022 11:40:19 : Impersonator [ Thread-11 ] DEBUG : "User [svcAccount] domain [dom] -> Authenticated"
03/04/2022 11:40:19 : AbstractConnector [ Thread-11 ] DEBUG : "ENTER AbstractConnector"
03/04/2022 11:40:19 : AbstractConnector [ Thread-11 ] DEBUG : "EXIT AbstractConnector"
03/04/2022 11:40:19 : ADConnectorServices [ Thread-11 ] DEBUG : "ENTER prepare"
03/04/2022 11:40:19 : ADConnectorServices [ Thread-11 ] DEBUG : "EXIT prepare"
03/04/2022 11:40:19 : ADConnectorServices [ Thread-11 ] DEBUG : "ENTER testConfiguration"
03/04/2022 11:40:19 : ADConnectorServices [ Thread-11 ] DEBUG : "ENTER bind"
03/04/2022 11:40:19 : ADConnectorServices [ Thread-11 ] DEBUG : "ENTER buildURLFromIdentity"
03/04/2022 11:40:19 : ADConnectorServices [ Thread-11 ] INFO : "Server [server.dom.foo.test] DN [ou=users,ou=tii,dc=dom,dc=foo,dc=test]"
03/04/2022 11:40:19 : Util [ Thread-11 ] DEBUG : "FQDN for server.dom.foo.test: server.dom.foo.test"
03/04/2022 11:40:19 : ADConnectorServices [ Thread-11 ] DEBUG : "Derived URL[LDAP://server.dom.foo.test/ou=users,ou=tii,dc=dom,dc=foo,dc=test] original [ou=users,ou=tii,dc=dom,dc=foo,dc=test]"
03/04/2022 11:40:19 : ADConnectorServices [ Thread-11 ] DEBUG : "EXIT buildURLFromIdentity"
03/04/2022 11:40:19 : ADConnectorServices [ Thread-11 ] INFO : "Bind with User new DirectoryEntry(LDAP://server.dom.foo.test/ou=users,ou=tii,dc=dom,dc=foo,dc=test, [email protected] authType=SecureSocketsLayer, ServerBind)"
03/04/2022 11:40:49 : ADConnectorServices [ Thread-11 ] DEBUG : "Failed to connect to the URL : LDAP://server.dom.foo.test/ou=users,ou=tii,dc=dom,dc=foo,dc=test : The server is not operational.
"
03/04/2022 11:40:49 : ADConnectorServices [ Thread-11 ] ERROR : "Caught exception in testConfigurationSystem.Exception: Failed to connect to the server for ou=users,ou=tii,dc=dom,dc=foo,dc=test:The server is not operational. . HRESULT:[0x8007203A]
   at sailpoint.services.ADConnectorServices.bind(String distinguishedName, Boolean isCrossForest, Boolean isCrossDomain, String serverToBind, Boolean isCrossDomainMove, Boolean bindForShadow)
   at sailpoint.services.ADConnectorServices.doTestConfiguration(Hashtable registry)"
03/04/2022 11:40:49 : ADConnectorServices [ Thread-11 ] DEBUG : "ENTER bind"
03/04/2022 11:40:49 : ADConnectorServices [ Thread-11 ] DEBUG : "ENTER buildURLFromIdentity"
03/04/2022 11:40:49 : ADConnectorServices [ Thread-11 ] INFO : "Server [server.dom.foo.test] DN [ou=fuu,ou=dom users,dc=dom,dc=foo,dc=test]"
03/04/2022 11:40:49 : Util [ Thread-11 ] DEBUG : "FQDN for server.dom.foo.test: server.dom.foo.test"
03/04/2022 11:40:49 : ADConnectorServices [ Thread-11 ] DEBUG : "Derived URL[LDAP://server.dom.foo.test/ou=fuu,ou=dom users,dc=dom,dc=foo,dc=test] original [ou=fuu,ou=dom users,dc=dom,dc=foo,dc=test]"
03/04/2022 11:40:49 : ADConnectorServices [ Thread-11 ] DEBUG : "EXIT buildURLFromIdentity"
03/04/2022 11:40:49 : ADConnectorServices [ Thread-11 ] INFO : "Bind with User new DirectoryEntry(LDAP://server.dom.foo.test/ou=fuu,ou=dom users,dc=dom,dc=foo,dc=test, [email protected] authType=SecureSocketsLayer, ServerBind)"
03/04/2022 11:41:19 : ADConnectorServices [ Thread-11 ] DEBUG : "Failed to connect to the URL : LDAP://server.dom.foo.test/ou=fuu,ou=dom users,dc=dom,dc=foo,dc=test : The server is not operational.
"
03/04/2022 11:41:19 : ADConnectorServices [ Thread-11 ] ERROR : "Caught exception in testConfigurationSystem.Exception: Failed to connect to the server for ou=fuu,ou=dom users,dc=dom,dc=foo,dc=test:The server is not operational. . HRESULT:[0x8007203A]
   at sailpoint.services.ADConnectorServices.bind(String distinguishedName, Boolean isCrossForest, Boolean isCrossDomain, String serverToBind, Boolean isCrossDomainMove, Boolean bindForShadow)
   at sailpoint.services.ADConnectorServices.doTestConfiguration(Hashtable registry)"
03/04/2022 11:41:19 : ADConnectorServices [ Thread-11 ] DEBUG : "ENTER bind"
03/04/2022 11:41:19 : ADConnectorServices [ Thread-11 ] DEBUG : "ENTER buildURLFromIdentity"
03/04/2022 11:41:19 : ADConnectorServices [ Thread-11 ] INFO : "Server [server.dom.foo.test] DN [ou=users,ou=coo,dc=dom,dc=foo,dc=test]"
03/04/2022 11:41:19 : Util [ Thread-11 ] DEBUG : "FQDN for server.dom.foo.test: server.dom.foo.test"
03/04/2022 11:41:19 : ADConnectorServices [ Thread-11 ] DEBUG : "Derived URL[LDAP://server.dom.foo.test/ou=users,ou=coo,dc=dom,dc=foo,dc=test] original [ou=users,ou=coo,dc=dom,dc=foo,dc=test]"
03/04/2022 11:41:19 : ADConnectorServices [ Thread-11 ] DEBUG : "EXIT buildURLFromIdentity"
03/04/2022 11:41:19 : ADConnectorServices [ Thread-11 ] INFO : "Bind with User new DirectoryEntry(LDAP://server.dom.foo.test/ou=users,ou=coo,dc=dom,dc=foo,dc=test, [email protected] authType=SecureSocketsLayer, ServerBind)"
03/04/2022 11:41:49 : ADConnectorServices [ Thread-11 ] DEBUG : "Failed to connect to the URL : LDAP://server.dom.foo.test/ou=users,ou=coo,dc=dom,dc=foo,dc=test : The server is not operational.
"
03/04/2022 11:41:49 : ADConnectorServices [ Thread-11 ] ERROR : "Caught exception in testConfigurationSystem.Exception: Failed to connect to the server for ou=users,ou=coo,dc=dom,dc=foo,dc=test:The server is not operational. . HRESULT:[0x8007203A]
   at sailpoint.services.ADConnectorServices.bind(String distinguishedName, Boolean isCrossForest, Boolean isCrossDomain, String serverToBind, Boolean isCrossDomainMove, Boolean bindForShadow)
   at sailpoint.services.ADConnectorServices.doTestConfiguration(Hashtable registry)"
03/04/2022 11:41:49 : ADConnectorServices [ Thread-11 ] DEBUG : "ENTER bind"
03/04/2022 11:41:49 : ADConnectorServices [ Thread-11 ] DEBUG : "ENTER buildURLFromIdentity"
03/04/2022 11:41:49 : ADConnectorServices [ Thread-11 ] INFO : "Server [server.dom.foo.test] DN [ou=boo,ou=dom users,dc=dom,dc=foo,dc=test]"
03/04/2022 11:41:49 : Util [ Thread-11 ] DEBUG : "FQDN for server.dom.foo.test: server.dom.foo.test"
03/04/2022 11:41:49 : ADConnectorServices [ Thread-11 ] DEBUG : "Derived URL[LDAP://server.dom.foo.test/ou=boo,ou=dom users,dc=dom,dc=foo,dc=test] original [ou=boo,ou=dom users,dc=dom,dc=foo,dc=test]"
03/04/2022 11:41:49 : ADConnectorServices [ Thread-11 ] DEBUG : "EXIT buildURLFromIdentity"
03/04/2022 11:41:49 : ADConnectorServices [ Thread-11 ] INFO : "Bind with User new DirectoryEntry(LDAP://server.dom.foo.test/ou=boo,ou=dom users,dc=dom,dc=foo,dc=test, [email protected] authType=SecureSocketsLayer, ServerBind)"
03/04/2022 11:42:04 : ADConnectorServices [ Thread-11 ] DEBUG : "EXIT bind"
03/04/2022 11:42:04 : ADConnectorServices [ Thread-11 ] INFO : "Bind to ou=boo,ou=dom users,dc=dom,dc=foo,dc=test is Successful"
03/04/2022 11:42:04 : ADConnectorServices [ Thread-11 ] DEBUG : "Parent Container is: LDAP://server.dom.foo.test/ou=dom users,dc=dom,dc=foo,dc=test"
03/04/2022 11:42:04 : ADConnectorServices [ Thread-11 ] DEBUG : "ENTER bind"
03/04/2022 11:42:04 : ADConnectorServices [ Thread-11 ] DEBUG : "ENTER buildURLFromIdentity"
03/04/2022 11:42:04 : ADConnectorServices [ Thread-11 ] INFO : "Server [server.dom.foo.test] DN [ou=joo,ou=dom users,dc=dom,dc=foo,dc=test]"
03/04/2022 11:42:04 : Util [ Thread-11 ] DEBUG : "FQDN for server.dom.foo.test: server.dom.foo.test"
03/04/2022 11:42:04 : ADConnectorServices [ Thread-11 ] DEBUG : "Derived URL[LDAP://server.dom.foo.test/ou=joo,ou=dom users,dc=dom,dc=foo,dc=test] original [ou=joo,ou=dom users,dc=dom,dc=foo,dc=test]"
03/04/2022 11:42:04 : ADConnectorServices [ Thread-11 ] DEBUG : "EXIT buildURLFromIdentity"
03/04/2022 11:42:04 : ADConnectorServices [ Thread-11 ] INFO : "Bind with User new DirectoryEntry(LDAP://server.dom.foo.test/ou=joo,ou=dom users,dc=dom,dc=foo,dc=test, [email protected] authType=SecureSocketsLayer, ServerBind)"
03/04/2022 11:42:04 : ADConnectorServices [ Thread-11 ] DEBUG : "EXIT bind"
03/04/2022 11:42:04 : ADConnectorServices [ Thread-11 ] INFO : "Bind to ou=joo,ou=dom users,dc=dom,dc=foo,dc=test is Successful"

Answer 1

• You are encountering this error because the ‘Application.xml’ file doesn’t contain correct configuration for connecting to the AD Server through the IQ service connector. The ‘application.xml’ file should contain the configuration to include the DC servers’ details in the following format.

Existing ‘application.xml’ configuration might be as below: -

 <entry key=”domainSettings”>
 <value>
 <List>
 <Map>
 <entry key=”authorizationType” value=”simple”/>
 <entry key=”domainDN” value=”DC=example,DC=com”/>
 <entry key=”password” value=”1:iIopEeOL5KrLoSjYKvh/Ww==”/>
 <entry key=”port” value=”389?/>
 <entry key=”servers”/>
 <entry key=”useSSL”>
 <value>
 <Boolean></Boolean>
 </value>
 </entry>
 <entry key=”user” value=”EXAMPLE\Administrator”/>
 </Map>
 </List>
 </value>
 </entry>

New ‘application.xml’ configuration should be as below: -

<entry key=”domainSettings”>
<value>
<List>
 <Map>
 <entry key=”authorizationType” value=”simple”/>
 <entry key=”domainDN” value=”DC=example,DC=com”/>
 <entry key=”password” value=”1:iIopEeOL5KrLoSjYKvh/Ww==”/>
 <entry key=”port” value=”389?/>
 <entry key=”servers”>
  <value>
  <List>
 <String>172.16.153.185</String>
 </List>
 </value>
 <entry key=”useSSL”>
  <value>
  <Boolean></Boolean>
 </value>
 </entry>
 <entry key=”user” value=”EXAMPLE\Administrator”/>
 </Map>
 </List>
 </value>
 </entry>

• By making the above changes, your IQ service should be able to ‘TEST’ connect to the domain controller IP as specified by you in the ‘application.xml’ file. Along with it, also ensure that the required ports for the sailpoint IQ service are open correctly from the member server to the domain controller as well as the internal AD replication ports are also open which are mentioned below: -

TCP UDP 135, 137, 138, 139, 445, 389, 636, 3268, 3269, 88, 53, 1512, 42, 49152-65535. These ports are associated with various services regarding AD, viz., RPC Endpoint mapper, DNS, WINS resolution, replication, RPC dynamic ports, etc.