KWTS Kerberos LDAP, SSO, Proxy authentication problems [Kaspersky Web Traffic Security]

[Antipova Anna]

Advice and Solutions (Forum Knowledgebase) Disclaimer. Read before using materials.

Introduction

Often problems with Kerberos are difficult to diagnose but they occur if you're deploying KWTS for the first time. There are three functional places in the product where Kerberos authentication can be used:

1. Proxy authentication
   This is needed for users to authenticate on the proxy server automatically without login prompt. If login prompt pops-up, then Kerberos authentication failed.
2. LDAP authentication 
   This is needed for KWTS to synchronize LDAP cache with LDAP servers (simply put - with domain controllers). This cache is used in Rules creation and if KWTS has user login information for a given session supplied by proxy server, then traffic can be matched against those Rules that are defined by groups in AD for example.

3. SSO
   This is needed to authenticate users on KWTS web administration console itself.

   SSO works only for one domain, as it is for KSMG as well.

Check the documentation https://support.kaspersky.com/KWTS/6.1/en-US/166491.htm 

Read Kerberos and LDAP parts.

Terminology

FQDN - https://en.wikipedia.org/wiki/Fully_qualified_domain_name . In our use cases looks like: kwts.example.com

SPN - Unique ID of the service in the network for authentication over the Kerberos protocol. In our use cases looks like: HTTP/<FQDN>@<realm Active Directory service in the upper case>  or  HTTP/*****@*****.tld

Creating keytabs for multiple nodes

For LDAP, SSO and Proxy authentication you need to create two keytabs:

1. Keytab For LDAP without SPN
2. Keytab for SSO and Proxy with SPN

In this example there are two servers in cluster kwts1.example.com and kwts2.example.com and Realm (Domain) is EXAMPLE.COM.

Please bear in mind that hostname of KWTS node in OS MUST be in lower-case. If it's in upper-case change hostname via command

hostnamectl set-hostname kwts1.example.com

First you remove any existing kwts users from AD and create new ones *****@*****.tld and *****@*****.tld

Then for LDAP you don't need SPN, so create LDAP keytab like so (replace <password> with user password):

ktpass.exe /crypto AES256-SHA1 /ptype KRB5_NT_PRINCIPAL /out C:\kwts-ldap.keytab /princ kwts-ldap-user@EXAMPLE.COM /pass <password>

You can now add C:\kwts-ldap.keytab to LDAP settings and force LDAP synchronization.

For SSO and Proxy authentication you then create a first keytab like so (do not use upper case letters in FQDN part kwts1.example.com/kwts2.example.com of SPN, this will not work for SSO):

ktpass.exe -princ HTTP/kwts1.example.com@EXAMPLE.COM -mapuser kwts-control-user@EXAMPLE.COM -crypto AES256-SHA1 -ptype KRB5_NT_PRINCIPAL -pass <password> -out C:\kwts-control-1.keytab

Then using this keytab you create a new keytab with a second record in it:

ktpass.exe -princ HTTP/kwts2.example.com@EXAMPLE.COM -mapuser kwts-control-user@EXAMPLE.COM -crypto AES256-SHA1 -ptype KRB5_NT_PRINCIPAL -pass <password> -in C:\kwts-control-1.keytab -out C:\kwts-control-2.keytab -setupn -setpass

If there are more servers, then please add more entries in the same manner.

You can now add C:\kwts-control-2.keytab to SSO settings.

When testing SSO you should use https://kwts1.example.com and https://kwts2.example.com URLs, not IP.

If you are asked for credentials then it means that SSO doesn't work. 

For SSO on Internet Explorer and Chrome it is important that https://kwts1.example.com and https://kwts2.example.com are added to Local Intranet zones in IE settings (refer to https://support.kaspersky.com/ksmg/228052 - article is for KSMG, but fully applicable to KWTS as well):

1. Open Internet Explorer and click the Settings gear icon in the top-right corner. Select Internet options.
2. Select the Security tab.
3. Select the Local Intranet zone and click the Sites button.
4. Make sure that the first two options, Include all local (intranet) sites not listed in other zones and Include all sites that bypass the proxy server are checked.
5. Click Advanced and add the KWTS addresses, one at a time, to the list of websites. In this example add https://kwts1.example.com and https://kwts2.example.com.  Click Close.
6. Click OK to save your configuration changes.

For Firefox:

1. Open the low level Firefox configuration page by loading the about:config page.
2. In the Search text box, enter: network.negotiate-auth.trusted-uris
3. Double-click the network.negotiate-auth.trusted-uris preference and enter KWTS address. Separate multiple addresses with a comma.
4. Click OK.

Now, if SSO works fine you can add the same C:\kwts-control-2.keytab to Proxy authentication settings and test it.

When testing proxy authentication make sure proxy address in browser settings is set to kwts1.example.com or kwts2.example.com. IP address will not work.

Notable differences and restrictions in Kerberos keytab requirements

1. For Proxy and SSO authentication FQDN that is specified in keytab must always match the real and used FQDN. For Proxy authentication the address that is used in browser proxy settings MUST match the FQDN in keytab.
   For SSO, the address in the address bar in the browser that is used to access KWTS web interface MUST match the FQDN in the keytab and MUST match the real FQDN of KWTS and FQDN that is configured in OS. But for LDAP the FQDN in keytab SPN should just have valid records in DNS including PTR record. It is also not necessary for LDAP keytab to have an SPN at all while you must have it for Proxy and SSO;

2. For LDAP authentication it is not possible to have multiple SPN entries in keytab. But in case of Proxy and SSO authentication you can create multiple entries. However for LDAP it is not needed (see 1);
3. You cannot have SPN duplicates. Meaning that you can't create two different keytabs that have duplicate SPN (which includes FQDN);
4. User which was used to create keytab must contain only latin characters in Distinguished Name, so in the entire path to the user in AD there must not have Cyrillic or other characters.

To sum up usually you must create two keytabs:

1. For Proxy and SSO that have all the SPNs with all FQDNs of proxy and secondary nodes (to which fallback if control node fails);
2. For LDAP that doesn't have an SPN or has one that just has an FQDN with valid DNS records but is not duplicate to any SPN in 1.

How KWTS connects to LDAP servers using keytab

There is no ldap server address configuration in KWTS.

It takes the REALM (Domain) from keytab,  for example EXAMPLE.COM, then the following DNS requests of type SRV are sent:

_ldap._tcp.example.com

_kerberos._tcp.example.com

In KWTS console such requests can be reproduced with

dig srv _ldap._tcp.example.com dig srv _kerberos._tcp.example.com

There you will see a list of domain controllers, ports, weighs and priorities. For more information on SRV records see https://en.wikipedia.org/wiki/SRV_record

LDAP servers are listed in _ldap._tcp.example.com , default port 389.

_kerberos._tcp.example.com is needed for Kerberos, default port 88.

Connection is tried to each one from the list (one at a time, with a timeout) until a it is successfully established or the list is exhausted.

LDAP+Kerberos diagnostics

To diagnose LDAP synchronization issues first turn on Debug level traces (link). Then reproduce the problem by clicking Synchronize button in LDAP settings.

In 10-20 minutes depending on the size of your domains and number of them you should be able to check traces either directly on the server like so:

grep LdapC /var/log/kwts-traces | less

or by getting the built-in collect and looking in kwts-traces files by other means (link).

For example, if you see the following errors:

Sep 29 15:30:01 srv-proxy2 KWTS LdapCache[33227]: 33227 DBG trying to connect ldap://server.local:389 Sep 29 15:30:01 srv-proxy2 KWTS LdapCache[33227]: 33227 WRN Couldn't connect ldap://server.local:389 Sep 29 15:30:01 srv-proxy2 KWTS LdapCache[33227]: 33227 ERR CheckFailedException - LDAP error (-2) : Local error - Cannot perform LDAP SASL interactive bind operation. At /tmp/buildbot/core_ldap_cache-kwts_linux-64/build/source/ldap/connection.cpp(203)

Then make sure you can connect to server.local:389 with telnet and verify that:

1. On KWTS you can resolve server.local by FQDN and resolve PTR for its IP. Domain controller PTR record should be matched to A record otherwise Kerberos will not work and the error will be exactly as in above log;
2. On server.local you can resolve KWTS by FQDN and resolve PTR for its IP;
3. Time on KWTS servers is synchronized properly with an NTP server.

How to use multiple domains in LDAP

1. Create multiple LDAP connections in LDAP settings, one for each domain and use a separate keytab for each;
2. Make sure that a specific DNS server can resolve _kerberos._tcp. and _ldap._tcp. SRV records for each domain.  For that in the main domain DNS server you can create stub DNS zones for each domain;
3. Configure KWTS to use that DNS server.

Proxy authentication diagnostics

Check squid logs:

/var/log/squid/cache.log  shouldn't contain errors regarding Kerberos or NTLM   /var/log/squid/access.log should contain usernames of authenticated users

For example:

negotiate_kerberos_auth.cc(182) : pid=63851 :2020/06/03 11:28:00| negotiate_kerberos_auth: ERROR: gss_accept_sec_context() failed: Unspecified GSS failure.  Minor code may provide more information. Request ticket server HTTP/kwts.test.local@TEST.LOCAL kvno 2 found in keytab but not with enctype rc4-hmac

Means that keytab was created with AES128 or AES256 encryption but the user with which it was created doesn't have AES128 or AES256 enabled in user settings (Properties → Account → Account Options → This account supports Kerberos AES 128/256 bit encryption).

Trace kinit:

1. Run on KWTS:

   KRB5_TRACE=/tmp/kr.tr kinit -Vkt /etc/squid/auth_krb5.keytab HTTP/FQDN@REALM

   where HTTP/FQDN@REALM is the SPN of used keytab. For standalone (not built-in proxy) instead of /etc/squid/auth_krb5.keytab specify the real path to keytab);

2. Check output of the command AND /tmp/kr.tr file, it should contain detailed trace.

SSO authentication diagnostics

Check logs in /var/log/kaspersky/kwts/extra/

For example in webapi.log if you see

ERROR:root:GSSError: (('Unspecified GSS failure. Minor code may provide more information', 851968), ('Request ticket server HTTP/*****@*****.tld found in keytab but does not match server principal HTTP/dpc-kwts-01.example.com@', 100005)) [pid: 14648|app: 0|req: 701/53178] 10.199.5.19 () {50 vars in 4127 bytes} [Wed Oct 14 14:12:46 2020] GET /web/api/get-session-info?cb=1602673966446 => generated 224 bytes in 12 msecs (HTTP/1.1 403) 3 headers in 93 bytes (1 switches on core 0)

That means that there are two PTR records in DNS for KWTS IP address. Remove one that is not for the FQDN that should be used to access KWTS web interface.

./celery.log:[2020-06-02 18:00:47,860: ERROR/ForkPoolWorker-1] there are no valid principal for HTTP service on kwts.example.com host in keytab data; <class 'kerberos.KrbError'>: ('Principal not found in keytab', -1)   ./webapi.log:ERROR:root:there are no valid principal for HTTP service on kwts.example.com host in keytab data; <class 'kerberos.KrbError'>: ('Principal not found in keytab', -1)

Means there is no SPN record in keytab for the FQDN which was accessed by the web browser.

./webapi.log:ERROR:root:GSSError: (('Unspecified GSS failure. Minor code may provide more information', 851968), ('Request ticket server HTTP/*****@*****.tld kvno 6 found in keytab but not with enctype rc4-hmac', 100005))

Means the keytab was created with AES128 or AES256 cryptography but it is not enabled in user settings in AD.

./webapi.log:ERROR:root:GSSError: (('Unspecified GSS failure. Minor code may provide more information', 851968), ('Request ticket server HTTP/*****@*****.tld kvno 8 not found in keytab; keytab is likely out of date', 100005))

Means keytab was created with wrong user password or password was changed after keytab was created.

Time out sync between KWTS and DC

On KWTS time is far behind DC:

./webapi.log:ERROR:root:GSSError: (('Unspecified GSS failure. Minor code may provide more information', 851968), ('Ticket not yet valid', 100005))

On KWTS time is far ahead DC:

./webapi.log:ERROR:root:GSSError: (('Unspecified GSS failure. Minor code may provide more information', 851968), ('Ticket expired', 100005))

On KWTS time is slightly ahead DC:

./webapi.log:ERROR:root:GSSError: (('Unspecified GSS failure. Minor code may provide more information', 851968), ('Clock skew too great', 100006))

Also you can upload the keytab to the server via WinSCP and trace kinit as in proxy authentication diagnostics.

If kinit is successful but browser doesn't authenticate check that web interface FQDN is added to trusted servers in browser settings. See for example: https://docs.cloudera.com/documentation/enterprise/5-12-x/topics/cdh_sg_browser_access_kerberos_protected_url.html

Please note that if the keytab that you are adding does not contain an SPN with FQDN from 'hostnamectl' command output, you will get "Invalid keytab file for the Control node". In that case change hostname to FQDN with:

hostnamectl set-hostname FQDN

Useful tricks

Run the following in PowerShell (on DC) to get the list all users for which keytabs were created with SPN that starts with "HTTP/"

Get-ADUser -Filter 'UserPrincipalName -like "HTTP/*"'

A faster way to find if there are duplicates:

setspn -X

This command would remove SPN for a specific user kaspersky:

setspn -D HTTP/FQDN kaspersky

On Windows workstations you can also get the current list of Kerberos tickets with

klist

Sometimes there might be an incorrect old ticket, in that case you can purge ticket:

klist purge

You can also request a ticket manually with

klist get HTTP/kwts.example.com