The other week I ran into an instance where a group of customers were unable to access Cisco Unified Intelligence Center. Upon further investigation I was unable to get to admin pages of much of the collaboration suite and call control systems from these users computers either. The suite was on various versions of 9.0.X and 9.1.X due to restraints with many third party integration’s. This issue will occur on anything using the cipher suite mentioned below and is not limited to these versions or applications. This is ultimately where the problem stems from but I’ll take you down the path.
To start I’m going to list the fixes in case you don’t want to read my troubleshooting methodology. Then I will walk you through my discovery and detail the fixes. Remember, which fix is correct for your situation will vary based on use case, security policies, etc.
Fix 1: Uninstall Microsoft Updates causing issue
Fix 2: Re-issue certificate (if possible) using strong ciphers (may require upgrading applications)
Fix 3: Use a different web browser
Fix 4: Re-order ciphers via Group Policy
As this issue just recently popped up I was attempting to diagnose the scope of the issue so I started gathering data. Attempting from my workstation I wasn’t having any issue getting to the pages. However, I use a different browser than the house wide standard. When I attempted using the standard browser I too was unable to access the web interfaces. So if it works in third party browsers but not Internet Explorer the question quickly becomes why?
I began to check with the team that roles out Windows workstations and maintains their application and OS patching. They had indicated that patches for the previous month had been rolled out. Off to Programs and Features to find what may have been installed on my workstation. After digging through Microsoft’s site I found an update that had installed that indicated it had adjusted some cipher suites and their order. That seems like a logical option as I’ve ran into many times where one browser can access a site but another can’t and it ends up related to the browsers various security options. So, I open up the site in a browser that works and verify the certificate.
As we can see in the screenshot above the certificate blatantly indicates what cipher(s) it is using. So now it’s time to look at the ciphers before and after the Microsoft updates. I have obtained a list in order from a workstation with the update, and one without.
Prior to the update the cipher list was as follows:
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256 TLS_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_RC4_128_SHA TLS_RSA_WITH_3DES_EDE_CBC_SHA TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P256 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P384 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P521 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P521 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P256 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P384 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P521 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P256 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P384 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P521 TLS_DHE_DSS_WITH_AES_128_CBC_SHA TLS_DHE_DSS_WITH_AES_256_CBC_SHA TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA TLS_RSA_WITH_RC4_128_MD5 SSL_CK_RC4_128_WITH_MD5 SSL_CK_DES_192_EDE3_CBC_WITH_MD5 TLS_RSA_WITH_NULL_SHA TLS_RSA_WITH_NULL_MD5 TLS_RSA_WITH_AES_128_CBC_SHA256 TLS_RSA_WITH_AES_256_CBC_SHA256 TLS_RSA_WITH_AES_128_GCM_SHA256 TLS_RSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P384 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P521 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P521 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P256 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P384 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P521 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P384 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P521 TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 TLS_RSA_WITH_NULL_SHA256 TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P256 TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P384 TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P521 TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P384 TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P521
As you can see the cipher that the certificate was using is at the top of the order. In this situation there is no issue getting to the admin pages. However, after the update the order of ciphers changes complete.
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P384 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P256 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P384 TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 TLS_DHE_RSA_WITH_AES_256_CBC_SHA TLS_DHE_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_AES_256_GCM_SHA384 TLS_RSA_WITH_AES_128_GCM_SHA256 TLS_RSA_WITH_AES_256_CBC_SHA256 TLS_RSA_WITH_AES_128_CBC_SHA256 TLS_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_AES_128_CBC_SHA TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P384 TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P256 TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P384 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P384 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P256 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P384 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P256 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P384 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P256 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P384 TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 TLS_DHE_DSS_WITH_AES_256_CBC_SHA TLS_DHE_DSS_WITH_AES_128_CBC_SHA TLS_RSA_WITH_3DES_EDE_CBC_SHA TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA TLS_RSA_WITH_RC4_128_SHA TLS_RSA_WITH_RC4_128_MD5 TLS_RSA_WITH_NULL_SHA256 TLS_RSA_WITH_NULL_SHA SSL_CK_RC4_128_WITH_MD5 SSL_CK_DES_192_EDE3_CBC_WITH_MD5
As we can see above there are now a bunch of ciphers preferred above the valid one our certificate is using. This is what ultimately ends up causing Internet Explorer to be unable to open the admin webpages. So now comes the question of…how do we fix it? There are a few options.
First, we can uninstall the updates that caused the cipher suite update. In this case to the best of my research abilities I have discovered three updates that could be potentially installed causing the cipher list to change.
https://support.microsoft.com/en-us/kb/3172605 is the July update that supersedes the June update below.
https://support.microsoft.com/en-us/kb/3161608 is the June update that is said to fix issues in the update below.
https://support.microsoft.com/en-us/kb/3161639 is the update that first re-ordered the ciphers
Naturally the ciphers were updated for improved security of the OS and Internet Explorer web browser so uninstalling the update (especially since some were roll up of multiple updates) this is likely a no go. This leads us to the question of updating the certificate.
While that sounds great, unfortunately some of the older collaboration and call control applications (as in CUCM, Unity Connection, UCCX, etc) do not give you an option of selected a cipher suite when generating a certificate. You could update the application as whole but you may have constraints around you that limit that ability.
An obvious option is to just plain out use a different web browser. Many of us working in the I.T. field likely already have multiple browsers but this impacted end users utilizing reporting dashboards and more. In an organization where standards are kept for user workstations another browser may not be allowed.
The final option I discovered before call the issue resolved was to simply reorder the cipher suite manually. You can do this via group policy. Either locally or business wide. To do it locally you open up Group Policy editor (in run dialog enter gpedit.msc) and navigate to Computer Configuration > Administrative Templates > Network > SSL Configuration Settings. Under this section you can reorder the ciphers to your desired output. In putting the cipher in question at the top access was re-established with the pages in question.