Problem uploading Thawte issued certificate on the Cisco WLC….Certificate not properly chained.

Recently I came upon couple of scenarios where the Cisco WLC would not accept a web-auth server cert issued by Thawte (Known CA). This is because the later version of the Cisco WLC (I believe 7.6 and above) need to have a chained certificate before you can upload it on the WLC.If you do further debugging on the WLC you will see the following error logs, which clearly points to the problem with the issuer certificate:

*TransferTask: Feb 12 12:26:05.987: Adding cert (7728 bytes) with certificate key password.
*TransferTask: Feb 12 12:26:06.015: sshpmCheckWebauthCert: Verification return code: 0
*TransferTask: Feb 12 12:26:06.015: Verification result text: unable to get issuer certificate
*TransferTask: Feb 12 12:26:06.015: Error at 2 depth: unable to get issuer certificate
*TransferTask: Feb 12 12:26:06.027: sshpmAddWebauthCert: Error decoding certificate, Deleting it.
*TransferTask: Feb 12 12:26:06.027: RESULT_STRING: Error installing certificate.
*TransferTask: Feb 12 12:26:06.027: RESULT_CODE:12
*TransferTask: Feb 12 12:26:06.027: Memory overcommit policy restored from 1 to 0
*emWeb: Feb 12 12:26:07.041: sshpmGetIdCertIndex: called to lookup cert >bsnSslWebauthCert<
*emWeb: Feb 12 12:26:07.041: sshpmGetIdCertIndex: found match in row 4
*emWeb: Feb 12 12:26:07.041: sshpmGetCID: called to evaluate <bsnSslWebauthCert>
*emWeb: Feb 12 12:26:07.041: sshpmGetCID: comparing to row 0, CA cert >bsnOldDefaultCaCert<

When you open the cert it does not appear to have any problem and the cert will look perfect. You OS will also not recognize it as invalid, this is because your laptop already has the Root and the Intermediate Certificate installed and even if the cert is not correctly chained it marks it as valid unlike the WLC.

 

Following is the mmc snapshot of the known Trusted CA on my laptop.

Now lets look at where the problem is:
==================================
One you open the certificate in a notepad you will see the following format:

Server Cert >>> Intermediate Cert >>> Root Cert (Generally the Root Cert should validate itself i.e the Root Cert is Root CA issuing itself a cert like below, where the issuer and the issued to is the same.)

The certificate looks something like this: (For security I have not shown the entire certificate).

Bag Attributes
localKeyID: 3B DB 85 15 63 AF CA B7 57 27 4E A3 E5 0B 84 32 1D AC 06 18
subject=/C=XX/ST=XX/L=Sydney/O=XX/OU=XX/CN=XY.com.au
issuer=/C=US/O=thawte, Inc./CN=thawte SSL CA – G2

—–BEGIN CERTIFICATE—–
MIIE/TCCA+WgAwIBAgIQF//T50TPBQL4+/7Iqh7dsTANBgkqhkiG9w0BAQsFADBB
—————-Snipping————————————
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMdGhhd3RlLCBJbmMuMRswGQYDVQQDExJ0
—–END CERTIFICATE—–

Bag Attributes: <No Attributes>
subject=/C=US/O=thawte, Inc./CN=thawte SSL CA – G2
issuer=/C=US/O=thawte, Inc./OU=Certification Services Division/OU=(c) 2006 thawte, Inc. – For authorized use only/CN=thawte Primary Root CA

—–BEGIN CERTIFICATE—–
MIIEsjCCA5qgAwIBAgIQFofWiG3iMAaFIz2/Eb9llzANBgkqhkiG9w0BAQsFADCB
———————Snipping————————————-
sjFuz4DliAc2UXu6Ya9tjSNbNKOVvKIxf/L157fo78S1JzLp955pxyvovrsMqufq
YBLqJop4
—–END CERTIFICATE—–

Bag Attributes: <No Attributes>
subject=/C=US/O=thawte, Inc./OU=Certification Services Division/OU=(c) 2006 thawte, Inc. – For authorized use only/CN=thawte Primary Root CA
issuer=/C=ZA/ST=Western Cape/L=Cape Town/O=Thawte Consulting cc/OU=Certification Services Division/CN=Thawte Premium Server CA/emailAddress=premium-server@thawte.com

—–BEGIN CERTIFICATE—–
MIIERTCCA66gAwIBAgIQM2VQCHmtc+IwueAdDX+skTANBgkqhkiG9w0BAQUFADCB
————————Snipping————————————
95OBBaqStB+3msAHF/XLxrRMDtdW3HEgdDjWdMbWj2uvi42gbCkLYeA=
—–END CERTIFICATE—–

So if we go through the certificate we see the certificate being issued to XY.com.au by thawte SSL CA – G2 (Intermediate CA).

Down the chain we see the Intermediate CA cert, cert issue to thawte SSL CA – G2 by thawte Primary Root CA (Can be a Root or another Intermediate CA).

Further down the chain we see thawte Primary Root CA being issued a cert by Thawte Premium Server CA and there is no other cert following this.

So the problem here is either thawte Primary Root CA can be a Root CA or an Intermediate CA. If it is a Root CA, the last cert is chain should had been for thawte Primary Root CA issued by thawte Primary Root CA itself.

If it is an intermediate CA, there should have been another cert down the chain, issued to Thawte Premium Server CA by itself, it being the Root.

In this scenario the WLC is looking for the Root Cert which is not there is the chain and thus marks the certificate as invalid.

How to fix this:
==============

Thawte do provide the Root CA and Intermediate CA cert on its website from where you can download the missing cert easily.

https://www.thawte.com/roots/index.html

So the fix would be either make thawte Primary Root CA as the Root CA and download the cert file for the Thawte website and replace the last cert in the chain, so that we have the cert for thawte Primary Root CA issued by thawte Primary Root CA.


Or, keep the same chain and download the Root CA cert for Thawte Premium Server CA and add it at the end of the chain so that the certificate chain is complete.

Once the chain is complete please follow the Cisco document to compile the cert along with the private key and get the final cert.

Please refer to the previous posts on certs:

http://rameshkumarroy.com/creating-chained-certificate-fro/

Hope this was helpful.

 

DHCP Fingerprinting

DHCP Fingerprinting is a method of detecting the end device OS based on the dhcp exchange packets. In today’s network where we are talking about IoE , BYOD it is required to identify the devices in your network and mark them accordingly.

Why do we need Fingerprinting:
========================

With BYOD personal devices are making their way into the workplace, and it is a tough job for the network administrators to dynamically detect these devices and make sure these devices are compliant and to enforce required polices on these devices. Detecting the devices type/OS is also part of the play.

Due to the proliferation of BYOD (Bring Your Own Devices)/mobile devices connecting mostly over the Wireless Network, it becomes difficult to identify and control the types of devices that can connect to the network, and once connected, to determine what access privileges they might have.

With DHCP Fingerprinting, DHCP Servers or devices like IPAM Controllers or Wireless Controllers, can use DHCP Fingerprinting to identify the device type, manufacturer name and OS of the clients/devices connecting to the network, categorize them into ACLs, and control which device can connect to the network and what it can do.

How it works:
===========

DHCP Fingerprinting is one of the methods that help us in identifying the OS on the devices bases on the dhcp option.

The complete DHCP process is like this:

 

The DHCP packets contain multiple options. One of the most important option which is used for dhcp fingerprinting is the option : 55 called Parameter request list, this option is present in the packets sent from the client end i.e the Discover and Request Packets.

 

The option 55: Parameter Request list in the above capture is :

1,6,15,44,3,33,150 and 43

A DHCP discover request asks for DHCP options in a specific sequence. This makes DHCP Fingerprinting possible – identifying a device or OS requesting an IP address based on the requested DHCP options.

Fingerbank has got a repository of such fingerprints:

https://fingerbank.inverse.ca/

Some of the captured fingerprints in hex:

Android_device    3C64686370636420342E302E3135
Android 2.X           3c6468637063642034
Android 2.2           3701792103061c333a3b
Android 2.3.X        0c616E64726F69645F
Android 4.0.X        37012103060f1c333a3b
Android 4.0.X(2)    37012103061c333a3b
Blackberry 2          3C426C61636B4265727279
Blackberry(2)         370103060F775ffc2c2e2f
iOS Device             370103060F77FC
iPad                        37011c02030f06770c2c2f1a792a
OS X 10.6               370103060f775ffc2c2e2f
OS X 10.7               370103060f775ffc2c2e
Win Mobile            3c4d6963726f736f66742057696e646f77732043450
Win Mobile6          370103060f2c2e2f

Aruba implementation of DHCP Fingerprinting:

http://community.arubanetworks.com/t5/Unified-Wired-Wireless-Access/COTD-DHCP-Fingerprinting-how-to-ArubaOS-6-0-1-0-and-above/td-p/11164

http://community.arubanetworks.com/t5/Controller-less-WLANs/DHCP-FINGERPRINTING-WITH-Aruba-Instant/ta-p/183272

Hope this was informative.

 

Understanding Browser’s user-agent

So basically the user-agent string is something which identifies your browser and provides certain system details to servers hosting the webpage you are visiting. When you visit a webpage, the browser sends the user-agent string to the server hosting the page that you are visiting. This string indicates which browser is being, its version number, and details about your system, such as operating system and version. The web server can use this information to provide content that is tailored for your specific browser.You can see the user-agent in the wireshark captures when you machine sends out the GET request or on the browser itself.

You can also check the user-agent on the browser itself. Lets see how:

Chrome:
=========

Type chrome://version in the address bar.

FireFox:
==========

Type about: in the address bar.

Internet Explorer:
=============

—————————
Message from webpage
—————————
Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.3; .NET4.0C; .NET4.0E; rv:11.0) like Gecko
—————————
OK
—————————

Type javascript:alert(navigator.userAgent) in the address bar, the user-agent string would show up in a dialog box. You can do CTRL+C to copy it.

While we might be considered user agent sniffing a horrible practice on the client side, however sniffing user agent is done quite a bit on the server side to serve up the appropriate page version of a site, or redirect to, for example, the mobile version of the site.  This can be a dangerous road but most large site with a separate mobile interface do it.

 

The following is the user agent for Firefox on a mobile device:
 Mozilla/5.0 (Mobile; rv:18.0) Gecko/18.0 Firefox/18.0

 

History of User Agent:
==================
I came across this blog which talks about the history of User-Agent.

 

 

In the beginning there was NCSA Mosaic, and Mosaic called itself NCSA_Mosaic/2.0 (Windows 3.1), and Mosaic displayed pictures along with text, and there was much rejoicing. And behold, then came a new web browser known as “Mozilla”, being short for “Mosaic Killer,” but Mosaic was not amused, so the public name was changed to Netscape, and Netscape called itself Mozilla/1.0 (Win3.1), and there was more rejoicing. And Netscape supported frames, and frames became popular among the people, but Mosaic did not support frames, and so came “user agent sniffing” and to “Mozilla” webmasters sent frames, but to other browsers they sent not frames.

 

And Netscape said, let us make fun of Microsoft and refer to Windows as “poorly debugged device drivers,” and Microsoft was angry. And so Microsoft made their own web browser, which they called Internet Explorer, hoping for it to be a “Netscape Killer”. And Internet Explorer supported frames, and yet was not Mozilla, and so was not given frames. And Microsoft grew impatient, and did not wish to wait for webmasters to learn of IE and begin to send it frames, and so Internet Explorer declared that it was “Mozilla compatible” and began to impersonate Netscape, and called itself Mozilla/1.22 (compatible; MSIE 2.0; Windows 95), and Internet Explorer received frames, and all of Microsoft was happy, but webmasters were confused.And Microsoft sold IE with Windows, and made it better than Netscape, and the first browser war raged upon the face of the land. And behold, Netscape was killed, and there was much rejoicing at Microsoft. But Netscape was reborn as Mozilla, and Mozilla built Gecko, and called itself Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.1) Gecko/20020826, and Gecko was the rendering engine, and Gecko was good. And Mozilla became Firefox, and called itself Mozilla/5.0 (Windows; U; Windows NT 5.1; sv-SE; rv:1.7.5) Gecko/20041108 Firefox/1.0, and Firefox was very good. And Gecko began to multiply, and other browsers were born that used its code, and they called themselves Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7.2) Gecko/20040825 Camino/0.8.1 the one, and Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.8.1.8) Gecko/20071008 SeaMonkey/1.0 another, each pretending to be Mozilla, and all of them powered by Gecko.

 

And Gecko was good, and IE was not, and sniffing was reborn, and Gecko was given good web code, and other browsers were not. And the followers of Linux were much sorrowed, because they had built Konqueror, whose engine was KHTML, which they thought was as good as Gecko, but it was not Gecko, and so was not given the good pages, and so Konquerer began to pretend to be “like Gecko” to get the good pages, and called itself Mozilla/5.0 (compatible; Konqueror/3.2; FreeBSD) (KHTML, like Gecko) and there was much confusion. Then cometh Opera and said, “surely we should allow our users to decide which browser we should impersonate,” and so Opera created a menu item, and Opera called itself Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; en) Opera 9.51, or Mozilla/5.0 (Windows NT 6.0; U; en; rv:1.8.1) Gecko/20061208 Firefox/2.0.0 Opera 9.51, or Opera/9.51 (Windows NT 5.1; U; en) depending on which option the user selected.

 

And Apple built Safari, and used KHTML, but added many features, and forked the project, and called it WebKit, but wanted pages written for KHTML, and so Safari called itself Mozilla/5.0 (Macintosh; U; PPC Mac OS X; de-de) AppleWebKit/85.7 (KHTML, like Gecko) Safari/85.5, and it got worse.

 

And Microsoft feared Firefox greatly, and Internet Explorer returned, and called itself Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0) and it rendered good code, but only if webmasters commanded it to do so. And then Google built Chrome, and Chrome used Webkit, and it was like Safari, and wanted pages built for Safari, and so pretended to be Safari. And thus Chrome used WebKit, and pretended to be Safari, and WebKit pretended to be KHTML, and KHTML pretended to be Gecko, and all browsers pretended to be Mozilla, and Chrome called itself Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.2.149.27 Safari/525.13, and the user agent string was a complete mess, and near useless, and everyone pretended to be everyone else, and confusion abounded.

 

 

Hope this was informative.

CIMC Secure Page not opening in Firfox while works for Chrome and IE.

While working on my UCS box I came across a problem, where the GUI/CICM page would not load on my Firefox browser while it worked fine on Chrome and IE.Doing some search I found that this has been already reported by Cisco under the Bug # CSCun04933.

Symptom:
============
Following error is observed while trying to access CIMC web page

SSL received a malformed Server Key Exchange handshake message. (Error code: ssl_error_rx_malformed_server_key_exch)

Conditions:
=========
When accessing CIMC web page with Firefox web browser version 27.0 and above.

Workaround:
===========
Change the max TLS version in Firefox.

1) Go to about:config
2) Search for ‘tls’
3) Change ‘3’ to ‘2’
4) Restart firefox

The workaround has worked for my issue seen on my Firefox version 35.0.1.

 

Double click on the Preference Name to change the value.

Hope this was helpful.

IPERF to measure throughput

Iperf is a handy tool to measure the bandwidth and the quality of a network link. It is a commonly used network testing tool that can create Transmission Control Protocol (TCP) and User Datagram Protocol (UDP) data streams and measure the throughput of a network that is carrying them.Iperf allows the users to vary various parameters that can be used for testing the network, or alternatively for optimizing and tuning a network. Iperf has a client and server functionality, and can measure the throughput between the two ends, either unidirectionally or bi-directionally.

Iperf can be installed very easily on any Linux or Microsoft Windows system, where one host can be configured as a client, the other one as server.

Setup required for running the iperf test:

1. Download the iperf setup, you can download it from: https://iperf.fr/
2. Copy the setup file on the two hosts you would be using to perform the test.
3. Set one host in the server mode and the other in the client mode with the following syntax:

To set the host in server mode use the command : iperf -s

C:\IOS\Imagesiperf-2.0.5-2-win32>iperf -s
————————————————————
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
————————————————————

To set the client in client mode use the command : iperf -c <server ip address>

C:\IOS\Imagesiperf-2.0.5-2-win32>iperf -c 192.168.1.5      // Where 192.168.1.5 is server ip address.

The other parameters available in iperf are:

C:\IOS\Imagesiperf-2.0.5-2-win32>iperf –help

Usage: iperf [-s|-c host] [options]
       iperf [-h|–help] [-v|–version]
Client/Server:
  -f, –format    [kmKM]   format to report: Kbits, Mbits, KBytes, MBytes
  -i, –interval  #        seconds between periodic bandwidth reports
  -l, –len       #[KM]    length of buffer to read or write (default 8 KB)
  -m, –print_mss          print TCP maximum segment size (MTU – TCP/IP header)
  -o, –output    <filename> output the report or error message to this specified file
  -p, –port      #        server port to listen on/connect to
  -u, –udp                use UDP rather than TCP
  -w, –window    #[KM]    TCP window size (socket buffer size)
  -B, –bind      <host>   bind to <host>, an interface or multicast address
  -C, –compatibility      for use with older versions does not sent extra msgs
  -M, –mss       #        set TCP maximum segment size (MTU – 40 bytes)
  -N, –nodelay            set TCP no delay, disabling Nagle’s Algorithm
  -V, –IPv6Version        Set the domain to IPv6
 
Server specific:
  -s, –server             run in server mode
  -U, –single_udp         run in single threaded UDP mode
  -D, –daemon             run the server as a daemon

 

Client specific:
============

 

-b, –bandwidth #[KM]    for UDP, bandwidth to send at in bits/sec
                           (default 1 Mbit/sec, implies -u)
  -c, –client    <host>   run in client mode, connecting to <host>
  -d, –dualtest           Do a bidirectional test simultaneously
  -n, –num       #[KM]    number of bytes to transmit (instead of -t)
  -r, –tradeoff           Do a bidirectional test individually
  -t, –time      #        time in seconds to transmit for (default 10 secs)
  -F, –fileinput <name>   input the data to be transmitted from a file
  -I, –stdin              input the data to be transmitted from stdin
  -L, –listenport #       port to receive bidirectional tests back on
  -P, –parallel  #        number of parallel client threads to run
  -T, –ttl       #        time-to-live, for multicast (default 1)
  -Z, –linux-congestion <algo>  set TCP congestion control algorithm (Linux only)
 
Miscellaneous:
  -x, –reportexclude [CDMSV]   exclude C(connection) D(data) M(multicast) S(settings) V(server) reports
  -y, –reportstyle C      report as a Comma-Separated Values
  -h, –help               print this message and quit
  -v, –version            print version information and quit
 
[KM] Indicates options that support a K or M suffix for kilo- or mega-

 

The TCP window size option can be set by the environment variable TCP_WINDOW_SIZE. Most other options can be set by an environment variable.
IPERF_<long option name>, such as IPERF_BANDWIDTH.
Report bugs to <iperf-users@lists.sourceforge.net>
C:\IOS\Imagesiperf-2.0.5-2-win32>

 

Server side:
=========

 

#iperf -s
———————————————————— 
Server listening on TCP port 5001 
TCP window size: 8.00 KByte (default) 
———————————————————— 
[852] local 10.1.1.1 port 5001 connected with 10.6.2.5 port 33453 
[ ID]   Interval          Transfer       Bandwidth 
[852]   0.0-10.6 sec   1.26 MBytes   1.03 Mbits/sec 

 

Client side:
=========
#iperf -c 10.1.1.1
———————————————————— 
Client connecting to 10.1.1.1, TCP port 5001 
TCP window size: 16384 Byte (default) 
———————————————————— 
[ 3] local 10.6.2.5 port 33453 connected with 10.1.1.1 port 5001 
[ 3]   0.0-10.2 sec   1.26 MBytes   1.05 Mbits/sec 

 

Another example:
Use the syntax with some additional parameters ” iperf.exe – c  <IP address of the server>   -P 10  -w 1000k ” (  -P refers to the number of parallel TCP streams and –w referes to the TCP window size  )

 

Hope this was helpful.

Using filters on Cisco WLC

The WLC outputs makes me crazy when you have to search for a specific entry in the logs. Recently I came across the filter option available on the Cisco WLCs. Not sure which code version it has been supported from, but it is awesome..

You can now use the ‘grep’ command to get the specific match. This is especially useful when the output of any commands is lengthy and you have to scroll down to get to the information that you are looking for. Lets take some examples and the related syntax.

(WLC-Primary) >grep ?

include        Include lines that match.
exclude        Exclude lines that match.

(WLC-Primary) >grep include ?

<pattern>      Pattern to be searched.

 

(WLC-Primary) >grep include uptime ?
<command>      Enter complete show command in double quotes.

 

Lets try to find the uptime of the WLC:

 

(WLC-Primary) >grep include ime “show sysinfo”
Press yes to continue(y)y
System Up Time…………………………….. 0 days 5 hrs 47 mins 57 secs
System Timezone Location……………………. (GMT +5:30) Colombo, New Delhi, Chennai, Kolkata
System Stats Realtime Interval………………. 5

 

There are 3 lines matching the pattern ime

 

Since this is case sensitive I searched for the value “ime”.

 

Lets see another example, let suppose I want to see all APs except a specific AP.

 

(WLC-Primary) >show ap summary
Number of APs……………………………… 3
Global AP User Name………………………… gce-apac
Global AP Dot1x User Name…………………… Not Configured
AP Name             Slots  AP Model              Ethernet MAC       Location          Country  IP Address       Clients
——————  —–  ——————–  —————–  —————-  ——-  —————  ——-
L3500-3              2     AIR-CAP3502I-E-K9     40:55:39:ca:8a:99  default location  AE       10.105.132.249   0
TEST-AP-1            2     AIR-CAP2602E-A-K9     6c:41:6a:78:d8:32  default location  US       10.105.132.247   0
AP-3602AP-1          2     AIR-CAP3602I-A-K9     e4:d3:f1:c9:04:ca  default location  US       10.105.132.251   0

 

We will use the option exclude this time.

 

(WLC-Primary) >grep exclude “L3500-3” “show ap summary”
Press yes to continue(y)y
Number of APs……………………………… 3
Global AP User Name………………………… gce-apac
Global AP Dot1x User Name…………………… Not Configured
AP Name             Slots  AP Model              Ethernet MAC       Location          Country  IP Address       Clients
——————  —–  ——————–  —————–  —————-  ——-  —————  ——-
TEST-AP-1            2     AIR-CAP2602E-A-K9     6c:41:6a:78:d8:32  default location  US       10.105.132.247   0
AP-3602AP-1          2     AIR-CAP3602I-A-K9     e4:d3:f1:c9:04:ca  default location  US       10.105.132.251   0

 

There are 12 lines not matching the pattern L3500-3

 

Hope this was informational.