In preparation for Identity Services Engine (ISE) 2.1 upgrade, I’ve replicated production environment in order to validate upgrade functionality. This way I had a chance to run through restore steps and test upgrade process. This post is about my findings and lessons learned.
Cisco ISE: 2.0
First of all, before restoring from backup verify and match ISE OS and patch versions.
Make sure new nodes have the same Web UI admin login. I had a different password and after restoring from backup GUI login did not work. Changing password on CLI with application reset-passwd ise admin did not help so I had to reset config. Once passwords matched I was able to login to GUI after the restore.
Keep hostnames the same otherwise you can not re-import Internal CA store and you will get a validation error. In a testing environment, where hostnames will be different, you will need to regenerate Root CA on ISE as discussed further.
After the upgrade when importing certificate for portals you may get this cosmetic error so just ignore it and certificate should install anyway.
If you are building a test/lab system do not forget to rename sponsor and mydevices shortcuts after the restore as it will continue to reference production portals.
One thing I noticed is if you are using external CA for certificate management there is no need to import Internal CA store. Clients with certificates provisioned through MS SCEP did connect successfully after the restore (assuming you are using wildcard System Certificates issued by MS CA and they were re-imported successfully). However, if you are using ISE internal CA for BYOD certificates provisioning you will need to import Internal CA store, otherwise you will get this error during authentication and BYOD flow may fail.
To fix this issue re-import Internal CA store or regenerate Root CA on ISE under Administration > System > Certificates > Certificate Signing Requests > ISE Root CA.
Overall, restore process was very intuitive and as long as you follow Cisco guidance recovery from failure will be a success.
Now, once you have a working replica of production system we get on with the upgrade.
Step 1. Upgrade the Virtual Machine (VM) OS to RHEL7. VM version needs to be VM10 otherwise, RHEL7 will not be available from a drop down menu.
To upgrade your VM you will need to turn it off and upgrade under Compatibility menu option. Upgrade takes a few seconds. Once done, it is good practice to start node back up for changes to propagate.
Step 2. Now when VM is up to proper version under Edit VM settings > VM Options change Guest OS to RHEL 7 (64-bit). You will need to shutdown ISE node again. Plan your time accordingly and verify this prior to the upgrade.
Step 3. Copy upgrade file to ISE nodes. In the distributed deployment, considering the time it takes to upgrade each box I went with GUI upgrade to automate the process, however, I ran into an issue of failing downloads over FTP. It was very annoying as it takes some time for the process to fail.
Pre-populating image through CLI does no good as GUI upgrade process did not detect it. What I ended up doing is upgrading from DISK repository. I did CLI copy of upgrade file to disk with this command.
copy ftp://username:pass@<IP>/ise-upgrade-file.tar.gz disk:/
Directory of disk:/
4096 Dec 15 2016 11:07:04 corefiles/
85732447 Dec 15 2016 15:45:25 ise-patchbundle-22.214.171.1246-PP1-161394.SPA.x86_64.tar.gz
6581931913 Dec 15 2016 15:55:26 ise-upgradebundle-2.0.x-to-126.96.36.1994.SPA.x86_64.tar.gz
And then I’ve created a repository for it which I referenced during the upgrade.
Start upgrade from DISK.
Even with the upgrade restarted from disk, it failed once again on one node (but after retrying 1 more time it completed file download successfully) and another node I had to actually reboot because I lost CLI access after multiple download failures.
As soon as secondary Admin node downloads and expands file bundle locally you can click Continue to proceed with the upgrade. You can wait for all nodes to be Ready for the upgrade or proceed with partial node upgrade. With a partial upgrade you will end up with split deployment where everything still functions but you can not make any changes.
While the Primary node is up you can monitor upgrade status of the nodes on Upgrade > Overview page.
Once you start upgrading primary PAN (should be the last one to upgrade) you will be forced to switch to new PAN and there will be no upgrade progress available in the GUI other than Node Status.
To check upgrade progress switch to CLI on old PAN and use this command.
show logging system ade/ADE.log tail
Checking for the line similar to this one will indicate upgrade completed.
interface=CLI, detail=Application upgrade completed successfully at time:
When all the nodes reconnect to PAN do not forget to follow Post-Upgrade Tasks and apply the latest patch.
With multi-node ISE deployment it is very important to test upgrade functionality in the lab environment prior to scheduled upgrade.
Do restore on a test node and verify different functionalities: Certificates, BYOD, Guest, Portals, etc.
Plan accordingly and budget enough time as there are several things that can delay and prolong upgrade process.
P.S. A few notes from post production upgrade:
- Got several health alerts from standby PAN2 regarding unreachable nodes. All was clear on primary PAN1 and next day alerts stopped. Seemed to be cosmetic
- Could not BYOD i-Devices after the upgrade. Turned out to be pre-existing profile under iOS General settings and
the beta version of OSiOS bug in version 10.3.1. Once profile was removed BYOD worked as expected. No fix for iOS bug yet. Check for latest update in this post.