My previous experience with ISE upgrade was somewhat painful but I did not have to dive into logs to move past a certain point. This time I had to dig into troubleshooting a bit deeper.
Right from the beginning, I got hit with expired self-signed certificate error so I went into System Certificates and renewed them. Another step I did was Operational Data Purge under System > Maintenance.
Next, I ran Upgrade Readiness (URT) bundle (available under 2.3 on CCO). I strongly recommend it because it is much quicker to run and in case it fails you do not need to rejoin Secondary PAN (because it gets dis-joined during actual upgrade). It is also much easier to generate and retrieve troubleshooting log file. URT bundle instruction can be found here.
If application was used before on this node then you need to remove it first with this command:
application remove urt
Copy bundle to ISE disk repository and run it with the following command:
application install ise-urtbundle-2.3.x-x.x <reponame>
For me, bundle failed on step 43/60. Obviously, no other info was provided so go figure.
– Data upgrade step 41/60, UPSUpgradeHandler(2.2.0.403)… Done in 0 seconds.
– Data upgrade step 42/60, DictionaryUpgradeRegistration(2.2.0.410)… Done in 0 seconds.
– Data upgrade step 43/60, UPSUpgradeHandler(2.3.0.100)… Failed.
– Failed
Final cleanup before exiting…
It was time for troubleshooting log review. Thankfully it automatically generates log bundle urt_logs.tar.gpg and encrypts it with the password.
Collecting log files …
– Encrypting logs bundle…
Please enter encryption password:
Please enter encryption password again to verify:
Encrypted URT logs(urt_logs.tar.gpg) are available in localdisk. Please reach out to Cisco TAC to debug
% Post-install step failed. Please check the logs for more details.
The file is stored on local disk so you just need to copy it over for review. I used FTP.
copy disk:/urt_logs.tar.gpg ftp://<FTP IP>/
I’ve used Kleopatra program to extract log files on windows machine.
Once extracted look for \urt_logs\opt\CSCOcpm\logs\urt path for file dbupgrade-data-global-xxx-xxx.log.
If you scroll all the way down this file there should be a reference to what element failed. For me, it was this:
Policy upgrade failed. Found the following errors:
PS Name: null ,Failure: During upgrade of PS: VPN , Got error: Failed to create authentication rule VPN of new Policy Set VPN – VPN
You can scroll up through the file and hopefully find more details on why it failed.
@@@ PsUpgrade: error- :Failed to create authentication rule VPN of new Policy Set VPN – VPN com.cisco.cpm.policy.pal.PalException: Result is not valid
This kind of gave me an idea that authentication rule called VPN was not converting properly during upgrade process.
This is what the rule looked like.
First I’ve tried to rename it which did not work. Then after multiple name changes and modifications I finally just simplified it by removing the shaded section and changed Default Rule to use VPN sequence for authentication instead of Internal Endpoints. After the change URT bundle completed successfully.
I’ve copied 2.3 upgrade files to all the ISE nodes local disk repository and started automatic upgrade which completed successfully. Applied SP1 and tested all connectivity without any issues.
When I thought I was done I went back to VPN policy to change it back to original setup and hit a nasty bug. I’ve added missing authentication rules and changed the Authentication Policy Default to “Deny Access” and lost the ability to make changes to it.
Entire policy was blanked out.
This was a bit unexpected and upsetting. Good thing I was doing it in the test environment (must have when you run ISE).