Cisco FirePOWER: Upgrade from 6.2.0 to 6.2.2 fails

Started this simple upgrade for Firepower Management Center (FMC) from version 6.2.0 to 6.2.2 and ended up with TAC case and many troubleshooting hours.

It was a non-patched install of 6.2.0. Readiness Check passed but after 70% of the actual upgrade I got an error:

System call to mkdir(/var/sf/sifile_download) Failed: mkdir /var/sf/sifile_download: File exists

Patch before upgrade

Turned out it is a bug fixed in Patch 2 which of course I did not have installed (whats the point of Readiness Check then some may ask). Tried several workarounds but none helped. So we’ve figured let’s upgrade to Patch 4 which should have a fix for this and many other bugs and let us upgrade to 6.2.2.

First, we had to remove upgrade lock and 6.2.2 update files folder.

:/new-root/tmp# rm -rf upgrade.lock/

:/tmp# rm -rf Update-6.2.2-81

Started Patch 4 install from CLI (because GUI is unavailable). Apparently file location for update files changes from version to version. For me, it was under /new-root.

:/new-root/tmp# install_update.pl /new-root/var/sf/updates/Sourcefire_3D_Defense_Center_S3_Patch-6.2.0.4-85.sh

but during the Patch install, we hit a DB health-check failure. We ran DB health-check

/new-root/tmp# DBCheck.pl

and it came back with multiple FATAL errors in the database. Next step was to attempt to repair database with slow repair command. I do not know if it actually fixes anything but for me, it did not work (use it at your own risk)

/var/lib/mysql/sfsnort# for i in `ls -1|sed ‘s/\….$//’|sort|uniq`; do repair_table.pl -farms $i;done &

If this did not work next step is to work with developers on fixing DB errors or re-image. Keep in mind there is no “reset to factory default” or simple “re-image”. For re-image you will need console/LoM access, ISO file, FTP and some time to get it back online. Plus if your back up is corrupt (DB corrupt) it will not do you any good and everything needs to be recreated from scratch.

Slow repair did not make any difference.

Upgrade to 6.2.2 modified DB schema but failed due to bug and because of DB schema changed I could not install previous version patch to fix the bug.

We ended up working with TAC on fixing every FATAL error in DB health check. Final results should look like this.

/var/sf# DBCheck.pl
running database integrity check with the following options:
– use exception directory /usr/local/sf/etc/db_exceptions
– check refererences
– check enterprise objects
– check schema
– check required data
– log to stderr

getting filenames from [/usr/local/sf/etc/db_updates/index]
getting filenames from [/usr/local/sf/etc/db_updates/base-6.2.0]
/usr/local/sf/etc/db_exceptions/db_exceptions.yaml
After Checking DB, Warnings: 0, Fatal Errors: 0

After that we also had to roll back 6.2.2 upgrade due to some of the services being down. Keep in mind this does not revert back DB changes so you still need to fix all the fatal errors first. Rollback can be done for major upgrades only, not patches.

/var/log/sf/Sourcefire_3D_Defense_Center_S3_Upgrade-6.2.2#upgrade_rollback.sh

Tail roll back log for status:

/var/log/sf/Sourcefire_3D_Defense_Center_S3_Upgrade-6.2.2# tailf upgrade_rollback.log

Finally after rollback completed and FMC rebooted back into original version we ran Patch 4 upgrade. It completed successfully and we are ready for 6.2.2 upgrade. We will see how that goes.