Started this simple upgrade for Firepower Management Center (FMC) from version 6.2.0 to 6.2.2 and ended up with TAC case and many troubleshooting hours.
It was a non-patched install of 6.2.0. Readiness Check passed but after 70% of the actual upgrade I got an error:
System call to mkdir(/var/sf/sifile_download) Failed: mkdir /var/sf/sifile_download: File exists
Patch before upgrade
Turned out it is a bug fixed in Patch 2 which of course I did not have installed (whats the point of Readiness Check then some may ask). Tried several workarounds but none helped. So we’ve figured let’s upgrade to Patch 4 which should have a fix for this and many other bugs and let us upgrade to 6.2.2.
First, we had to remove upgrade lock and 6.2.2 update files folder.
:/new-root/tmp# rm -rf upgrade.lock/
:/tmp# rm -rf Update-6.2.2-81
Started Patch 4 install from CLI (because GUI is unavailable). Apparently file location for update files changes from version to version. For me, it was under /new-root.
:/new-root/tmp# install_update.pl /new-root/var/sf/updates/Sourcefire_3D_Defense_Center_S3_Patch-220.127.116.11-85.sh
but during the Patch install, we hit a DB health-check failure. We ran DB health-check
and it came back with multiple FATAL errors in the database. Next step was to attempt to repair database with slow repair command. I do not know if it actually fixes anything but for me, it did not work (use it at your own risk)
/var/lib/mysql/sfsnort# for i in `ls -1|sed ‘s/\….$//’|sort|uniq`; do repair_table.pl -farms $i;done &
If this did not work next step is to work with developers on fixing DB errors or re-image. Keep in mind there is no “reset to factory default” or simple “re-image”. For re-image you will need console/LoM access, ISO file, FTP and some time to get it back online. Plus if your back up is corrupt (DB corrupt) it will not do you any good and everything needs to be recreated from scratch.
Slow repair did not make any difference.
Upgrade to 6.2.2 modified DB schema but failed due to bug and because of DB schema changed I could not install previous version patch to fix the bug.
We ended up working with TAC on fixing every FATAL error in DB health check. Final results should look like this.
running database integrity check with the following options:
– use exception directory /usr/local/sf/etc/db_exceptions
– check refererences
– check enterprise objects
– check schema
– check required data
– log to stderr
getting filenames from [/usr/local/sf/etc/db_updates/index]
getting filenames from [/usr/local/sf/etc/db_updates/base-6.2.0]
After Checking DB, Warnings: 0, Fatal Errors: 0
After that we also had to roll back 6.2.2 upgrade due to some of the services being down. Keep in mind this does not revert back DB changes so you still need to fix all the fatal errors first. Rollback can be done for major upgrades only, not patches.
Tail roll back log for status:
/var/log/sf/Sourcefire_3D_Defense_Center_S3_Upgrade-6.2.2# tailf upgrade_rollback.log
Finally after rollback completed and FMC rebooted back into original version we ran Patch 4 upgrade. It completed successfully and we are ready for 6.2.2 upgrade. We will see how that goes.
3 comments On Cisco FirePOWER: Upgrade from 6.2.0 to 6.2.2 fails
That is not good. Thanks for the heads up. The processes of patching/upgrading for this product seem to be overly complicated. Read the release notes twice and prey!. And it also seems to take an awful long time to apply some of these patches/upgrades.
Yes, this was a fun one. For any upgrade I always now pre-download file to sensor and it makes updates much faster.
Hi good day.
I had the same problem that you described in the publication, luckily, the DB did not have problems and I could do the roolback of my update.
So now I have version 18.104.22.1687.
One question, could you update your FMC? Which jumps did you have to do?