An Image Problem
They say write about what you know. Even if what you know about is reimaging Cisco firewalls? They probably didn’t mean write about that but never mind we’ll press on and not hold our breaths for the movie to come out.
For reasons I won’t go into I recently had to re-image a Cisco FMC 1600 and two 3100 series FTDs. My main takeaway from this is don’t. Don’t do it… unless you absolutely must and have a few days you don’t have plans for. In this instance, it was to ‘reset’ the FMC and its associated FTD firewalls to a known good status after various issues.
I offer up this small slice of experience to try and help anyone who has to do the same thing and also as an aide memoire in case, God forbid, I ever have to do it again myself.
This is what I used:
- Console cable ( I used this one: https://www.amazon.co.uk/dp/B01AFNBC3K?psc=1&ref=ppx_yo2ov_dt_b_product_details).
- USB to RJ45 Ethernet adaptor (if you’re copying the software from a laptop with no RJ45 port)
- Ethernet cable
- TFTP Server software (https://www.solarwinds.com/free-tools/free-tftp-server)
- FTP Server software (https://filezilla-project.org/)
- SFTP Server software (https://www.solarwinds.com/free-tools/free-sftp-server)
- VGA monitor, keyboard.
- FMC Software (in this instance):
- Cisco_Secure_FW_Mgmt_Center-7.2.5-208-Restore.ISO
- Cisco_Secure_FW_Mgmt_Center_Patch-7.2.5.1-29.sh.REL Patch
- FTD Software (in this instance):
- cisco-ftd-fp3k.7.2.5-208.SPA
- Cisco_FTD_SSP_FP3K_Patch-7.2.5.1-29.sh.REL.tar
- Laptop (preferably with admin rights – I shut down my Windows firewall when I was using the FTP/TFTP and SCP server software – I was cabled directly into firewalls so no risks were involved. If you do this, do remember to turn it back on again!)
- All certificates exported from the FMC into PKCS12 format (The restore process unhelpfully ‘fails’ the certs)
- Access to the relevant SmartLicense account – or a colleague with this.
Re-imaging and Restoring the FMC
I used the following process:
The process details being able to do the re-image from the CLI console or from a monitor and keyboard plugged into the FMC. I tried from the CLI but it didn’t work. I didn’t get the option to go into the restore boot menu. At some point I saw unintelligible blocks as if a text menu was failing to be rendered properly. However, the process worked as described using the monitor and keyboard.
Having re-imaged the FMC and run through its setup process, I was able to browse on to the GUI, upload the patch to upgrade the FMC to the same version as the backup, then restore the backup. This bit was relatively simple.
FMC / FTD Musings.
Before we progress to re-imaging and restoring the FTDs a few thoughts…
I find the distinction between what’s contained in the FMC backup and what’s contained in the FTD one a little hazy. I know the FMC has the lion’s share of the configuration: access polices, objects etc and I guess the FTD backup has everything under the FMC’s ‘Device’ menu: interface settings, routes, NAT and VPN. (Having said that, the NAT policy is applied from the FMC and survived me flattening an FTD in the lab so this isn’t 100% true.)
I apologise for the next bit, it gives me a mild headache reading it back.
As I recall, Cisco TAC told me that if I couldn’t add the newly re-imaged FTDs to the newly restored FMC with the same ip addresses as the existing FTDs defined on it and that I would have to delete these from the FMC first. But I had to connect the reimaged FTDs to the FMC to be able to upgrade them to the same version OS (7.2.5.1) as the FTD backup that I would ultimately restore. Had I not had to do this I could have perhaps got away restoring the backup offline and the re-adding it.
In any event, I didn’t want to delete the FTDs from the FMC console (where they now appeared after I had restored the backup) as I was worried I would lose some element of the configuration. So I added the new FTDs with different ip addresses, upgraded them and then restored the backups. But we’re getting ahead of ourselves, we need to reimage them first which this link describes how to do:
This procedure seems to end a step too soon. When it had finished I connected to the ftd shell (?) with the command ‘connect ftd’ and was taken to the setup process for the device.So, anyway, I did the above and restored the backup from the CLI with a command similar to the one below (The forward slash indicates the file will be found in the top level directory of the SCP user ‘user’. The SCP server is on a machine with the ip address 10.10.10.1 ):
restore remote-manager-backup location 10.10.10.1 user / firewall_backup_Primary_20240107225954.tar
OK, so we’ve reimaged and restored the FMC, we’ve reimaged the FTDs, added them to the FMC and upgraded them to the relevant patch. Once we restore the FTDs with the backup the FMC recognises them as its original FTDs and welcomes them with open arms and the temporary re-imaged FTDs become firewalls non-grata, forgtotten relics that can then be deleted. So we’re done? There were a couple of gotchas…
The temporary FTDs seemed to confuse SmartLicensing. I had to re-register the FMC with a new licensing token from SmartLicense and then re-add the licences to the correct FTDs after having deleted the temporary ones.
OK, so for the final gotcha. I had exported the certificates from the FMC as I know these would be ‘failed’ by the FTD restore. I found when I came to import these I got a ‘cannot find trustpoint’ error. It transpired, that when FTDs in a HA pair are first restored the HA is disabled. They can be brought back into a happy HA state by right clicking and selecting ‘Restore HA’. After this has been done they form a happy HA pair. Then you can import the certs, then you can deploy the policy.
That is all there is to it.