‘Invalid Server Public host Key’ after Ironport ESA upgrade

Recently I had to upgrade Cisco Ironport ESA from 9.1 to 9.7.0. As we know during the upgrade machine gets disconnected from the cluster and later needs to be joined back in manually. Upgrade took place as expected without any issues, machines reconnected back to the cluster but CONNSTATUS command returned <Invalid Server Public host Key> error. Clustercheck command indicated status as administratively disconnected. An email alert was a bit more descriptive – Invalid host key – No public host keys matched the remote host.

 Components:
Cisco Ironport ESA: 9.7.0-125

I started with checking command line logconfig > hostconfig data. The only key I found there was Ironport MSA. I did not see any hostkeys from ESA appliances which were part of the cluster. Previously I came across similar issues on MSA when I had to decommission and re-add new appliance with same IP and hostname. In that case, I would delete appliance, clear all related hostkeys and re-add it successfully. This time, since there was nothing to delete I had to find another way.

Solution to my issue came with the command SCAN.

* Since I’ve already reconnected appliance I went back and disconnected them from the cluster.

(Cluster ESA)> logconfig

Choose the operation you want to perform:
– NEW – Create a new log.
– EDIT – Modify a log subscription.
– DELETE – Remove a log subscription.
– SETUP – General settings.
– LOGHEADERS – Configure headers to log.
HOSTKEYCONFIG – Configure SSH host keys.
– CLUSTERSET – Set how logs are configured in a cluster.
– CLUSTERSHOW – Display how logs are configured in a cluster.

[]> hostkeyconfig

Choose the operation you want to perform:
– NEW – Add a new key.
– EDIT – Modify a key.
– DELETE – Remove a key.
SCAN – Automatically download a host key.
– PRINT – Display a key.
– HOST – Display system host keys.
– FINGERPRINT – Display system host key fingerprints.
– USER – Display system user keys.

[]> scan
 
Please enter the host or IP address to lookup.
//Enter IP address of the other ESA in the cluster.

[]> 192.168.1.1

Choose the ssh protocol type:
//Choose default options and hit Enter

1. SSH2:rsa
2. SSH2:dsa
3. All
[3]>

//Keys will be displayed…..Select Y to add the keys

Add the preceding host key(s) for 192.168.1.1? [Y]>

//Re-run the SCAN for remaining ESA’s. If done hit Enter until you exit out of clusterconfig mode and commit changes

(Cluster ESA)> commit

Perform the same steps on remaining ESA appliances.

Reconnect appliances to the cluster.

(Cluster ESA)> clusterconfig

Choose the operation you want to perform:
– ADDGROUP – Add a cluster group.
– SETGROUP – Set the group that machines are a member of.
– RENAMEGROUP – Rename a cluster group.
– DELETEGROUP – Remove a cluster group.
– REMOVEMACHINE – Remove a machine from the cluster.
– SETNAME – Set the cluster name.
– LIST – List the machines in the cluster.
– CONNSTATUS – Show the status of connections between machines in the cluster.
– COMMUNICATION – Configure how machines communicate within the cluster.
– DISCONNECT – Temporarily detach machines from the cluster.
RECONNECT – Restore connections with machines that were previously detached.
– PREPJOIN – Prepare the addition of a new machine over CCS.
[]> reconnect

Check communication status with connstatus command. All appliances should be listed without any errors.

[]> connstatus

At this time cluster communication should be fixed. If you have MSA then you need to check Centralized Services > Security Appliances tab.  Most likely connection will be broken because hostkeys changed.  Run the same SCAN command on MSA to download new keys from ESA appliances. Do not forget to commit changes. Once done re-establish connection to each security appliance under Security Appliances tab.

2 comments On ‘Invalid Server Public host Key’ after Ironport ESA upgrade

Leave a Reply to Ricardo F. Cancel Reply

Your email address will not be published.

Site Footer

Sliding Sidebar