Hi
Today we upgraded to the latest versions of Veeam/AHV/plugin.
We have 12 worker nodes in our 22 node cluster, and they were all updated today by enabling the "Obtain updates from rpm repositories" open and then testing them. Then afterwards we switch off the automatic updates.
Now that our nightly backups have kicked off we're noticing that when workers are shutting down and then re-starting for another backup, they sometime hang for 15 minutes at this stage:
>3/31/2025 8:32:08 PM Success Connection to the worker service was established successfully 15 min 6 sec
We see this sometimes also:
>3/31/2025 9:22:12 PM Warning Failed to synchronize update settings of the worker: worker IRIS-WORK-10-N: Timeout for Ready state —
And even though we have automatic updates switched off it then hangs for another 15 minutes trying to:
>3/31/2025 9:22:12 PM Running Checking known repositories for updates —
Taking 15 or 30 mins to start 1 worker is a huge problem for us as no other workers get started during this time as Veeam only starts 1 worker at a time. It doesn't seem to be limited to the same workers getting stuck, it seems to be random at this stage.
This is going to have a huge impact on our backups tonight, pushing them to much later in the day tomorrow (1500 VMs total).
I know this is a new problem and will need to be looked into, but our customer is going to have a fit, we've been fighting different problems with Veeam for AHV since we started using it on Nutanix and the reliability just isn't there. We upgraded to this latest version to fix the scheduler issue that has hit us twice already, only to find now we have an issue with worker startups.
Is there a way to avoid workers from powering down for a set amount of time to allow another job to start using it, without the whole shutdown/startup cycle each time?
Today we upgraded to the latest versions of Veeam/AHV/plugin.
We have 12 worker nodes in our 22 node cluster, and they were all updated today by enabling the "Obtain updates from rpm repositories" open and then testing them. Then afterwards we switch off the automatic updates.
Now that our nightly backups have kicked off we're noticing that when workers are shutting down and then re-starting for another backup, they sometime hang for 15 minutes at this stage:
>3/31/2025 8:32:08 PM Success Connection to the worker service was established successfully 15 min 6 sec
We see this sometimes also:
>3/31/2025 9:22:12 PM Warning Failed to synchronize update settings of the worker: worker IRIS-WORK-10-N: Timeout for Ready state —
And even though we have automatic updates switched off it then hangs for another 15 minutes trying to:
>3/31/2025 9:22:12 PM Running Checking known repositories for updates —
Taking 15 or 30 mins to start 1 worker is a huge problem for us as no other workers get started during this time as Veeam only starts 1 worker at a time. It doesn't seem to be limited to the same workers getting stuck, it seems to be random at this stage.
This is going to have a huge impact on our backups tonight, pushing them to much later in the day tomorrow (1500 VMs total).
I know this is a new problem and will need to be looked into, but our customer is going to have a fit, we've been fighting different problems with Veeam for AHV since we started using it on Nutanix and the reliability just isn't there. We upgraded to this latest version to fix the scheduler issue that has hit us twice already, only to find now we have an issue with worker startups.
Is there a way to avoid workers from powering down for a set amount of time to allow another job to start using it, without the whole shutdown/startup cycle each time?
Statistics: Posted by Amarokada — Mar 31, 2025 8:45 pm
