So our case is still ongoing unfortunately with no resolution.
I can note that the log gap appears to be due to the way that Veeam uploads the data. At least that's what I think based on logs, it moves the data into the cache and then upload from there without adding any logs to the Move.web logs for some reason. For us our source local repository is on a proxy server that's not the "master", and the target object repository is on the "master" server, which causes the gaps in the logs to appear. We found this out by moving our target object repository to the same proxy that has the local repository, and then all the Move.web logs started showing Blob upload entries (when advanced logging is enabled). Apparently the blob upload entries do show on the server that has the object storage attached under the Veeam.Archiver.Proxy logs instead of one of the Move.web logs.
Unfortunately moving the object repository had a negative effect. In reviewing the history, all of our jobs were completing anywhere from 1:45 to 1:55 when running 4 concurrently. One thing we noticed was how consistent this time frame was, and it didn't seem to be affected by site size or object count; we would have almost 10 jobs in a row finish with a runtime with a minute variance of each other. The only thing that changes the runtime of the jobs is how many are going concurrently, reducing concurrent jobs will make them go quicker, but all jobs will still take roughly the same amount of time as other jobs regardless of size or object count. After moving the object repository from the "master" server to the same proxy server hosting the local archive, it made all jobs take a consistent 2:40 runtime with 4 concurrently.
I really think there's something going on. The fact a site migration with 25mb and 50 items takes nearly the exact same time to upload as a 900mb and 5000 item site really says there's something going on. Opening a case with Microsoft they confirmed there's no throttling on their end and we are well under our limits. To date we haven't been able to see any bottlenecks in our chain, nothing on Azure site, nothing in the pipes to azure (10G direct internet or 10G over an express route); and Veeam just states the bottleneck is the target.
One thing that was noted by the tech though is that just running the Get-VBOEntityData command is extremely slow. For me it has always taken anywhere from 10 to 20 minutes to run the command against a repository; apparently this isn't expected and the tech said it should be instant.
So at this point we are planning on moving everything to the master server, both the target object repository as well as the source local repository (detaching the VMDK from the original proxy and attaching it to the master) in hopes it helps, but honestly I don't think so at this point. There is constant traffic going to the object repositories, but it's almost like there's some hardcoded throttle somewhere within Veeam that keeps uploads at a very consistent rate.
I can note that the log gap appears to be due to the way that Veeam uploads the data. At least that's what I think based on logs, it moves the data into the cache and then upload from there without adding any logs to the Move.web logs for some reason. For us our source local repository is on a proxy server that's not the "master", and the target object repository is on the "master" server, which causes the gaps in the logs to appear. We found this out by moving our target object repository to the same proxy that has the local repository, and then all the Move.web logs started showing Blob upload entries (when advanced logging is enabled). Apparently the blob upload entries do show on the server that has the object storage attached under the Veeam.Archiver.Proxy logs instead of one of the Move.web logs.
Unfortunately moving the object repository had a negative effect. In reviewing the history, all of our jobs were completing anywhere from 1:45 to 1:55 when running 4 concurrently. One thing we noticed was how consistent this time frame was, and it didn't seem to be affected by site size or object count; we would have almost 10 jobs in a row finish with a runtime with a minute variance of each other. The only thing that changes the runtime of the jobs is how many are going concurrently, reducing concurrent jobs will make them go quicker, but all jobs will still take roughly the same amount of time as other jobs regardless of size or object count. After moving the object repository from the "master" server to the same proxy server hosting the local archive, it made all jobs take a consistent 2:40 runtime with 4 concurrently.
I really think there's something going on. The fact a site migration with 25mb and 50 items takes nearly the exact same time to upload as a 900mb and 5000 item site really says there's something going on. Opening a case with Microsoft they confirmed there's no throttling on their end and we are well under our limits. To date we haven't been able to see any bottlenecks in our chain, nothing on Azure site, nothing in the pipes to azure (10G direct internet or 10G over an express route); and Veeam just states the bottleneck is the target.
One thing that was noted by the tech though is that just running the Get-VBOEntityData command is extremely slow. For me it has always taken anywhere from 10 to 20 minutes to run the command against a repository; apparently this isn't expected and the tech said it should be instant.
So at this point we are planning on moving everything to the master server, both the target object repository as well as the source local repository (detaching the VMDK from the original proxy and attaching it to the master) in hopes it helps, but honestly I don't think so at this point. There is constant traffic going to the object repositories, but it's almost like there's some hardcoded throttle somewhere within Veeam that keeps uploads at a very consistent rate.
Statistics: Posted by bg.ranken — Dec 21, 2023 9:32 pm
