Hi, thank you for the input MPECSInc and nmdange, I appreciate it.
Some feedback.
We had a maintenance window this past weekend where we tested the live migration workaround. Success.
We saw a 10x decrease in latency, we were able to successfully put nodes into maintenance without write latency spiking into the seconds. In fact, before live migrations, the avg latency was ~25ms+. Once we live migrated all the VMs, our avg latency dropped to sub ms digits. We then proceeded to put nodes into maintenance one by one and the highest spike we saw was 5ms, no high increase in write latency into the 100s, none of the previous issues. We would start seeing VMs fallover once the node has been paused for ~20 minutes. Now, nothing, stable, and latencies were low. VMs were happy, everything was happy.
The cluster and its storage were stable. Our avg latency now during production hours, is ~2ms, and this is with the same IO footprint on the volumes. Further monitoring shows that the latencies start increasing after backups, but once I live migrate all the VMs, it drops again.
I am confident that this CBT(RCT)/ReFS bug is present on our platform.
Regarding our storage layout, MPECSInc, thank you for bringing this to my attention, I have over subscribed the pool a bit, and will be correcting that in a future maintenance window. But, I don't think this has much of an impact on the cluster if I look at the metrics now.
Regarding the NICs, yes, we are also firm believers in Mellanox/NVIDIA. However, this build was specced by our suppliers, and I will say, we have had our share of issues with the Intel NICs and their drivers, but with the current driver we have in place, they are behaving themselves nicely.
We have provided our findings (This forum, the case numbers in it, and the results of the maintenance over the weekend) on our MS Case, and will be speaking to an Escalation Team Lead and Support Engineer tomorrow, who will share their findings.
Some feedback.
We had a maintenance window this past weekend where we tested the live migration workaround. Success.
We saw a 10x decrease in latency, we were able to successfully put nodes into maintenance without write latency spiking into the seconds. In fact, before live migrations, the avg latency was ~25ms+. Once we live migrated all the VMs, our avg latency dropped to sub ms digits. We then proceeded to put nodes into maintenance one by one and the highest spike we saw was 5ms, no high increase in write latency into the 100s, none of the previous issues. We would start seeing VMs fallover once the node has been paused for ~20 minutes. Now, nothing, stable, and latencies were low. VMs were happy, everything was happy.
The cluster and its storage were stable. Our avg latency now during production hours, is ~2ms, and this is with the same IO footprint on the volumes. Further monitoring shows that the latencies start increasing after backups, but once I live migrate all the VMs, it drops again.
I am confident that this CBT(RCT)/ReFS bug is present on our platform.
Regarding our storage layout, MPECSInc, thank you for bringing this to my attention, I have over subscribed the pool a bit, and will be correcting that in a future maintenance window. But, I don't think this has much of an impact on the cluster if I look at the metrics now.
Regarding the NICs, yes, we are also firm believers in Mellanox/NVIDIA. However, this build was specced by our suppliers, and I will say, we have had our share of issues with the Intel NICs and their drivers, but with the current driver we have in place, they are behaving themselves nicely.
We have provided our findings (This forum, the case numbers in it, and the results of the maintenance over the weekend) on our MS Case, and will be speaking to an Escalation Team Lead and Support Engineer tomorrow, who will share their findings.
Statistics: Posted by cptkommin — Nov 07, 2024 6:47 pm
![Madonna – Who’s That Girl (1987/2020) [FLAC 24bit/192kHz]](http://imghd.xyz/images/2023/06/08/0093624951148_6004dceff9e1db710a9.jpg)
