Case Study: Forensics of Slow Migrated Builds

Case Study: Forensics of Migrated Builds Slow in the Cloud

The Problem

A client decided to move CI infrastructure from on-premise to Azure. The project was effectively running several dozen similar jobs which were migrated into the cloud. Even though the on-premise hardware was outdated, a much more powerful 8-core 32GB RAM virtual machine on Azure added about 10 hours to a 3 hour 40 minute build. The root cause was unclear since no structural changes had been made to the execution code.

The Investigation

I instrumented the pipeline with granular timing on each build phase and correlated the data with infrastructure metrics. The build agents themselves were performing within specification. The delay pattern pointed upstream, away from the pipeline configuration.

Root Cause

The on-premise license server was misconfigured for the new network topology after migration. Every license checkout added a timeout-and-retry cycle which was executed per translation unit. The same misconfiguration was silently causing some developer workstation slowdowns, as IDE license checks hit the same bottleneck.

The Fix

I corrected the license server configuration to match the post-migration network layout. Pipeline duration dropped by the full 10-hour regression plus additional time (up to 43% faster), making the Azure pipeline faster than the original on-premise baseline. I implemented monitoring to detect new misconfigurations after future license updates.

Result

• Pipeline duration: builds on Azure now run 32% to 43% faster than the on-premise baseline
• Some developer workstation slowdowns resolved as a side effect
• Cloud resources spent 32% to 43% less time per build, delivering results to developers sooner and reducing cost because resources could be deprovisioned earlier
• Scope: 50 pipelines benefited from this improvement
• Total timeline: 5 days from engagement start to verified result