Why We Chose Hundreds of Pipelines

People sometimes see the number of Azure Data Factory pipelines in one of our environments and assume it is a sign that the design got away from us. That is a fair question. It is also a question with a context-dependent answer.

We chose a more granular shape because a small team supports a changing system in one environment, and recovery matters to us. A count of pipelines does not prove an architecture is good. In our case, the count represents visible boundaries we can name, monitor, and rerun.

What We Were Optimizing For

Long data workloads encounter late files, changed schemas, unavailable services, and ordinary data-quality surprises. When that happens, the useful question is not whether a diagram looks elegant. It is whether an operator can find the affected work, understand its dependencies, and recover without replaying a large amount of already successful work.

Smaller pipelines gave us practical restart points. They also made it easier to assign clear names, inspect run history, and change one part of a load without turning every fix into a broad release. That is a trade: we accepted more artifacts in exchange for smaller operational units.

The Trade We Did Not Avoid

Granularity creates its own work. Pipelines need conventions, domain grouping, dependency documentation, consistent notifications, and review. Without those, a collection of small units becomes hard to navigate rather than easier to operate.

That is why I do not recommend “hundreds of pipelines” as a general rule. A metadata-driven orchestration pattern, a larger pipeline, or another orchestration tool may be the better choice for a different team, workload, or operating model.

For us, the architecture was useful because it fit the constraints we actually had:

• a small team that develops and supports the system;
• workloads that change over time;
• a need to rerun or repair a bounded portion of the load; and
• operators who benefit from seeing the work in named pieces.

What I Would Improve Now

The lesson is not that the existing shape is permanent. If we evolved it today, I would invest in clearer domain grouping, standardized retry and notification patterns, richer metadata, and better generated documentation. Those improvements would make the boundaries easier to navigate without discarding the recovery behavior that has served us.

Architecture is not a contest to produce the smallest number. It is a set of choices about failure, recovery, change, and the people who must operate the system after the original builders are gone. In this environment, visible complexity has been easier to support than hidden complexity.