Chapter 11. Pipeline Architecture Style
Also called pipes and filters — one of the oldest and most recurring styles, appearing the moment functionality is split into discrete parts. You already know it from Unix shells (Bash, Zsh), functional-language constructs, and MapReduce tools. It works at both low-level and higher-level business-application scales.
Topology
The topology is just pipes and filters.
- Pipes form the communication channel between filters. Each is typically unidirectional and point-to-point (not broadcast) for performance, accepting input from one source and directing output to another. Any data format works, but architects favour small payloads for speed.
- Filters are self-contained, independent, and generally stateless, each doing exactly one task. Composite work is handled by a sequence of filters, not a single one.
Four filter types:
- Producer — the starting point, outbound only (the source).
- Transformer — takes input, optionally transforms some/all of it, forwards it (functional map).
- Tester — takes input, tests criteria, optionally produces output based on the test (akin to functional reduce).
- Consumer — the termination point; persists the result to a database or displays it.
Example
ETL tools and orchestrators/mediators (e.g. Apache Camel) use this style. The book’s worked example streams service telemetry into Apache Kafka, then processes it through a pipeline: a Service Info Capture producer subscribes to the Kafka topic; a Duration Filter tester checks whether the data is duration-related and routes it either to a Duration Calculator transformer or onward to an Uptime Filter tester (which routes uptime data to an Uptime Calculator, or ends the pipeline if neither). A Database Output consumer persists results to MongoDB. The clean separation of concerns means a new tester filter (e.g. for DB connection wait time) can be slotted in without touching existing filters — illustrating the style’s extensibility.
Trade-offs
The pipeline architecture is technically partitioned (logic split by filter type) and, being a monolithic deployment, is always a single architecture quantum.
Characteristic ratings (1 = poorly supported, 5 = defining strength):
| Characteristic | Rating |
|---|---|
| Cost | High (cheap) — primary strength |
| Simplicity | High — primary strength |
| Modularity | High — primary strength |
| Deployability | ~3 / average |
| Testability | ~3 / average |
| Reliability | 3 / medium |
| Elasticity | 1 |
| Scalability | 1 |
| Fault tolerance | 1 |
| Number of quanta | Always 1 |
Reasoning:
- Cost, simplicity, and modularity are the strengths — monolithic (no distributed complexity), simple, cheap. Modularity comes from the separation between filter types: any filter can be modified or replaced without impacting the others (e.g. changing the duration calculation in the Kafka example).
- Deployability & testability rate slightly higher than the layered style thanks to filter modularity, but it’s still a monolith, so deployment ceremony/risk/frequency and full-monolith testing still apply.
- Reliability is medium (3), like layered — no network/bandwidth/latency issues, but capped by monolithic deployment and the testability/deployability limits.
- Elasticity & scalability rate 1 — monolithic deployment, single quantum; scaling individual functions needs complex techniques the style isn’t suited to.
- Fault tolerance rates 1 — one out-of-memory condition crashes the whole unit; availability suffers from high MTTR (2–15+ minute startups).
Source
Mark Richards & Neal Ford, Fundamentals of Software Architecture, Chapter 11.