Scaling Data Pipelines in Apache Spark