Senior Data Engineer & Technical Delivery Lead (Python / Databricks)

Senior Data Engineer & Technical Delivery Lead (Python / Databricks) Location: London (Hybrid – 2-3 days onsite) Contract: Initial 6 months We're currently supporting a major Investment Bank that's expanding its Data Engineering function and is looking to hire multiple contractors across Senior Data Engineer and Technical Delivery Lead level. Whether you're a highly technical Senior Data Engineer looking to remain hands-on, or an experienced Technical Lead who enjoys combining delivery leadership with coding, we'd be keen to speak with you. You'll be joining a large-scale cloud data transformation programme, helping design, build and optimise enterprise data platforms using Python, Databricks and Spark within a modern Lakehouse environment. Key Responsibilities • Design, develop and maintain scalable data pipelines using Python and Databricks. • Build, optimise and support enterprise ETL/ELT workflows using Apache Spark and Delta Lake. • Design and develop robust data models and Lakehouse architectures. • Implement and manage data workflows within the Databricks ecosystem. • Ensure high levels of data quality, governance and pipeline reliability. • Optimise performance across large-scale distributed data processing platforms. • Collaborate with architects, product owners, analysts and engineering teams to deliver high-quality data solutions. • Implement monitoring, logging and alerting across critical data platforms. • Drive engineering best practice through code reviews, mentoring and technical collaboration. • For Technical Delivery Lead opportunities, provide technical leadership, architectural oversight and end-to-end delivery of complex data engineering initiatives. Required Skills & Experience • Strong commercial experience developing data engineering solutions using Python. • Extensive hands-on experience with Databricks, including Workflows, Notebooks and Delta Lake. • Strong experience with Apache Spark / PySpark. • Proven experience building scalable ETL/ELT pipelines. • Strong SQL and data modelling skills. • Experience designing modern Lakehouse architectures. • Experience working with cloud platforms including AWS, Azure or GCP. • Experience with modern data warehousing technologies such as Snowflake, Redshift or BigQuery. • Experience with orchestration tools such as Airflow or Databricks Workflows. • Previous experience working within Agile delivery environments. Desirable Experience • Databricks certifications. • Kafka or Structured Streaming. • CI/CD and DevOps practices. • Docker and Kubernetes. • Exposure to Machine Learning pipelines. • Experience within Financial Services or other highly regulated environments. About You You'll be an experienced Data Engineer or Technical Lead with a passion for building modern, scalable data platforms. Comfortable working in a collaborative Agile environment, you'll enjoy solving complex technical problems and delivering high-quality engineering solutions. If you're applying at Technical Delivery Lead level, you'll also bring strong leadership, stakeholder management and mentoring experience whilst remaining hands-on technically.