Data Engineer

ESSENTIAL ROLES & RESPONSIBILITIES • Identify and understand customer data-centric use cases within regulated financial services environments • Design and implement data ingestion, processing, and transformation pipelines on Azure • Build and maintain data pipelines for cleaning, normalisation, enrichment, and preparation • Apply appropriate data modelling techniques and architecture patterns, with a strong focus on medallion architecture • Orchestrate, monitor, and optimise Azure Databricks jobs and Azure Data Factory pipelines across development, UAT, and production environments • Configure platforms, clusters, and compute resources to optimise performance, cost, and reliability • Use automated CI/CD pipelines to manage, deploy, and version data artefacts and pipelines • Operationalise workflows developed by analysts and data scientists • Support customers in adopting Azure data, analytics, and machine learning services • Ensure secure storage, processing, and quality of customer data • Ensure networking and security best practices are applied when designing and operating data solutions • Design solutions for processing large volumes of data using batch and streaming approaches • Collaborate with analytics teams on data visualisation best practices and reporting enablement • Ensure all solutions are well-documented, including pipelines, schemas, transformations, and operational runbooks GOVERNANCE & REPORTING • Maintain accurate documentation of data pipelines, schemas, transformations, and deployment processes • Support data governance initiatives including lineage, metadata management, and access control • Contribute to service reporting, risk tracking, and continuous improvement actions • Ensure data environments are audit-ready and aligned with governance standards TECHNOLOGY STACK (AZURE) Cloud Platform: • Microsoft Azure Data Engineering & Analytics: • Azure Databricks (development, UAT, and production) • Azure Data Factory • Azure Synapse Analytics (where applicable) Machine Learning & AI: • Azure Machine Learning (limited non-production usage) • Azure Document Intelligence Databases: • Microsoft SQL Server / Azure SQL Database (primary platforms) • PostgreSQL (limited use) • MySQL (limited use) Data Processing: • Batch and streaming data pipelines Security & Governance: • Role-based access control (RBAC) • Data encryption and key management • Audit logging and monitoring DevOps: • CI/CD pipelines for data artefacts and infrastructure BEHAVIOURAL COMPETENCIES – ORGANISATIONAL & BEHAVIOURAL FIT • Positive mindset and enthusiasm for learning new technologies • Collaborative and supportive team player • Strong sense of ownership and accountability • Methodical, analytical approach to problem-solving • Strong understanding of ethical data usage in regulated environments CRITICAL COMPETENCIES – TECHNICAL FIT Essential: • Strong SQL skills • Programming experience with Python and/or Scala • Hands-on experience with Azure-based data platforms • Experience designing, building, and maintaining data pipelines • Strong understanding of data modelling (relational and analytical), including medallion architecture • Experience orchestrating and optimising Databricks and Data Factory workloads • Experience using CI/CD pipelines for data and analytics solutions • Strong awareness of security, networking best practices, GDPR, and PII handling Desirable: • Experience with Azure Databricks in production environments • Familiarity with Azure Machine Learning and AI services • Exposure to data visualisation tools (e.g. Power BI) • Experience with big data frameworks (Spark, Kafka) • Knowledge of data governance, lineage, and metadata tooling SHIFT & WORKING PATTERN • Standard business hours, with participation in an on-call rota as required • Occasional weekend engineering coverage will be required, typically limited to a small number of planned weekends per year to support business continuity, resilience testing, or disaster recovery activities