Databricks Platform Engineer

Platform Architecture & Engineering responsibilities: • Design and implement scalable Databricks Lakehouse platforms on AWS and/or Azure aligned to client requirements • Architect end-to-end data platforms including ingestion, storage (Delta Lake), processing, and consumption layers • Build and configure cloud infrastructure using infrastructure-as-code (e.g. Terraform & Declarative Automation Bundles(DAB's)) • Establish secure, compliant environments including networking (VNet/VPC, Private Link), identity (IAM/Entra ID), data governance (Unity Catalog), and access controls • Define environment strategies (dev/test/prod), CI/CD pipelines, and release processes for Databricks deployments • Implement monitoring, logging, cost optimisation, and performance tuning across the platform • Design and implement data pipelines using Delta Live Tables, Auto Loader, and Databricks Workflows for both batch and streaming workloads Client Delivery & Enablement responsibilities: • Work directly with clients to translate business and technical requirements into scalable platform designs • Lead technical workshops, architecture sessions, and whiteboarding engagements with client stakeholders • Support rapid prototyping and proof-of-concept builds within Databricks to demonstrate platform capabilities and accelerate client adoption • Provide best practice guidance on Lakehouse architecture, data modelling, workload optimisation, and cost management • Produce high-quality technical documentation including architecture diagrams, architecture decision records (ADRs), runbooks, and deployment guides • Enable client teams through structured knowledge transfer, training, and platform handover • Collaborate with data engineers, data scientists, and product teams to ensure successful delivery outcomes Governance & Security: • Implement Unity Catalog for centralised data governance, including access control (RBAC/ABAC), data lineage, audit logging, and compliance enforcement • Apply security best practices across platform design: network isolation, secret management, encryption at rest and in transit, and identity federation • Ensure platform designs meet client regulatory and compliance requirements (e.g. GDPR, ISO 27001, sector-specific standards) What success looks like in the role: • Delivery of robust, secure, and scalable Databricks platforms that meet client performance and cost expectations • Clear, well-architected solutions that balance flexibility, governance, and operational efficiency • Strong client relationships built on trust, technical credibility, and effective communication • Accelerated client adoption of the Lakehouse platform through well-designed enablement and documentation • Reduced deployment time through reusable infrastructure patterns and automation • Proactive identification of risks, trade-offs, and optimisation opportunities across platform design and delivery • Contribution to the organisation’s growing body of reusable platform accelerators, reference architectures, and internal knowledge Competencies and Behaviours: • 3+ years experience in data platform engineering, cloud engineering, or similar roles • Strong hands-on experience with Databricks, including Apache Spark, Delta Lake, Workflows • Proven experience designing and deploying data platforms on AWS and/or Azure (e.g. ADLS, S3, VNet/VPC, IAM) • Experience with infrastructure-as-code tools (e.g. Terraform preferred) and CI/CD pipelines (e.g. Azure DevOps, GitHub Actions) • Solid understanding of data architecture concepts including Lakehouse medallion architecture and dimensional modelling • Familiarity with security and governance frameworks (e.g. RBAC, ABAC, data masking, audit, compliance standards) • Excellent communication skills with the ability to explain complex technical concepts to non-technical stakeholders • Comfortable working in a client-facing consultancy environment with multiple concurrent engagements • Proactive, self-driven, and able to take ownership of end-to-end platform delivery • Willingness to travel within the UK as required • Right to work in the UK