EYEY - GDS Consulting - AI And DATA -AI Data Platform Engineer - Databricks- Senior
About this role
The opportunity
Build and operate enterprise-scale Data & AI platforms leveraging Databricks Lakehouse architecture. The role focuses on scalable data engineering, reusable platform engineering, governed self-service analytics, AI-enabled data products, APIs, enterprise service integration, Git-based delivery, Data SRE, data security, Immuta-style governance, and agentic automation using Databricks-native services including Genie, AI/BI, Mosaic AI, Unity Catalog, Vector Search, Lakeflow, and Delta Lake.
Your key responsibilities
Databricks Data Engineering
- Design and implement scalable ETL/ELT pipelines using PySpark, Spark SQL, Delta Lake, Databricks Workflows, Lakeflow, and Delta Live Tables.
- Develop reusable ingestion frameworks supporting batch, streaming, event-driven, CDC, file-based, and API-based processing patterns.
- Build curated, analytics-ready, and AI-ready data products with strong quality, lineage, semantic context, and operational controls.
- Optimise workloads for performance, cost, cluster/serverless usage, storage layout, and reliability.
Lakehouse & Platform Engineering
- Create reusable platform accelerators for workspace onboarding, pipeline templates, deployment standards, logging, monitoring, and support runbooks.
- Implement Git connectivity, branching strategy, pull requests, code reviews, CI/CD, deployment bundles, and controlled environment promotion.
- Integrate Databricks with enterprise APIs, source systems, orchestration platforms, governance tools, security services, and downstream analytics consumers.
- Contribute to architecture reviews, technical design documentation, release management, and platform operations.
Genie, AI/BI & Agentic Enablement
- Configure and manage Databricks Genie Spaces, AI/BI dashboards, and governed natural-language analytics over trusted data products.
- Enable Mosaic AI, MLflow, Vector Search, RAG, GraphRAG, agentic workflows, semantic retrieval, and AI-ready data products.
- Apply AI-assisted operations for schema drift detection, anomaly detection, pipeline failure diagnosis, automated documentation, and data quality recommendations.
Governance, Security & Data SRE
- Implement Unity Catalog governance including RBAC/ABAC, lineage, audit logging, data masking, privacy controls, policy enforcement, and AI governance.
- Integrate with Immuta, Microsoft Purview, IAM, secrets management, monitoring tools, and enterprise access workflows.
- Build Data SRE capabilities covering observability, incident management, restartability, SLA/SLO tracking, root-cause analysis, FinOps, and production readiness.
Skills and attributes for success
Skill / capability area - Details
- Core platform - Databricks Lakehouse, Delta Lake, Lakeflow, Delta Live Tables, Databricks Workflows, Databricks SQL, Unity Catalog.
- AI and GenAI - Databricks Genie, AI/BI, Mosaic AI, MLflow, Vector Search, RAG, GraphRAG, Agentic AI, LLMOps, APIs expertise
- Engineering - Python, PySpark, SQL, APIs, Git, CI/CD, unit testing, integration testing, data pipeline testing, GitHub Copilot.
- Cloud and DevOps - AWS, Azure or GCP, Terraform/OpenTofu, Kubernetes, Docker, GitHub Actions, Azure DevOps, Jenkins, policy-as-code, Databricks Asset Bundle
- Governance and reliability - Unity Catalog, Immuta, Purview, lineage, audit, masking, data quality, observability, Data SRE, FinOps, Responsible AI, AI security.
To qualify for the role, you must have
- 5-10 years of experience in data engineering, data platform operations, analytics engineering, platform engineering, or AI platform enablement.
- Strong hands-on implementation experience with cloud data platforms, APIs, Git connectivity, CI/CD, governed access patterns, SRE practices, and production operations.
- Preferred certifications aligned to the relevant cloud/platform stack, data engineering, DevOps, security, governance, and AI/ML engineering.
Ideally, you'll also have
- Strong hands-on engineer with architecture awareness, delivery ownership, and a platform engineering mindset.
- Comfortable turning platform standards into reusable frameworks, secure implementation patterns, operational controls, and production-ready services.
- Able to mentor engineers, collaborate with architects/security/SRE teams, and adopt newer AI-native and agentic engineering methods.
What we look for
- Strong hands-on engineer with architecture awareness, delivery ownership, and a platform engineering mindset.
- Comfortable turning platform standards into reusable frameworks, secure implementation patterns, operational controls, and production-ready services.
- Able to mentor engineers, collaborate with architects/security/SRE teams, and adopt newer AI-native and agentic engineering methods.
What working at EY offers
At EY, we're dedicated to helping our clients, from start-ups to Fortune 500 companies, and the work we do with them is as varied as they are.
You get to work with inspiring and meaningful projects. Our focus is education and coaching alongside practical experience to ensure your personal development. We value our employees and you will be able to control your own development with an individual progression plan. You will quickly grow into a responsible role with challenging and stimulating assignments. Moreover, you will be part of an interdisciplinary environment that emphasises high quality and knowledge exchange. Plus, we offer:
- Support, coaching and feedback from some of the most engaging colleagues around
- Opportunities to develop new skills and progress your career
- The freedom and flexibility to handle your role in a way that's right for you
EY | Building a better working world
EY exists to build a better working world, helping to create long-term value for clients, people and society and build trust in the capital markets.
Enabled by data and technology, diverse EY teams in over 150 countries provide trust through assurance and help clients grow, transform and operate.
Working across assurance, consulting, law, strategy, tax and transactions, EY teams ask better questions to find new answers for the complex issues facing our world today.
- Start applying with LinkedIn
- Apply Now Start
-
Please wait...
- Connect with us
- Our locations
- My EY
- Site map
- Legal and privacy
- Labor condition applications
- Cookie Consent Manager
-
Opens in a new tab.
-
Opens in a new tab.
-
Opens in a new tab.