EYEY - GDS Consulting - AI And DATA - Databricks Architect - Senior Manager
About this role
The opportunity
Senior Data & AI Architect with 13+ years of experience in data engineering and data architecture, strong hands-on expertise in Databricks, Hadoop, Spark/PySpark, and cloud data platforms.
This role combines deep technical ownership of enterprise Lakehouse and data platform architecture with hands-on experience building Agentic AI and Retrieval-Augmented Generation (RAG) solutions on Databricks. The ideal candidate is equally comfortable designing large-scale, production-grade data pipelines, architecting modern AI-driven solutions, and engaging directly with clients on strategic proposals and solution design.
Your key responsibilities
- Architect, design, and deliver scalable Data & AI solutions on the Databricks Lakehouse platform, covering ingestion, transformation, storage, governance, and consumption layers.
- Design and optimize large-scale distributed data processing pipelines using Hadoop, Apache Spark, and PySpark.
- Build reusable, production-grade Python frameworks and pipelines for batch and streaming data engineering workloads.
- Define and implement cloud data architecture on AWS and/or Azure, covering storage, IAM, networking, security, and monitoring, with native Databricks integration.
- Own end-to-end ETL/ELT design, data ingestion, transformation, orchestration, and data quality frameworks across the platform.
- Design dimensional data models (star/snowflake schemas, fact/dimension tables, SCD) to support analytical and reporting use cases.
- Design and build Agentic AI solutions on Databricks, including LLM integration, tool/function calling, agent workflows, guardrails, and enterprise data integration.
- Architect and implement RAG (Retrieval-Augmented Generation) solutions using Databricks — document ingestion, chunking, embeddings, vector search, retrieval, grounding, and evaluation.
- Establish and enforce engineering best practices for Git-based version control, code reviews, and CI/CD across development, test, and production environments.
- Lead performance optimization and production deployment of Databricks jobs, workflows, and pipelines at scale.
- Participate in strategic RFP/RFI responses, technical proposals, solution shaping, effort estimation, and architecture presentations.
- Act as a trusted technical advisor in customer workshops, translating business requirements into robust technical architectures.
- Mentor engineering teams on Databricks, Spark, cloud architecture, and modern AI/RAG engineering practices.
Skills and attributes for success
13+ years of experience in Data Engineering / Data Architecture, with strong hands-on experience across Databricks, Hadoop, Spark/PySpark, and cloud data platforms.
- Databricks
- Strong expertise in Databricks Lakehouse, Delta Lake, Databricks SQL, and Unity Catalog.
- Hands-on experience with Jobs/Workflows, performance optimization, and production deployments.
- Hadoop & Spark
- Strong knowledge of the Hadoop ecosystem, HDFS, and Hive.
- Strong hands-on experience with Apache Spark and PySpark for large-scale distributed data processi
- Python
- Strong proficiency in Python for building scalable, reusable, production-grade data pipelines and frameworks.
- Cloud
- Strong experience with AWS and/or Azure, including cloud storage, IAM, networking, security, and monitoring.
- Experience integrating Databricks with cloud-native services.
- Data Engineering
- Expertise in ETL/ELT, batch and streaming processing, data ingestion, transformation, orchestration, and data quality.
- Proven ability to design scalable, production-ready pipeline architectures.
- SQL & Data Modelling
- Strong SQL skills, with experience in dimensional modelling, star/snowflake schemas, fact/dimension models, SCD, and analytical data models.
- Agentic AI
- Hands-on experience designing and building Agentic AI solutions with Databricks, including LLMs, tool/function calling, agent workflows, guardrails, and enterprise data integration.
- RAG (Retrieval-Augmented Generation)
- Strong experience building RAG solutions using Databricks, including document ingestion, chunking, embeddings, vector search, retrieval, grounding, and evaluation.
- Git & CI/CD
- Strong knowledge of Git, branching strategies, pull requests, and code reviews.
- Experience with CI/CD pipelines and deployment across multiple environments.
- Experience with Databricks Asset Bundles and/or Terraform is a plus.
- Strategic RFPs & Solutioning
- Ability to participate in strategic RFP/RFI responses, solution shaping, technical proposals, and effort estimations.
- Experience delivering architecture presentations and leading customer workshops.
- Client Communication
- Strong client-facing and communication skills, with the ability to understand business requirements, translate them into scalable technical solutions, and clearly communicate architectural decisions to both technical and non-technical stakeholders.
Ideally, you’ll also have
- Prior experience in a consulting, systems integrator, or client-facing delivery role.
- Relevant cloud or Databricks certifications (e.g., Databricks Certified Data Engineer / Solutions Architect, AWS/Azure certifications).
- Exposure to enterprise data governance, MLOps, or LLMOps practices.
What working at EY offers
At EY, we’re dedicated to helping our clients, from start–ups to Fortune 500 companies — and the work we do with them is as varied as they are.
You get to work with inspiring and meaningful projects. Our focus is education and coaching alongside practical experience to ensure your personal development. We value our employees and you will be able to control your own development with an individual progression plan. You will quickly grow into a responsible role with challenging and stimulating assignments. Moreover, you will be part of an interdisciplinary environment that emphasizes high quality and knowledge exchange. Plus, we offer:
- Support, coaching and feedback from some of the most engaging colleagues around
- Opportunities to develop new skills and progress your career
- The freedom and flexibility to handle your role in a way that’s right for you
EY | Building a better working world
EY exists to build a better working world, helping to create long-term value for clients, people and society and build trust in the capital markets.
Enabled by data and technology, diverse EY teams in over 150 countries provide trust through assurance and help clients grow, transform and operate.
Working across assurance, consulting, law, strategy, tax and transactions, EY teams ask better questions to find new answers for the complex issues facing our world today.
- Start applying with LinkedIn
- Apply Now Start
-
Please wait...