Databricks Engg - Systems Integration Specialist

Req ID: 392716

NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.

We are currently seeking a Databricks Engg - Systems Integration Specialist to join our team in Bangalore, Karnātaka (IN-KA), India (IN).

Key Responsibilities

• Design, build, and maintain production data pipelines on Azure Databricks using PySpark, Delta Lake, and a bronze/silver/gold medallion architecture - owning solutions end-to-end from ingestion through curated, consumption-ready datasets.
• Ingest data from diverse sources (Oracle, Netezza, Mongo DB, SQL Server, Synapse, Azure Data Lake, APIs, file feeds) using ADF and Databricks Workflows, handling incremental loads, CDC, schema drift, and late-arriving data.
• Build reusable, config-driven frameworks and libraries for ingestion, transformation, and data quality rather than one-off scripts, enabling the wider team to deliver faster.
• Embed testing and data quality into every pipeline - unit and integration tests, source-to-target reconciliation, and automated DQ gates (null, uniqueness, freshness, row-count thresholds) with clear failure handling and alerting.
• Own CI/CD and environment promotion through Azure DevOps/Git, Databricks Asset Bundles, and infrastructure-as-code, delivering repeatable, auditable deployments across dev, UAT, and production.
• Tune performance and manage cost - diagnose skew, shuffle, and small-file issues, optimize Delta tables and cluster configurations, and continuously reduce runtime and compute spend.
• Provide production support and reliability engineering - monitor SLAs, perform root-cause analysis on failures, and drive design changes and operational improvements that prevent recurrence.
• Contribute to data modelling, data analysis, and solution architecture on demand - profiling sources, producing logical/physical and dimensional models, and translating business requirements into well-structured datasets.
• Design and build data products in alignment with enterprise standards, governance policies, and compliance requirements to support AI model development and training.
• Conduct and act on code reviews, upholding engineering standards, secure coding practices, and compliance requirements including Unity Catalog access controls, PII masking, and audit obligations.
• Document all work and support delivery - design documents, data dictionaries, lineage, and runbooks — plus pipeline demos, UAT support, production deployment, and clear, proactive communication of risks and blockers.
• Coordinate daily with onsite and offshore teams to ensure efficient development, support, and continuous delivery of system enhancements.
• Address data quality issues identified during processing or reported by users, ensuring accurate data correction and backfilling.
• Strong experience in data management, including data classification, data treatment, and data governance standards and practices.
• Proficiency in leveraging AI-assisted development tools such as GitHub Copilot for ETL development, data pipeline engineering, code generation, troubleshooting, and automation of repetitive development tasks.
• Hands-on experience in generating integrated test data across multiple data sources and tables to support lower-environment testing and validation.