data
Posted 10 hours agoSr. Data Architect (Databricks)
at Techwave
On-site
Responsibilities
- Define enterprise data architecture aligned with business and technology strategies.
- Lead cloud data modernization and migration initiatives.
- Develop scalable, secure, and highly available data platforms.
- Data Engineering Design ETL/ELT frameworks using Apache Spark and Databricks.
- Architect batch and real-time streaming data pipelines.
- Design Medallion Architecture (Bronze, Silver, Gold).
- Define data ingestion strategies for structured, semi-structured, and unstructured data.
- Implement Change Data Capture (CDC) and incremental processing.
- Data Modeling Design enterprise data models using: Dimensional Modeling Star Schema Snowflake Schema Data Vault Build semantic models for reporting and analytics.
- Define notebook, job, and code organization strategies.
- Improve partitioning, clustering, and file layouts.
- Reduce cloud infrastructure costs through autoscaling and efficient cluster sizing.
- Implement monitoring for compute utilization and query performance.
- Security & Governance Implement Unity Catalog for centralized governance.
- Design role-based access control (RBAC).
- Establish metadata management, lineage, and auditing.
- Ensure compliance with enterprise security and regulatory standards.
- Design data pipelines for machine learning workloads.
- Support Generative AI and Retrieval-Augmented Generation (RAG) solutions by integrating Databricks with vector databases and AI services where appropriate.
- Implement Infrastructure as Code (Terraform preferred).
- Automate deployment of notebooks, jobs, workflows, and platform configurations.
- Establish version control and branching strategies.
- Leadership & Governance Lead architecture reviews and design workshops.
- Define technical standards and best practices.
- Mentor Data Engineers and Technical Leads.
- Collaborate with business stakeholders, project managers, and cloud architects.
- Migrate legacy data warehouses to Databricks.
- Build scalable ETL/ELT pipelines using Spark.
- Implement Delta Lake and Unity Catalog.
- Establish governance, security, and metadata management.
- Optimize performance and cloud costs.
- Deliver curated datasets for BI, AI/ML, and analytics.
- Define DevOps, CI/CD, and operational best practices.
- Lead architecture reviews and guide technical teams throughout the project lifecycle.
Requirements
- We ensure whoever is associated with the brand is being challenged at every step and is provided with all the necessary opportunities to excel in life.
- Job Description Solution Architecture Design end-to-end Lakehouse architectures using Databricks.
- Databricks Platform Architecture Design and optimize: Delta Lake Unity Catalog Databricks Workflows Delta Live Tables (DLT) Databricks SQL MLflow Lakeflow (where applicable) Establish workspace standards and architecture patterns.
- Performance & Cost Optimization Optimize Spark jobs and SQL workloads.
- AI & Advanced Analytics Architect AI/ML platforms using Databricks.
- Enable ML lifecycle management with MLflow.
- DevOps & Automation Define CI/CD strategies using Azure DevOps or GitHub Actions.
- Required Technical Skills Databricks Databricks Lakehouse Platform Apache Spark (PySpark and Spark SQL) Delta Lake Unity Catalog Delta Live Tables Databricks Workflows Databricks SQL MLflow Structured Streaming Cloud Platforms Microsoft Azure (preferred) Azure Data Lake Storage Gen2 Azure Data Factory Azure Synapse Analytics Azure Key Vault Microsoft Entra ID AWS or Google Cloud experience is also valuable.
- Data Engineering ETL/ELT Architecture Data Lakes Data Warehousing Data Modeling CDC Streaming Architecture Metadata Management Programming Python PySpark SQL Scala (preferred) Databases SQL Server Oracle PostgreSQL Snowflake BI & Analytics Power BI Tableau Semantic Modeling DevOps Git Azure DevOps GitHub Terraform CI/CD Pipelines Preferred
- Experience designing enterprise data platforms on Azure, AWS, or GCP. Proven
- experience in large-scale data migration and cloud modernization projects.
- Experience integrating enterprise systems such as SAP, Oracle, Salesforce, Workday, or other business applications.
- Qualifications Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
- Master's degree is preferred.
- Preferred Certifications Databricks Certified Data Engineer Professional Databricks Certified Data Engineer Associate Databricks Certified Machine Learning Professional (preferred) Microsoft Certified: Azure Data Engineer Associate Microsoft Certified: Azure Solutions Architect Expert Azure Fundamentals (AZ-900) Key Competencies Enterprise architecture and solution design Distributed data processing expertise Data governance and security Performance tuning and optimization Cost optimization Stakeholder
- A Databricks Architect serves as the technical leader for modern data platforms, ensuring that solutions are scalable, secure, governed, and aligned with business objectives while enabling advanced analytics and AI workloads.
Experience
- Experience 10–15+ years of IT experience. 5+ years of hands-on Databricks architecture and implementation experience.
Additional details
- Techwave , we are always in an exercise to foster a culture of growth, and inclusivity.
- Who are we? Techwave is a leading global IT and engineering services and solutions company revolutionizing digital transformations.
- We believe in enabling clients to maximize the potential and achieve a greater market with a wide array of technology services, including, but not limited to, Enterprise Resource Planning, Application Development, Analytics, Digital, and the Internet of things (IoT).
- Founded in 2004, headquartered in Houston, TX, USA, Techwave leverages its expertise in Digital Transformation, Enterprise Applications, and Engineering Services to enable businesses accelerate their growth.
- Plus, we're a team of dreamers and doers who are pushing the boundaries of what's possible.
- Provide technical leadership during project delivery.
- Responsibilities Design enterprise Lakehouse architectures.