REGISTRIEREN

Jobbörse

Finde Jobs in deiner Nähe – ob vor Ort, hybrid oder remote.
  • Ähnliche Jobs zu: Data Engineer (Data Bricks and AWS Cloud)
XX
Data Bricks Data EngineerJConnect Infotech • United States
Jetzt Bewerben
XX

Data Bricks Data Engineer

JConnect Infotech
  • US
    United States
  • US
    United States
Jetzt Bewerben

Über

Data Bricks Data Engineer
Location: Seattle, WA / Bellevue, WA / Everett, WA / Renton, WA / Richardson, TX / Plano, TX / Dallas, TX / St. Louis, MO / Charleston, SC / Arlington, VA Duration: Fulltime Job Description: • Awareness of IBM Data Stage ETL/ELT data integration tool to understand existing code. • Develop, test, deploy, optimize, and monitor large-scale data processing workloads in Azure Databricks ETL. • Ensure and lead the efforts to review legacy Data Stage legacy code and migrated Databricks code to ensure functionality is not deviated. • Strong programming skills in Python and PySpark. • Advanced proficiency writing SQL for analytics and ETL processes. • Proven experience building and optimizing complex data pipelines in Azure. • Hands-on experience with Azure Databricks: cluster management, job scheduling, workspace governance. • Strong working knowledge of core Azure services: Storage Account, Synapse, Key Vault, VMSS, Function Apps, Web Apps, Log Analytics Workspace, service principals, and managed identities. • Experience with container services (ACA, container instances) and containerized data workloads. • Familiarity with Azure networking concepts and secure network integration for data platforms. • Experience creating Azure infrastructure using ARM templates. • Proficient with GitLab and Azure DevOps for CI/CD and source control workflows. • Strong analytical, problem-solving, and communication skills; proven ability to work cross-functionally. • Experience working in Agile teams and understanding of data governance frameworks. • Hands-on experience provisioning Databricks resources with Terraform; ability to author and maintain Terraform templates and modules. • Demonstrated experience implementing cluster autoscaling and autoscaling policies through Terraform. • Experience creating reusable Terraform modules and implementing infrastructure-as-code best practices (module structure, state management, remote backends). • Proven experience working on Databricks platform operations, including cluster configuration, job orchestration, and platform optimization. • Experience configuring high-availability Databricks deployments and operating across multiple availability zones/regions. • Familiarity with Metastore/Unity Catalog configuration and metadata governance in Databricks. • Hands-on experience building data pipelines and ingestion workflows into medallion-layer architectures (bronze/silver/gold). • Strong scripting skills (Python, Bash, or similar) and familiarity with CI/CD for Terraform and Databricks deployments. • Strong troubleshooting, performance tuning, and cost optimization skills. Responsibilities: • Design, develop, and maintain end-to-end data pipelines and ETL/ELT workflows using PySpark and Python. • Implement, optimize, and monitor large-scale data processing workloads in Azure Databricks, including cluster configuration, autoscaling, and governance. • Build and maintain data integration and orchestration solutions using Azure services to meet performance, availability, and security requirements. • Collaborate with data consumers, thread authors/owners, and stakeholders to gather business requirements, prioritize needs, and translate analytical objectives into technical designs. • Implement secure data access patterns using Azure Active Directory, Managed Identities, and service principals. • Author Infrastructure-as-Code for Azure resources (ARM templates) and deploy consistent, repeatable environments. • Configure and operate Azure components including Storage Account, Synapse, Key Vault, VMSS, Function Apps, Web Apps, Log Analytics Workspace, Azure Container Apps / container instances, and related services. • Collaborate with networking and security teams to design and implement Azure networking for data solutions. • Implement monitoring, alerting, and cost optimization for data workloads (Log Analytics, metrics, and dashboards). • Use GitLab and Azure DevOps for source control, CI/CD pipelines, and release management. • Follow Agile/Scrum practices and participate in sprint planning, standups, and retrospectives. • Ensure solutions meet data governance, lineage, and compliance requirements. • Operations Support and Oncall Support for Production Issues and Deployments. If you are interested, please send me your updated resume ASAP with below details: Full Name: Current Location/Zip: Contact Number: E-Mail Id: Alternate Email Id: Visa/Work Permit Status: Current Rate/Salary: Expected Base Salary: Notice Period/Availability to Start: Skype ID: Willingness to relocate to job location: Any Relocation Concern (family/house/weather): Current/Previous Employer Name: Preferred Interview timings (Specify Time zone): Any other job opportunity in process & at which stage: Overall Experience Summary: LinkedIn URL: Looking forward for your response.. Thanks and Regards, Kashish Agarwal Jconnect Infotech Inc. 168 Barclay Center Ste. 347, Cherry Hill, NJ 08034 Email: kashish.a@jconnectinc.com LinkedIn ID: www.linkedin.
  • United States

Sprachkenntnisse

  • English
Hinweis für Nutzer

Dieses Stellenangebot stammt von einer Partnerplattform von TieTalent. Klick auf „Jetzt Bewerben”, um deine Bewerbung direkt auf deren Website einzureichen.