Databricks Lead EngineerApply Now Full Job Title: Databricks Lead Engineer Job Number: R2109-27087 Location: Illinois; Irving, Texas; Charlotte, North Carolina; Evansville, Indiana; Fort Worth, Texas; Wilmington, Delaware; Baltimore, Maryland Date Posted: 09/03/2021
The Lead Data Engineer - Databricks Expert will report to the Enterprise Data Services team and will building convert legacy data pipelines from SAS to Databricks to help the analytics and data science team in OneMain organization. The candidate will be central critical role for establishing the databricks infrastructure for building data pipelines, provide best practices and recommendations for using databricks, work closely with data scientists and be the point of contact for all things related to databricks.
Design, develop, implement, and tune large-scale distributed systems and pipelines that process large volume of data; focusing on scalability, low-latency, and fault-tolerance in every system built
Developing scalable and re-usable frameworks for ingesting large data in Databricks. Exposure to delta lake framework.
Databricks engineering - query tuning, performance tuning, troubleshooting, and debugging Spark or other big data solutions.
Provides technical leadership, fosters a team environment, and provides mentorship and feedback to technical resources.
Deep understanding of ETL/ELT design methodologies, architecture, strategy, and tactics for complex ETL solution.
Data processing/transformation using various technologies such as spark and cloud Services.
Understand current data engineering pipelines using legacy SAS tools and convert to modern pipelines using Databricks.
5+ years’ experience as data engineer.
Must have 2+ Years in implementing data engineering solutions with Databricks.
This is hands on role building data pipelines using Databricks. Hands-on technical experience with Apache Spark.
Must have deep expertise in one of the programming languages for data processes (Python, Scala). Experience with Python, PySpark, Hadoop, Hive and/or Spark to write data pipelines and data processing layers
Must have worked with relational databases like Snowflake. Good SQL experience for writing complex SQL transformation.
Performance Tuning of Spark SQL running on S3/Data Lake/Delta Lake/ storage and Strong Knowledge on Databricks and Cluster Configurations.
Nice to have Databricks administration including security and infrastructure features of databricks.
Experience with Development Tools for CI/CD, Unit and Integration testing, Automation and Orchestration
Who We Are:
OneMain Financial is the country’s largest lending-exclusive financial company, a trusted name in lending for over 100 years. Since 2005 alone, we have looked beyond customers credit scores to lend more than $152 billion to 16.2 million people looking for simple, affordable loans.
With branches across 44 states, we're proud partners of the families and communities we serve. They turn to us to help meet important financial needs, including debt consolidation, medical expenses, household bills and auto purchases. It’s all about doing the right thing – a mission that hasn’t changed for more than 100 years.
Because we want our team members to bring us their very best every day, we believe they deserve the right opportunities and benefits. That’s why we packed our comprehensive benefits package for full- and some part-timers with:
Health and wellbeing options for team members and their dependents
Up to 4% matching 401(k)
Paid time off
Paid volunteer time
OneMain Holdings, Inc. is an Equal Employment Opportunity (EEO) and Affirmative Action (AA) employer. Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender perception or identity, national origin, age, marital status, protected veteran status, or disability status.