Be an early applicantLess than 25 applicants

Company

Original Job Post

myGwork - LGBTQ+ Business Community · 3 days ago

Lead Data Engineer

New York, United States

Full-time

Onsite

Senior Level

$97K/yr - $196K/yr

8+ years exp

Wonder how qualified you are to the job?

Maximize your interview chances

Internet

Insider Connection @myGwork - LGBTQ+ Business Community

Discover valuable connections within the company who might provide insights and potential referrals, giving your job application an inside edge.

Responsibilities

Design & Build data pipelines with an emphasis on scale, performance and reliability.

Provide technical expertise in the areas of design and implementation of Data Lake solution powered by Databricks on AWS cloud.

Ensure data governance principles adopted, data quality checks and data lineage implemented in each hop of the data.

Partner with the data teams, enterprise architecture organization to ensure best use of standards for the key data domains and use cases.

Continuous learner with an eye on emerging trends around data lake architecture and enterprise data solutions.

Ensure compliance through the adoption of enterprise standards and promotion of best practice / guiding principles aligned with organization standards.

Qualification

Find out how your skills align with this job's requirements. If anything seems off, you can easily click on the tags to select or unselect skills to reflect your actual expertise.

Big Data TechnologiesJavaScalaPythonSparkStreaming Data PipelinesKafkaConfluentData LakeData WarehouseETLELT PipelinesDatabricksSnowflakeAzure Data LakeDatabase TechnologiesAnalytical TechnologiesMPP DatabasesNoSQL DatabasesCloud PlatformsAWSAzureGCPEC2S3AKSEKSAWS CertificationPublic Cloud CertificationContinuous Delivery

Required

Bachelors or Masters degree in Computer Science or Information Technology

8+ years of experience building solutions in big data technologies.

Strong experience programming with more than one of Java, Scala, Python, Spark.

Hands-on experience designing and building streaming data pipelines using Kafka, Confluent etc.

Strongly prefer experience building Data Lake & Data warehouse solutions using ETL,ELT pipelines on Databricks, Snowflake, Azure Data Lake etc.

Strong understanding of database and analytical technologies in the industry including MPP and NoSQL databases

Strong understanding of cloud platforms like AWS, Azure, or GCP, and their services (e.g., EC2, S3, AKS, EKS, etc.).

AWS or any public cloud certification is a must.

Preferred

Experience in continuous delivery through CI/CD pipelines, containers and orchestration technologies.

Experience With Machine Learning Libraries and Frameworks (TensorFlow, MLlib) is an added advantage.

Expert knowledge of Agile approaches to software development and able to put key Agile principles into practice to deliver solutions incrementally.

Monitors industry trends and directions; develops and presents substantive technical recommendations to senior management

Excellent analytical thinking, interpersonal, oral, and written communication skills with strong ability to influence both IT and business partners

Ability to prioritize and manage work to critical project timelines in a fast-paced environment

Financial services industry experience is an added advantage.

Databricks experience or certifications is an added advantage.

Return to Work: Have you taken time out for caring responsibilities and are now looking to return to work? As part of our Return to Work initiative, Restart, we are encouraging enthusiastic and talented returners to apply, and will actively support your return to the workplace.