38 applicants

Company

InterSources Inc · 17 hours ago

Senior Data Engineer

Mesa, AZ

Full-time

Remote

Senior Level

5+ years exp

Maximize your interview chances

Artificial Intelligence (AI)Cyber Security

Growth Opportunities

Insider Connection @InterSources Inc

Discover valuable connections within the company who might provide insights and potential referrals.
Get 3x more responses when you reach out via email instead of LinkedIn.

Responsibilities

Team up with the engineering teams and enterprise architecture (EA) to define standards, design patterns, accelerators, development practices, DevOps and CI/CD automation

Create and maintain the data ingestion, quality testing and audit framework

Conduct complex data analysis to answer the queries from Business Users or Technology team partners either directly from Analysts or stemmed from one of the Reporting tools suchs PowerBI, Tableau, OBIEE.

Build and automate the data ingestion, transformation and aggregation pipelines using Azure Data Factory, Databricks / Spark, Snowflake, Kafka as well as Enterprise Scheduler tools such as CA Workload automation or Control M

Setup and evangelize the metadata driven approach to data pipelines to promote self service

Setup and continuously improve the data quality and audit monitoring as well as alerting

Constantly evaluate the process automation options and collaborate with engineering as well as architecture to review the proposed design.

Demonstrate mastery of build and release engineering principles and methodologies including source control, branch management, build and smoke testing, archiving and retention practices

Adhere to and enhance and document the design principles, best practices by collaborating with Solution and in some cases Enterprise Architects

Participate in and support the Data Academy and Data Literacy program to train the Business Users and Technology teams on Data

Respond SLA driven production data quality or pipeline issues

Work in a fast-paced Agile/Scrum environment

Identify and assist with implementation of DevOps practices in support of fully automated deployments

Document the Data Flow Diagrams, Data Models, Technical Data Mapping and Production Support Information for Data Pipelines

Follow the Industry standard data security practices and evangelize the same across the team.

Qualification

Find out how your skills align with this job's requirements. If anything seems off, you can easily click on the tags to select or unselect skills to reflect your actual expertise.

Azure Data FactoryDatabricks/SparkSQLData AnalysisPython/PySparkETL PlatformsAgile MethodologiesDevOps PracticesPowerBITableauOBIEETerraformAzure CLIPowerShellKubernetesDockerGitLabSLA Driven SupportData Quality Monitoring

Required

5+ years of experience in an Enterprise Data Management or Data Engineering role

3+ years of hands on experience in building metadata driven data pipelines using Azure Data Factory

Databricks / Spark for Cloud Datalake

5+ years hands on experience with using one or more of the following for data analysis and wrangling Databricks, Python / PySpark, Jupyter Notebooks

Expert level SQL knowledge on databases such as but not limited to Snowflake, Netezza, Oracle, Sql Server, MySQL, Teradata

3+ years of hands on experience on one or more of big data technologies such as Cloudera Hadoop, Pivotal, Vertica, MapR

Experience working in a multi developer environment and hands on experience in using either azure devops or gitlab

Preferably experienced in SLA driven Production Data Pipeline or Quality support

Experience or strong understanding of the traditional enterprise ETL platforms such as IBMDatastage, Informatica, Pentaho, Ab Initio etc.

Functional knowledge of some of the following technologies - Terraform, Azure CLI, PowerShell, Containerization (Kubernetes, Docker)

Functional knowledge of one or more Reporting tools such as PowerBI, Tableau, OBIEE

Team player with excellent communication skills, ability to communicate with the customer directly and able to explain the status of the deliverables in scrum calls

Ability to implement Agile methodologies and work in an Agile DevOps environment

Bachelor's degree in Computer Science or Engineering or Mathematics or related field and 5+ years of experience in various cloud technologies within a large-scale organization

Personal Attributes: Self-starter, Collaborative, Curious, Strong work ethic, highly motivated, Team oriented

Experience designing and building complex data pipelines in an agile environment

Expertise on data analysis and wrangling using sql, python, databricks

Experience with modern cloud development and design concepts; software development lifecycle; multi-developer code versioning and conflict resolution; planning, design, and problem resolution enterprise data applications / solutions

Demonstrated ability in developing a culture that embraces innovation, and challenges existing paradigms