CloudHive · 8 hours ago

Data Engineer - W2

United States

Contract

Remote

Mid Level

Maximize your interview chances

Staffing & Recruiting

Insider Connection @CloudHive

Discover valuable connections within the company who might provide insights and potential referrals.
Get 3x more responses when you reach out via email instead of LinkedIn.

Responsibilities

Design, develop, and maintain robust data pipelines using Python programming language to extract data from Snowflake and SAP systems and transform it into the desired format for the target system.

Implement efficient data ingestion, transformation, and loading processes to ensure accurate and reliable data transfer between systems.

Collaborate with stakeholders to understand data requirements, source systems, and target systems, and design appropriate data pipelines accordingly.

Write clean, optimized, and scalable code to handle large volumes of data and ensure efficient data flow throughout the pipeline.

Monitor and optimize data pipeline performance, identifying and resolving issues, bottlenecks, and data quality problems.

Define and implement data transformation rules and logic to clean, filter, aggregate, and transform data from Snowflake and SAP into the required format for the target system.

Leverage Python libraries and frameworks such as Pandas, NumPy, or PySpark to manipulate and process data efficiently during the transformation process.

Ensure data quality and integrity by applying data validation, normalization, and standardization techniques.

Develop data mapping and conversion scripts to handle schema differences and ensure data consistency across systems.

Establish connectivity with Snowflake and SAP systems, extracting data through APIs, database connectors, or other relevant methods.

Integrate and synchronize data from multiple sources, ensuring data consistency and coherence.

Collaborate with IT teams to implement secure and efficient data transfer mechanisms, adhering to data governance and compliance policies.

Develop error handling and exception management strategies to handle data transfer failures and ensure data integrity during the integration process.

Document the data pipeline design, architecture, and implementation details, including data source specifications, transformation rules, and target system requirements.

Collaborate with cross-functional teams, including data analysts, data scientists, and business stakeholders, to understand their data needs and provide necessary support.

Participate in meetings and discussions to align data engineering initiatives with business goals.

Stay up-to-date with emerging technologies, tools, and best practices in data engineering and make recommendations for process improvements.

Qualification

Find out how your skills align with this job's requirements. If anything seems off, you can easily click on the tags to select or unselect skills to reflect your actual expertise.

PythonData pipelinesSnowflakeSAPETL processesData transformationSQLData integrationPandasNumPyPySparkData governance