200+ applicantsPosted by Agency
Paradigm Nat'l ยท 2 days ago
Senior Data Engineer
United States
Full-time
Remote
Senior Level
7+ years exp
Wonder how qualified you are to the job?
Maximize your interview chancesStaffing and Recruiting
Hiring Manager
Tim Warren
Insider Connection @Paradigm Nat'l
Responsibilities
Process unstructured data into a form suitable for analysis.
Support the business with ad hoc data analysis and build reliable data pipelines.
Implementation of best practices and IT operations in mission-critical tighter SLA data pipelines using Airflow.
Query Engine Migration from Dremio to Redshift.
Leverage multiple AWS Data & Analytic Services (e.g., Glue, Kinesis, S3), SQL (e.g., PostgreSQL, Redshift, Athena), NoSQL (e.g., DocumentDB, MongoDB), Kafka, Docker, Spark (AWS EMR and DataBricks), Airflow, Dremio, Qubole, etc.
Utilize AWS extensively, requiring experience with AWS cloud and AWS Data & Analytics certification.
Build the infrastructure for optimal extraction, transformation, and loading of data from various sources using SQL and cloud big data technologies like DataBricks, Snowflake, Dremio, and Qubole.
Build processes supporting data transformation, data structures, metadata, dependency, and workload management.
Manipulate, process, and extract value from large datasets.
Create a platform for building complex data pipelines using orchestration tools like Airflow and Astronomer.
Implement real-time sync between OLTP and OLAP using AWS technologies like realtime sync between AWS Aurora and AWS Redshift.
Qualification
Find out how your skills align with this job's requirements. If anything seems off, you can easily click on the tags to select or unselect skills to reflect your actual expertise.
Required
7+ years of real-world Data Engineering experience.
Programming experience, ideally in Python and other data engineering languages like Scala
Programming knowledge to clean structure and semi-structure datasets.
Experience processing large amounts of structured and unstructured data. Streaming data experience is a plus.
Experience building and optimizing big data data pipelines, architectures, and data sets.
Background in Linux
Build the infrastructure required for optimal extraction, transformation, and loading of data from various data sources using SQL and other cloud big data technologies like DataBricks, Snowflake, Dremio, and Qubole.
Build processes supporting data transformation, data structures, metadata, dependency, and workload management
A successful history of manipulating, processing, and extracting value from large, disconnected datasets.
Experience creating a platform on which complex data pipelines are built using orchestration tools like Airflow, and Astronomer.
Experience with real-time sync between OLTP and OLAP using AWS technologies like realtime sync between AWS Aurora and AWS Redshift.
Preferred
Experience with AWS cloud and AWS Data & Analytics certification will help you hit the ground running.
Company
Paradigm Nat'l
At our skilled IT staffing and recruiting agency, we place a strong emphasis on fostering genuine relationships between employees and employers.
2-10 employees
Funding
Current Stage
Early StageLeadership Team
Scott Pollard
Managing Partner
Tim Warren
Managing Partner
Company data provided by crunchbase
