Shamrock Trading Corporation · 2 days ago

Data Engineer

Overland Park, KS

Full-time

Remote

Entry, Mid Level

Wonder how qualified you are to the job?

Maximize your interview chances

Financial ServicesSoftware

Comp. & Benefits

Hiring Manager

Nickie (Branting) Firkins

Insider Connection @Shamrock Trading Corporation

Discover valuable connections within the company who might provide insights and potential referrals, giving your job application an inside edge.

Responsibilities

Work with Data architects to understand current data models, to build pipelines for data ingestion and transformation.

Design, build, and maintain a framework for pipeline observation and monitoring, focusing on reliability and performance of jobs.

Surface data integration errors to the proper teams focusing on: Ensuring timely processing of new data, Performance of data pipelines, Integrity and quality of source data.

Hands-on experience building data-lake style infrastructures using streaming data set technologies (particularly with Apache Kafka)

Qualification

Find out how your skills align with this job's requirements. If anything seems off, you can easily click on the tags to select or unselect skills to reflect your actual expertise.

AWSData PipelinesAWS LambdaDockerECSMSKAirflowDatabricksUnity CatalogPythonPandasNumpyBoto3SimpleSalesforceApache SparkKafkaKafka ConnectSchema RegistryAvroTerraformInfrastructure as CodeCustomer InteractionProblem-SolvingCommunicationRelational DatabasesNoSQL DatabasesPostgresRedshiftMongoDBVersion Control

Required

Bachelor’s degree in computer science, data science or related technical field, or equivalent practical experience

Experience building and maintaining AWS based data pipelines: currently utilizing AWS Lambda, Docker / ECS, MSK, Airflow, Databricks, Unity Catalog

Development experience utilizing two or more of the following: Python (Pandas/Numpy, Boto3, SimpleSalesforce), Databricks (pySpark, pySQL, DLT), Apache Spark, Kafka and the Kafka Connect ecosystem (schema registry and Avro), Terraform (or other infrastructure as code platform)

Enthusiasm for working directly with customer teams (Business units and internal IT)