Colossus Technologies Group · 7 hours ago
Sr. Data Engineer - Databricks/Telemetry - remote
United States
Part-time
Remote
Senior Level
$170K/yr - $210K/yrColossus Technologies Group is seeking a Senior Data Engineer to join their Observability Data Platform team at a public healthcare company. The role involves designing and operating large-scale ingestion pipelines into a Security Data Lake on Azure to support advanced analytics and AI-driven insights, as well as managing complex log routing and schema design.
Computer & Network Security
Responsibilities
Design and operate large-scale ingestion pipelines into a Security Data Lake on Azure
Handle extremely high-throughput log flows from a wide range of sources
Build and manage complex log routing, parsing, enrichment, and forwarding using Cribl
Landing data in Databricks and downstream platforms like Splunk
Own schema design and transformations
Map diverse telemetry into OCSF to create consistent, high-quality datasets that can be reliably consumed by analytics, detection, and AI systems
Create data pipelines and models that directly power machine learning, automated detection, and intelligent observability capabilities
Qualification
Required
Experience with Azure (data and cloud infrastructure)
Experience with Databricks (Spark, Delta Lake)
Experience with Cribl (log ingestion, parsing, enrichment, and routing)
Experience with Splunk
Experience in building large-scale log and telemetry pipelines
Experience with streaming and batch data processing
Knowledge of OCSF and schema normalization
Experience with data platforms supporting AI and machine learning workloads
Ability to design and operate large-scale ingestion pipelines into a Security Data Lake
Experience handling extremely high-throughput log flows from a wide range of sources
Ability to build and manage complex log routing, parsing, enrichment, and forwarding
Ownership of schema design and transformations
Ability to map diverse telemetry into OCSF to create consistent, high-quality datasets
Experience in creating data pipelines and models that power machine learning, automated detection, and intelligent observability capabilities
Company
Colossus Technologies Group
At Colossus Technologies Group, we provide top-tier cybersecurity and digital trust staffing, IT consulting, and project management solutions, including privacy, security, and GRC SaaS platform solution implementation.
Founded in 2021
2-10 employees
Funding
Current Stage
Early StageLeadership Team
Kort Evans
Founder and Principal Cybersecurity Consultant
Company data provided by crunchbase