E-IT ยท 4 weeks ago
Senior Splunk Infrastructure Engineer
Charlotte, NC
Contract
Onsite
Senior Level
5+ years expE-IT is seeking a Senior Splunk Infrastructure Engineer to manage the health, performance, and stability of multiple Splunk clusters. The role involves Splunk administration, supporting Splunk Enterprise Security, and ensuring optimal performance through various infrastructure management tasks.
Information ServicesInformation TechnologySoftware Engineering
Responsibilities
Manage the health, performance, and stability of multiple Splunk clusters (Search Head Clusters, Indexer Clusters, and Heavy Forwarders)
Maintain and support the underlying infrastructure for Splunk ES, ensuring optimal performance for security operations
Manage applications, parsing rules, and data pipelines on Heavy Forwarders to ensure efficient data ingestion and routing
Design and maintain High Availability (HA) and Disaster Recovery (DR) strategies to ensure business continuity and platform resilience across regions
Champion proactive engineering practices by implementing safe deployment strategies for SaltStack configurations, including canary testing, validation, and staged rollouts to minimize production incidents
Maintain and write complex SaltStack states and formulas to manage Splunk configurations and underlying Linux VMs
Provision, monitor, and scale infrastructure within Google Cloud Platform
Perform deep-dive troubleshooting on Linux systems (kernel tuning, disk I/O, memory management) to ensure optimal Splunk performance
Participate in the on-call rotation to respond to critical incidents affecting Splunk infrastructure availability and performance, ensuring 24/7 reliability
Execute distinct maintenance windows, version upgrades, and patching cycles
Update runbooks and technical documentation within our repository to ensure knowledge sharing, specifically focusing on configuration changes and pipeline architecture
Qualification
Required
Splunk Expertise: 5+ years of experience administering large-scale Splunk Enterprise environments. Proven experience with Indexer Clustering and Search Head Clustering
Advanced SPL (Splunk Processing Language): Proficiency in writing, debugging, and optimizing complex SPL queries to support dashboarding, alerting, and data analysis
Splunk Enterprise Security (ES): Hands-on experience administering and maintaining Splunk ES environments
Configuration Management: Strong proficiency with SaltStack (Salt). You must be comfortable writing custom states and managing configurations for thousands of nodes
Unix/Linux Internals: Deep understanding of Linux administration (RHEL/CentOS/Ubuntu). Ability to debug resource contention, file system issues, and network bottlenecks
Cloud Infrastructure: Hands-on experience with Google Cloud Platform (GCP), specifically GCE and networking
Scripting: Proficiency in Python or Bash for automation tasks
Preferred
Terraform & Infrastructure Workflow: Experience using Terraform for infrastructure provisioning, combined with SaltStack for configuration management. Ideal candidates understand the distinction and interaction between provisioning (Terraform) and configuration (Salt)
Kubernetes & GitOps: Experience with Helm and Flux for managing Kubernetes resources, utilizing GitOps methodologies to drive consistent and automated deployments
"Done For You" / Platform Engineering: Experience building "Done for you" solutions or paved paths that abstract infrastructure complexity for internal customers, enabling self-service and standardization
Multi-Cloud Architecture: Experience managing infrastructure across multiple cloud providers (e.g., GCP, AWS, Azure), understanding the nuances of hybrid networking and data locality
OpenTelemetry (OTEL): Knowledge of migration strategies from Splunk Universal Forwarders to OTEL Collectors to standardize data formats
Experience working in a highly regulated fintech environment
Company
E-IT
E-IT Professionals Corp. (EIT) is an award-winning consulting firm that specializes in IT recruitment management and staffing.
51-200 employees
H1B Sponsorship
E-IT has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (8)
2024 (3)
2023 (16)
2022 (10)
2021 (16)
2020 (28)
Funding
Current Stage
Growth StageLeadership Team
Rao Nalamothu
Chief Executive Officer
Company data provided by crunchbase