Apply on Employer Site

TestingXperts · 1 day ago

Senior Splunk Infrastructure Engineer

Charlotte, NC

Contract

Onsite

Senior Level

5+ years exp

TestingXperts is seeking a Senior Splunk Infrastructure Engineer to assist in managing large-scale logging and monitoring platforms. The role involves maintaining, upgrading, and optimizing Splunk infrastructure on Google Cloud Platform, while working closely with Observability engineers.

DevOpsInformation TechnologyPenetration TestingQuality AssuranceSoftwareUsability Testing

H1B Sponsor Likely

Responsibilities

Splunk Administration: Manage the health, performance, and stability of multiple Splunk clusters (Search Head Clusters, Indexer Clusters, and Heavy Forwarders)

Splunk Enterprise Security (ES) Support: maintain and support the underlying infrastructure for Splunk ES, ensuring optimal performance for security operations

Heavy Forwarder & Pipeline Management: Manage applications, parsing rules, and data pipelines on Heavy Forwarders to ensure efficient data ingestion and routing

Reliability & HA/DR: Design and maintain High Availability (HA) and Disaster Recovery (DR) strategies to ensure business continuity and platform resilience across regions

Safe Configuration Management: Champion proactive engineering practices by implementing safe deployment strategies for SaltStack configurations, including canary testing, validation, and staged rollouts to minimize production incidents

Infrastructure as Code: Maintain and write complex SaltStack states and formulas to manage Splunk configurations and underlying Linux VMs

GCP Operations: Provision, monitor, and scale infrastructure within Google Cloud Platform

System Optimization: Perform deep-dive troubleshooting on Linux systems (kernel tuning, disk I/O, memory management) to ensure optimal Splunk performance

On-Call Support: Participate in the on-call rotation to respond to critical incidents affecting Splunk infrastructure availability and performance, ensuring 24/7 reliability

Maintenance & Upgrades: Execute distinct maintenance windows, version upgrades, and patching cycles

Documentation: Update runbooks and technical documentation within our repository to ensure knowledge sharing, specifically focusing on configuration changes and pipeline architecture

Qualification

Splunk ExpertiseAdvanced SPLConfiguration ManagementUnix/Linux InternalsCloud InfrastructureScriptingTerraform & Infrastructure WorkflowKubernetes & GitOpsMulti-Cloud ArchitectureOpenTelemetry (OTEL)

Required

5+ years of experience administering large-scale Splunk Enterprise environments

Proven experience with Indexer Clustering and Search Head Clustering

Proficiency in writing, debugging, and optimizing complex SPL queries to support dashboarding, alerting, and data analysis

Hands-on experience administering and maintaining Splunk ES environments

Strong proficiency with SaltStack (Salt). You must be comfortable writing custom states and managing configurations for thousands of nodes

Deep understanding of Linux administration (RHEL/CentOS/Ubuntu). Ability to debug resource contention, file system issues, and network bottlenecks

Hands-on experience with Google Cloud Platform (GCP), specifically GCE and networking

Proficiency in Python or Bash for automation tasks

Preferred

Experience using Terraform for infrastructure provisioning, combined with SaltStack for configuration management

Experience with Helm and Flux for managing Kubernetes resources, utilizing GitOps methodologies to drive consistent and automated deployments

Experience building 'Done for you' solutions or paved paths that abstract infrastructure complexity for internal customers, enabling self-service and standardization

Experience managing infrastructure across multiple cloud providers (e.g., GCP, AWS, Azure), understanding the nuances of hybrid networking and data locality

Knowledge of migration strategies from Splunk Universal Forwarders to OTEL Collectors to standardize data formats

Experience working in a highly regulated fintech environment

Company

TestingXperts

Glassdoor4.1

Next Gen QA & Software Testing Company

Founded in 1996

Mechanicsburg, Pennsylvania, USA

1001-5000 employees

https://www.testingxperts.com/

H1B Sponsorship

TestingXperts has a track record of offering H1B sponsorships. Please note that this does not guarantee sponsorship for this specific role. Below presents additional info for your reference. (Data Powered by US Department of Labor)

Distribution of Different Job Fields Receiving Sponsorship

Represents job field similar to this job

Trends of Total Sponsorships

2025 (10)

2024 (1)

2020 (1)