GovCIO · 19 hours ago
Senior Systems Architect-Engineer
United States
Full-time
Remote
Mid, Senior Level
$105K/yr - $145K/yr
7+ years expGovCIO is currently hiring for a Senior Systems Architect-Engineer to support their Administrative Office of the US Courts NLS project. The role involves designing, implementing, and operating various Splunk environments and ensuring optimal performance and security of data operations.
ConsultingIT InfrastructureIT ManagementManagement Consulting
No H1B
Security Clearance Required
Responsibilities
Design, implement, and operate the Splunk Core, Enterprise Security, IT Service Intelligence (i.e., ITSI), Phantom (Security Orchestration, Automation, and Response (SOAR)), Splunk Cloud, Splunk On-Call, and Multi-Site Index Clustering environment
Monitor overall Splunk health through the Monitoring Console (DMC) including indexer, search head, and cluster master status
Track indexing rates, license usage, queue health, and search concurrency to identify performance or ingestion issues early
Monitor CPU, memory, and disk utilization across all Splunk components to ensure optimal resource usage
Respond promptly to health alerts, DMC warnings, or anomalies observed on monitoring dashboards
Investigate and resolve common user-reported issues such as access problems, failed searches, or non-triggering alerts
Troubleshoot data ingestion, parsing, and indexing issues across Universal Forwarders, Heavy Forwarders, and HEC endpoints
Investigate missing or duplicate logs, timestamp errors, or sourcetype misassignments and escalate complex parsing issues to Engineering
Validate new data source onboardings by confirming sourcetype assignment, timestamp accuracy, and field extraction integrity
Support data source owners with forwarder deployment, syslog setup, and connectivity troubleshooting during initial onboarding
Maintain data flow visibility from source → forwarder → indexer to confirm data completeness and performance
Rotate and update credentials, API keys, or tokens used in data inputs, integrations, alerts, and scheduled searches
Manage RBAC user and role mappings, handling access requests, entitlement reviews, and permission troubleshooting
Provide end-user assistance with SPL searches, reports, alerts, and dashboards, including query optimization tips
Maintain and update knowledge base articles, SOPs, and FAQs for repeatable issues and troubleshooting steps
Log and escalate platform or parsing issues to the Engineering team with evidence such as logs, screenshots, and correlation IDs
Open and manage Splunk Support cases for platform-level bugs, license problems, or critical system faults
Monitor and manage ITSI service health, including KPIs, correlation searches, NEAP policies, and summary index latency
Troubleshoot ITSI-related issues such as broken KPIs, delayed episodes, or missing notable events
Perform capacity management by monitoring index growth, bucket rotation, and frozen data retention policies
Conduct periodic system maintenance tasks, including orphaned object cleanup and knowledge object review
Verify and maintain compliance with data governance and retention policies, ensuring secure and auditable configurations
Participate in DR testing and validation to ensure Splunk data recovery and HA configurations are functioning as expected
Document incidents, RCA findings, and preventive actions for future reference
Collaborate closely with the Engineering team for escalations, root-cause investigations, and deployment verifications
Qualification
Required
Bachelor's with 10 years (or commensurate experience) OR Masters Degree or higher (in a related discipline) with 7 years experience
Expert skills in Enterprise Security, ITSI, SOAR, and the Splunk product line
Able to design, implement, and operate the Splunk Core, Enterprise Security, IT Service Intelligence (i.e., ITSI), Phantom (Security Orchestration, Automation, and Response (SOAR)), Splunk Cloud, Splunk On-Call, and Multi-Site Index Clustering environment
Must be able to obtain and maintain AOUSC Public Trust
Benefits
Employee Assistance Program (EAP)
Corporate Discounts
Learning & Development platform, to include certification preparation content
Training, Education and Certification Assistance*
Referral Bonus Program
Internal Mobility Program
Pet Insurance
Flexible Work Environment
Company
GovCIO
GovCIO is a business consulting firm that focuses on cyber security, digital, data, management and mission services, and IT services.
1001-5000 employees
Funding
Current Stage
Late StageLeadership Team
Lynn Oakes
Sr. Vice President, Contracts at GovCIO
Andre Green
Vice President Special Operations Support Solutions (SOSS) GOVCIO
Recent News
Washington Technology
2026-01-14
2026-01-14
Washington Technology
2025-11-26
Company data provided by crunchbase