CatchProbe Intelligence Technologies · 1 month ago
Senior Database Engineer(Elastic/Mongo/Hadoop)
San Francisco, CA
Full-time
Hybrid
Senior Level
5+ years expCatchProbe Intelligence Technologies is seeking a Senior Database Engineer with expertise in NoSQL databases such as MongoDB, ElasticSearch, and Hadoop. The role involves administration, maintenance, and performance analysis of database clusters, as well as collaboration with application teams to optimize database solutions.
AnalyticsArtificial Intelligence (AI)Big DataCyber SecurityInformation TechnologyIntrusion DetectionNetwork SecurityRisk ManagementSoftware
Responsibilities
Develops solutions for very complex and wide reaching systems engineering problems. Sets new policies and procedures to handle future issues. Creates systems engineering and architectural documentation to be used by others to build and maintain systems
Operating Systems & Disk Management: Provides in-depth knowledge of Operating System internals to aid in solving complex problems. Acts as an authority on at least one supported Operating System. Mentors and trains more junior team members on Operating System concepts, configuration, tuning and troubleshooting techniques. Creates basic task automation scripts in Shell, Perl, VBscript, or similar. May manage servers remotely in a distributed environment
Database Platform Management: Masters understanding of database concepts, availability, performance, usage and configuration. Sets up, fixes and tunes complex standard and non-standard replication. Uses knowledge of existing database platforms to evaluate and recommend new technologies. Uses database knowledge to solve issues on unfamiliar products. Creates and maintains database policies, standards and overall documentation including availability, replication, availability and backup and recovery policy, service level agreement, baseline architecture, change management, access to production, unsupported HW/SW, security & audit violations and risk acceptance
Storage and Backup: Sets up, solves problems and tunes complex SAN software issues. Uses knowledge of existing storage platforms to evaluate and recommend new technologies. Uses storage knowledge to solve issues on unfamiliar products. Creates and maintain policies, standards and overall documentation including availability and backup & recovery, service level agreement, baseline architecture, change management, access to production, unsupported HW/SW, security & audit violations and risk acceptance
Scripting and Development: Develops software in several modern languages. Develops database-backed systems and has a solid grasp of DB schema and query performance. Designs horizontally-scalable solutions with innovative use of storage and networking, including solid APIs for integration with other systems. Applies professional standard methodologies in day-to-day work like revision control, unit testing, or other. Applies statistical data analysis techniques
Networking: Recommends or helps architect an entire system, including network design and topology. Authority in understanding and performing TCP dumps, snoop and other network sniffers. Understands and applies knowledge of most protocols (TCP/IP, HTTP, UDP etc.)
Application Technologies (Web Servers, J2EE, Applications Servers): Provides authoritative recommendations and advice to the team and/or department in the areas of web services, OS and storage, including being an active liaison to Development, QA and the Business. Provides scaling, design, costing, troubleshooting and impact analysis consultation
Analyzes systems and makes recommendations to prevent possible problems. Takes lead on issue resolution activities using knowledge of complex and company-wide systems
Leads end-to end audit of monitors and alarms based on subsystem knowledge. Takes the lead on defining the requirements for new tools required for operations
Applies time management and project management skills to lead the resolution of issues in a timely and organized manner, effectively communicating necessary information. May consult directly with clients or third party vendors; provides domain expertise
Consistent exercise of independent judgment and discretion in matters of significance
Regular, consistent and punctual attendance
Qualification
Required
Must have experience with MongoDB installations, upgrades, support on MongoDB
Responsible for administration, maintenance, Performance analysis, and Capacity planning for Mongo dB/Elastic/Hadoop clusters
Coordinate and plan with Application teams on Mongodb capacity planning for new applications
Should have knowledge of using mongo base tools - like mongodump , mongoexport ,mongorestore, mongoimport , mongostat , mongotop
Must be well-versed with JSON scripting , writing queries in mongo in shell scripts and in the mongo shell
Should be able to support sharded clusters and perform upgrades and other config maintenance on sharded clusters
Must be able to address , monitor and manage capacity requirements - all aspects CPU , memory and storage
Must be able to assist application teams with assessment and/or resolution of performance bottlenecks observed in the MongoDB Database tier of their stack
Must be aware of different authentication/authorization methods used in Mongo - SCRAM-SHA1 , X509 , LDAP - including reconfiguration of instances with TLS/SSL and/or LDAP
Candidate must also be able to develop automated solutions for ad-hoc script execution requests, ad-hoc report generation, upgrades , installs
Experienced in NoSQL DB technologies
Must be aware of how to use Cloud manager and share relevant metrics for a given deployment when an issue arises
Must have experience with docker, deploying Mongo containers running on docker and supporting all aspects of MongoDB administration needs within a docker container
Knowledge of administration and support of Hadoop systems will be an added advance
Deploy Hadoop(Bigdata) cluster, Comm/decommissioning of nodes, track jobs, monitor services like zookeeper, hbase, SOLR indexing, configure name-node HA, schedule, configuring backups & restore
To develop script to review logs and alert us in case of long running queries
Demonstrable expertise (deployment, use) in some of the following data storage technologies: Postgres/MySQL (“on-prem”, RDS, Aurora), Kafka / Kinesis (“on-prem” or managed)
Strong Understanding of various relational and non-relational database technologies along with their benefits, downsides and best use-case and help application teams to use the correct database technology based on their specific business use case
5+ year installing, automating, scaling and supporting NoSQL databases such as MongoDB, ElasticSearch, Hadoop, among other emerging technologies
1-2 years Experience working with the databases in public Clouds like Hetzner, AWS, Azure and GCP
Proficiency in automation
Knowledge of Ansible, Python, Terraform
Willingness and commitment to learn other database, automation, and Cloud technologies
Great communication and collaboration skills
Software development experience and knowledge of modern software development processes
Ability to multi-task and prioritize with little to no supervision, providing team leadership skills
Ability to work well under pressure
Consistent exercise of independent judgment and discretion in matters of significance
Excellent communication skills
Highly driven, highly involved, highly proactive
Datalake cluster ownership and Technical point of contact for all applications on Hadoop cluster
Responsible for new application on-boarding in Datalake by reviewing requirement and design
Assist existing and new applications to come up with most optimized and suitable solutions for their requirement
L3 point of contact for issues related to Hadoop platform
Develops solutions for very complex and wide reaching systems engineering problems. Sets new policies and procedures to handle future issues. Creates systems engineering and architectural documentation to be used by others to build and maintain systems
Operating Systems & Disk Management: Provides in-depth knowledge of Operating System internals to aid in solving complex problems. Acts as an authority on at least one supported Operating System. Mentors and trains more junior team members on Operating System concepts, configuration, tuning and troubleshooting techniques. Creates basic task automation scripts in Shell, Perl, VBscript, or similar. May manage servers remotely in a distributed environment
Database Platform Management: Masters understanding of database concepts, availability, performance, usage and configuration. Sets up, fixes and tunes complex standard and non-standard replication. Uses knowledge of existing database platforms to evaluate and recommend new technologies. Uses database knowledge to solve issues on unfamiliar products. Creates and maintains database policies, standards and overall documentation including availability, replication, availability and backup and recovery policy, service level agreement, baseline architecture, change management, access to production, unsupported HW/SW, security & audit violations and risk acceptance
Storage and Backup: Sets up, solves problems and tunes complex SAN software issues. Uses knowledge of existing storage platforms to evaluate and recommend new technologies. Uses storage knowledge to solve issues on unfamiliar products. Creates and maintain policies, standards and overall documentation including availability and backup & recovery, service level agreement, baseline architecture, change management, access to production, unsupported HW/SW, security & audit violations and risk acceptance
Scripting and Development: Develops software in several modern languages. Develops database-backed systems and has a solid grasp of DB schema and query performance. Designs horizontally-scalable solutions with innovative use of storage and networking, including solid APIs for integration with other systems. Applies professional standard methodologies in day-to-day work like revision control, unit testing, or other. Applies statistical data analysis techniques
Networking: Recommends or helps architect an entire system, including network design and topology. Authority in understanding and performing TCP dumps, snoop and other network sniffers. Understands and applies knowledge of most protocols (TCP/IP, HTTP, UDP etc.)
Application Technologies (Web Servers, J2EE, Applications Servers): Provides authoritative recommendations and advice to the team and/or department in the areas of web services, OS and storage, including being an active liaison to Development, QA and the Business. Provides scaling, design, costing, troubleshooting and impact analysis consultation
Analyzes systems and makes recommendations to prevent possible problems. Takes lead on issue resolution activities using knowledge of complex and company-wide systems
Leads end-to end audit of monitors and alarms based on subsystem knowledge. Takes the lead on defining the requirements for new tools required for operations
Applies time management and project management skills to lead the resolution of issues in a timely and organized manner, effectively communicating necessary information. May consult directly with clients or third party vendors; provides domain expertise
Regular, consistent and punctual attendance
Preferred
Strong scripting experience with Python (preferred), and Shell (secondary)
Knowledge/experience in development programming languages such as Java, Go, Nodejs, html, CSS, bootstrap etc. is a plus
Knowledge/experience on AI/ML is a big plus
Company
CatchProbe Intelligence Technologies
CatchProbe is the world’s first AI-driven, SaaS-based, centralized, and autonomous intelligence platform — IntelligencySuite (version: IDA Mountain) — delivering actionable intelligence solutions across Web Intelligence, OSINT, Deception Systems, Threat Intelligence, and Digital Crime Analytics.
51-200 employees
Funding
Current Stage
Growth StageTotal Funding
$5MKey Investors
Applied Value
2022-08-01Pre Seed· $1.5M
2021-10-01Pre Seed· $1.5M
2021-02-07Pre Seed· $1M
Company data provided by crunchbase