DNI (Delaware Nation Industries) ยท 6 hours ago
Databricks Developer with Security Clearance
Atlanta, GA
Full-time
Remote
Mid Level
3+ years exp
Maximize your interview chancesBusiness DevelopmentConsulting
No H1B
Security Clearance Required
Insider Connection @DNI (Delaware Nation Industries)
Get 3x more responses when you reach out via email instead of LinkedIn.
Responsibilities
Design, develop, and maintain data pipelines using Databricks with PySpark and Spark SQL to process large-scale datasets on the Azure platform.
Write PySpark code and SQL queries for efficient data transformation, manipulation, and analysis.
Experience writing to Delta Lake Storage for data persistence, ensuring that data is processed and stored in an optimized, performant manner.
Collaborate with cross-functional teams to understand business requirements and translate them into scalable data pipeline solutions.
Work with Azure Synapse for data warehousing and analytical processing in a cloud environment, optimizing performance and ensuring data accessibility.
Work closely with DevOps teams to implement CI/CD pipelines for automated deployment and integration of data pipelines.
Ensure best practices for data governance, security, and performance optimization are followed when designing and maintaining data pipelines.
Troubleshoot, debug, and optimize complex data pipelines to ensure smooth operation and minimal downtime.
Provide technical guidance to team members and participate in code reviews to maintain high coding standards and quality.
Collaborate with stakeholders to deliver high-quality, data-driven solutions that meet business objectives.
Qualification
Find out how your skills align with this job's requirements. If anything seems off, you can easily click on the tags to select or unselect skills to reflect your actual expertise.
Required
3-5 years of experience as a Databricks Developer or similar role with a strong focus on PySpark coding and SQL development.
Proficient in PySpark and SQL for data transformation, querying, and analysis.
Experience writing to Delta Lake Storage and understanding how to optimize the performance of data stored in Delta format.
Strong experience with Spark SQL for querying and managing large datasets in a distributed computing environment.
Experience working with Azure Synapse for cloud-based data warehousing and analytics.
Solid understanding of DevOps principles and experience with CI/CD pipelines for automated deployment using tools such as Azure DevOps or similar.
Strong problem-solving skills and the ability to work independently as well as part of a team.
Excellent communication skills to effectively collaborate with stakeholders at various levels of the organization.
Bachelor's degree in Computer Science, Information Technology, Data Engineering, or a related field (or equivalent work experience).
Preferred
Prior exposure to healthcare or scientific domains is a plus.
Familiarity with Azure Data Factory for orchestrating data workflows in the cloud.
Exposure to other big data technologies such as Apache Hadoop or Apache Flink.
Experience with data governance tools and techniques, including data lineage, auditing, and security best practices in cloud environments.
Familiarity with containerization technologies such as Docker or Kubernetes.
Company
DNI (Delaware Nation Industries)
DNI (Delaware Nation Industries) is the parent organization for all Information Technology focused businesses owned by the Delaware Nation.
201-500 employees
Funding
Current Stage
Growth StageLeadership Team
Jerry Kennedy
Chief Team Servant (aka President & CEO)
Jay Graham
President
Recent News
2024-02-18
Company data provided by crunchbase
