Position Summary
We are seeking a Data Engineer with a strong focus on security data and threat intelligence. In this role, you will design, develop, and optimize data pipelines, ensuring seamless data ingestion, transformation, and storage of security-related datasets roughly Petabyte/day. Collaborating with cross-functional teams-such as security analysts, DevOps, and data scientists-you will play a pivotal role in driving data-driven security solutions and mitigating organizational risks.
Responsibilities
- Data Pipeline Development: Design, build, and maintain scalable, reliable pipelines to handle large volumes of security data from various sources (e.g., logs, threat feeds, SIEM systems).
- Data Architecture & Modeling: Develop data models and database structures to efficiently store and manage threat intelligence, access logs, and event data.
- ETL/ELT Processes: Implement ETL/ELT processes that guarantee high-quality data for downstream analytics, machine learning models, and reporting tools.
Security Data Integration: Collaborate with security engineers and analysts to integrate new data sources, optimize data transformations, and ensure robust data governance.
- Monitoring & Optimization: Monitor the performance of data pipelines, troubleshoot any issues, and implement optimizations for throughput, latency, and cost.
- Collaboration & Stakeholder Engagement: Work closely with information security, software engineering, and data science teams to define data requirements and deliver scalable solutions.
- Documentation & Best Practices: Maintain comprehensive technical documentation, and champion best practices around data security, privacy, and compliance standards (e.g., GDPR, SOC 2).

- Education: Bachelor's or Master's degree in Computer Science, Information Systems, or related field. Equivalent work experience may be considered.
Technical Expertise:
- Proven experience with Big Data tools (e.g., Hadoop, Spark) and cloud platforms (AWS, Azure, or GCP).
- Proficiency in SQL and at least one programming language such as Python, Java, or Scala.
- Hands-on experience with data pipeline frameworks (e.g., Airflow, Luigi) and message queuing technologies (e.g., Kafka).
- Security Domain Knowledge(nice to have):
Familiarity with security tools (SIEM, IDS/IPS) and frameworks (MITRE ATT&CK).
Experience in processing and analyzing large-scale security event logs or threat intelligence data.
Analytical Mindset: Strong problem-solving skills and the ability to effectively analyze complex security datasets.
Data Governance & Compliance: Understanding of data security, encryption, and privacy regulations.
- Collaboration & Communication: Excellent interpersonal skills, with a track record of successfully working in cross-functional teams and communicating technical concepts to non-technical stakeholders.