Lead Data Systems Engineer – Cybersecurity (Data Engineering Focus)
Location: New York, NY (Hybrid – 3 days onsite)
Duration: 12 months (long-term program)
Interview: Single stage (Technical + Behavioural)
We’re supporting a major Financial Services organisation building out its Cybersecurity Data Platform, a critical function responsible for detecting, analysing, and preventing large-scale digital threats.
This is not a standard Data Engineering role. You will operate at the intersection of high-volume data processing and machine learning, building the core infrastructure that enables real-time threat detection.
The role is weighted 80 percent towards Data Engineering and 20 percent towards Machine Learning integration. It is suited to someone who is strong on fundamentals and experienced in moving and processing large-scale data within high-throughput environments.
What You’ll Be Doing
- Designing and building production-grade data pipelines for high-volume security telemetry
- Developing near real-time ingestion frameworks using Databricks
- Transforming raw data into ML-ready datasets for threat detection models
- Working closely with ML teams to support GenAI-driven security use cases
- Optimising pipelines for performance, scalability, and reliability
- Handling both structured and semi-structured data at scale
What We’re Looking For
- Senior Data Engineering experience in large-scale environments
- Strong Python expertise (Pandas / DataFrames)
- Deep experience with PySpark / Apache Spark
- Hands-on experience with Databricks + AWS (S3, Glue, EMR, Lambda)
- Proven track record working with high-throughput streaming (Kafka)
- Strong understanding of data modelling (SCD, pipeline automation, scalable design)
- Experience building pipelines beyond basic batch processing – true real-time or near real-time systems
Darwin Recruitment is acting as an Employment Business in relation to this vacancy.
