Staff Platform Engineer - Big Data Platform Management

Requirements (Minimum Qualifications)

• A degree or professional qualification in Computer Science, Engineering, or a related field 
• At least 8 years of experience in managing Big Data Platforms. Proven experience with technologies like Spark, Iceberg, Trino, Elastic Stack, NiFi, Kafka, Airflow.
• Expertise in Big Data Engineering, especially in building streaming/batch pipelines, architecting data warehouses, and deploying AI/ML infrastructure.
• Capable of managing and utilizing diverse storage protocols, including S3, NFS, SMB, and HDFS. 
• Strong knowledge in securing applications through SSL/TLS encryption, data at rest encryption, and robust authentication/authorization (AuthN/AuthZ) using OIDC/OAuth2. 
• Comprehensive understanding of core networking principles, including firewall configuration, DNS management, routing and subnetting. 
• Experience with scripting languages, such as Bash, Python, Scala and Java. 
• Proficient in Kubernetes orchestration, leveraging its ecosystem for automated scaling, self-healing, and seamless CI/CD integration. 
• Experience in supporting search workloads in multi-terabytes and billions of records 
• Experience with agile development methodologies and version control systems, such as Git.
• Experience in leading engineering teams.

Why join us?

• The work is purposeful and meaningful 
• You will work with the best engineers
• We work with modern technologies and tech stacks
• We have excellent engineering culture and work-life balance 
• We aspire to engineering and operational excellence 
• We empower to innovate
• We grow together as a family 

Responsibilities

Design, implement and manage the Big Data Infrastructure platform, including Hadoop, Spark, NoSQL databases, and data lakes.
Manage Big Data clusters, including monitoring, troubleshooting, and optimizing cluster performance, as well as managing job scheduling and resource allocation.
Design and implement storage solutions for Big Data workloads, including HDFS, S3, and other object storage systems.
Ensure the security and governance of the Big Data Infrastructure platform, including managing access controls, implementing data encryption, and conducting regular security audits.
Collaborate with data engineers and data scientists to optimize Big Data application performance and provide technical support and guidance on using the Big Data Infrastructure platform.
Plan and manage Big Data resource capacity, including forecasting, procurement, and deployment of new hardware and software.

Nice to Have

• Certifications in Big Data, such as Hadoop Certified Developer or Spark Certified Developer.
• Experience in managing physical and virtual infrastructures.
• knowledgeable in deploying and scaling LLM-based and agentic AI/ML applications.