Data Processing Engineer - I/O (Remote, Worldwide)

Imagine working at the very core of data innovation, where your engineering prowess directly shapes how massive datasets are processed for everything from traditional analytics to the next generation of AI. At DataPelago, we're not just iterating on existing solutions; we're building a groundbreaking engine that redefines performance and scale for tools like Apache Spark and Flink. If you're passionate about optimizing the fundamental input and output operations that power these systems, and you thrive on tackling complex, high-impact challenges, then this remote Data Processing Engineer I/O role is an extraordinary chance to leave your mark on the future of data.

Overview

DataPelago is revolutionizing data processing for both traditional analytics and cutting-edge GenAI preprocessing. We are creating an innovative data processing engine that is transforming how major platforms like Apache Spark, Apache Flink, and Ray handle diverse, large-scale data. Our team drives advances in hardware-accelerated computing, parallel processing, query optimization, distributed systems, machine learning, and cloud-native computing. As a Data Processing Engineer I/O, you will play a crucial role in enhancing the data read and write capabilities of our engine. This is a fully remote, worldwide opportunity, allowing you to contribute from wherever you are.

Key Responsibilities

As a key individual contributor, you will advance the read and write functions of DataPelago’s data processing engine.
You will enhance the functional breadth of our engine, supporting a wider array of data types, sources, and sinks.
Drive significant improvements in the performance and scalability of data ingress and egress operations.
Ensure the highest levels of reliability for our engine when processing large scale data.
Collaborate with cross-functional teams to integrate new I/O features and optimize existing ones.
Research and implement innovative solutions for efficiently handling diverse data formats and storage systems.

Requirements

Strong experience in designing and implementing high-performance data processing systems.
Demonstrated expertise in optimizing data read and write operations from various sources and to various sinks.
Proficiency in one or more programming languages commonly used in data processing, such as Java, Scala, C++, or Go.
Solid understanding of distributed systems concepts and architectures.
Familiarity with various data formats including Parquet, ORC, Avro, CSV, and JSON.
Experience with cloud platforms like AWS, Azure, or GCP and their storage services.
A proven track record of solving complex technical problems and delivering robust, scalable solutions.
Excellent communication skills and the ability to work effectively in a globally distributed team.

What You'll Gain

An unparalleled opportunity to make a significant impact on a category-defining product at the forefront of data processing innovation.
The chance to work on cutting-edge technologies including hardware-accelerated computing, distributed systems, and machine learning.
A collaborative and supportive remote work environment with a team of brilliant engineers.
Extensive opportunities for professional growth and skill development as you tackle challenging and rewarding projects.
The satisfaction of shaping the future of accelerated data processing.