Phase: Initial phase (foundational)
Required experience: 7+ years in data engineering, building and operating production data pipelines.
Role summary
Builds the end-to-end data pipelines that bring every source into the platform reliably. Responsible for source integrations, ingestion, transformation, and data processing that feed the lakehouse and, later, the graph and ML layers. This is the core delivery role for the initial pilot use cases.
Key responsibilities
- Build and operate end-to-end data pipelines from source systems into the lakehouse / data fabric.
- Integrate structured, semi-structured, document, and time-series sources into a common platform.
- Implement data processing, cleansing, and transformation for the priority pilot use cases.
- Ensure pipeline reliability, monitoring, and data quality across all feeds.
- Work with the architect to align pipelines to platform standards and modeling conventions.
- Prepare curated datasets for the visualization and MLOps workstreams.
Must-have skills and experience
- Strong data engineering background building production pipelines end to end.
- Hands-on experience with Azure data services and readiness to work with Snowflake.
- Experience integrating diverse sources: Oracle ERP, MongoDB, PostgreSQL, Cassandra, Redis, InfluxDB, and time-series data.
- Solid data processing skills (batch and streaming) and strong SQL.
- Experience with object and file storage such as MinIO / NFS.
- Data quality, testing, and pipeline observability practices.
Nice to have
- Snowflake production experience.
- Exposure to OT / IoT data feeds.
- Familiarity with orchestration frameworks and infrastructure-as-code.
Relevant stack
Azure Data Factory / Synapse, Snowflake, Oracle ERP, MongoDB, PostgreSQL, Cassandra, Redis, InfluxDB, time-series sources, MinIO / NFS, APIs.
General attributes
- Proactive and self-driven, able to take ownership and move work forward without waiting to be told.
- AI-enabled in day-to-day work, comfortable using AI tools and copilots to accelerate delivery and quality.
- Strong self-learner who stays current with evolving tools, platforms, and practices.
- Good team player who collaborates well across engineering, operations, and stakeholder groups.
Required Skills
Broad source integration
Data quality, testing, pipeline observability
Azure data services
OT/IoT exposure
Snowflake production experience
Object/file storage — MinIO/NFS
