Job Description
We are seeking a skilled Data Engineer to join our team and take ownership of designing, maintaining, and improving our data delivery processes. This role is pivotal to organizing and unlocking internal and vendor datasets, ensuring reliable, accurate, and fast data pipelines to meet our business needs.
You will work alongside an experienced team, including cross-training existing team members for awareness and troubleshooting capabilities, while primarily being responsible for delivering efficient and robust ETL solutions.
Key Responsibilities:
- ETL raw data (primarily CSV and JSON formats) into database tables (MySQL and PostgreSQL) using Airflow, Python, Pandas, Polars, and Linux CLI utilities.
- Create, maintain, and optimize database tables for performance, leveraging indexes, basic partitioning, and other techniques.
- Design, build, and maintain reliable, accurate, and high-performance data pipelines.
- Write and maintain complex SQL queries for data analysis and transformation.
- Work with APIs, Docker, GitHub, and other tools to support data engineering tasks.
- Collaborate with cross-functional teams to meet business objectives related to data delivery and reporting.
- Optionally work with AWS services like S3 to move data in and out of the cloud ecosystem.
Required Skills & Experience:
- Proficiency in Python and Airflow (creating and managing DAGs).
- Experience with ETL processes and working with structured and semi-structured data formats (CSV, JSON).
- Strong SQL skills with the ability to write and optimize complex queries.
- Familiarity with database systems such as MySQL and PostgreSQL.
- Experience with Docker, GitHub, and APIs.
Preferred Qualifications:
- Experience working with AWS services, particularly S3.
- Knowledge of PostGIS and spatial queries (training will be provided if not already familiar).
- Familiarity with basic data pipeline design for datasets under 1TB in size.
Why Join Us?
- Work on meaningful, business-critical data projects with a manageable dataset size.
- Contribute to the foundational development of the data delivery process.
- Opportunities for growth and cross-training with existing team members.
- Flexible and collaborative work environment.
If you are passionate about creating simple yet effective data engineering solutions and enjoy working on projects that directly impact business outcomes, we’d love to hear from you!
