We are looking for a Junior Data Engineer to join our team focusing on data pipeline development. You will work on scheduled processes and databases that manage streaming IoT and ML data. Your efforts will contribute to generating insights and data aggregations for NFT minting and supporting projects like EnerGPT and finance service.
Project description: Our customer is dedicated to helping oil and gas companies produce clean energy profitably. By leveraging AI and Computer Vision, they automate Health, Safety, and Environment (HSE), Environmental, Social, and Governance (ESG), and operational processes. =
Technical stack: Python, Airflow, Computer Vision, IoT sensory data, Docker, AWS Aurora, AWS RDS, Lambda, MSK Kafka, Streamlit, Gitlab, Pandas, Matplotlib, and Plotly.
What you’ll need:
- Understanding core principles of data engineering, including data modeling, data warehousing, and data integration techniques
- Experience with ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) processes
- Familiarity with building and maintaining data pipelines
- Knowledge of HTTP methods and RESTful APIs
- Understanding of webhooks for real-time data updates
- Experience with multithreading for performance optimization
- Ability to write and execute unit tests
- Good English and strong soft skills
Nice-to-have skills:
- Previous experience in a startup environment
- Familiarity with Terraform for infrastructure as code
- A degree in Computer Science or a related field is preferred
What you’ll be doing:
- Develop and maintain data pipelines using Python and Apache Airflow
- Implement ETL/ELT processes
- Manage and optimize databases, including AWS S3 and PostgreSQL
- Develop and consume REST APIs using FastAPI
- Perform data processing and analysis with SQL queries
- Collaborate with team members to ensure data integrity and efficient workflows
WE OFFER:
- Opportunity for professional growth and development
- Challenging and interesting work environment with team members based in Ukraine, Poland, the USA, etc.
- You will have the option to work either from our office or remotely
If you are passionate about data engineering and excited to work with cutting-edge technologies in a dynamic environment, we would love to hear from you!