Project Overview:
The client is hiring a Data Engineer with AirFlow expertise. The candidate needs to have Data Engineering experience, be and AirFlow expert, and have senior skills.
Project Responsibilities:
Re-platform their current data ingestion processes from their traditional inhouse to cloud native setting. Also, re-organizing the system to reuse the components and building custom DAGs.
Must haves
• AirFlow
• Python
• GCP Cloud services - Big Query, DataFlow, Cloud Composer
• experience with configuration driven ETL data flow pipelines and scheduling, Apache Spark
• Experience with DevOps: CircleCI
• Stream processing, Batch processing
Nice-to-haves:
• Data Warehousing
• BI tooling (Looker, PowerBI, etc)
Expected length of the engagement:
• 12 months (Extension)
Location/Overlap requirements:
• Worldwide - Good overlap with Central Timezone (5 hrs), the client is not considering talents from Asia, because it would be too late in their timezone.
Client Budget: $7000 - $8000 per month