Role Title: Lead PySpark Developer
📍 Location: Owings Mills, MD (On-site)
📅 Contract Length: 6 Months
🎓 Education: Bachelor’s Degree Required
🧳 Experience Level: Mid-Senior (7+ years)
🛂 Visa Sponsorship: Not available
📦 Relocation Assistance: Not offered
💼 Industry: Information Technology & Services
💻 Interview Format: Virtual/On-site as required
💰 Rate Per Hour: $68 per hour
Lead PySpark Developer – Build Scalable Cloud Data Solutions in a High-Impact Role
Are you a data engineering leader with deep experience in PySpark and AWS cloud platforms? We're seeking a Lead PySpark Developer to architect, develop, and optimize large-scale, cloud-native big data solutions. You’ll be joining a forward-thinking team focused on innovation, data governance, and performance tuning.
Key Responsibilities
Lead the design, development, and deployment of scalable PySpark-based big data pipelines
Architect and maintain ETL pipelines for both structured and unstructured data
Collaborate with data engineers, scientists, and business teams to translate requirements into technical solutions
Optimize Apache Spark performance using caching, partitioning, and tuning best practices
Ensure data security, compliance, and governance throughout the pipeline lifecycle
Guide junior engineers, review code, and enforce engineering best practices
Leverage tools such as Airflow for workflow scheduling and DBT/AWS Astronomer for data pipeline automation
Champion CI/CD, version control, and DevOps practices in all aspects of data engineering
Key Skills & Experience
✅ 10+ years in big data and distributed computing
✅ 7+ years with AWS Cloud Computing
✅ Proven hands-on experience with:
PySpark, Apache Spark, Python
SQL & NoSQL databases: DB2, PostgreSQL, Snowflake
✅ Strong grasp of data modeling and ETL workflows
✅ Working knowledge of workflow orchestrators like Airflow
✅ Familiarity with DevOps, CI/CD pipelines, and containerization tools like Docker/Kubernetes
✅ Excellent communication and leadership skills to guide teams and collaborate cross-functionally
Nice to Have
Experience with AWS Astronomer and DBT
Exposure to cloud-native data warehousing and real-time data streaming
Strong understanding of enterprise security, compliance, and cost optimization in AWS
Who This Role is For
A lead-level data engineer or big data architect ready to own the full pipeline lifecycle
Someone looking to build impactful data solutions in a highly visible, mission-critical environment
A mentor and team player who thrives in collaborative, fast-paced, and cloud-first settings
Ready to lead in a role where your technical decisions shape enterprise data infrastructure? This position combines deep technical challenge with the opportunity to mentor and lead.