Databricks Data Engineer – Sales Incentive Compensation
📍 Location: New York, New York
🏢 Work Arrangement: Hybrid
📄 Contract Length: Six Months
💰 Hourly Rate: Approximately $75 per hour
💼 Employment Type: Contract
🚫 Visa Sponsorship: Not Available
🚚 Relocation Assistance: Not Available
🏦 Industry: Insurance and Financial Services
Build the Data Infrastructure Behind Enterprise Sales Compensation
A leading global organization is seeking an experienced Databricks Data Engineer to design, develop and optimize enterprise data solutions supporting analytics, performance reporting and Sales Incentive Compensation.
This is not purely a pipeline-development position. The successful candidate will combine strong Databricks engineering expertise with an understanding of incentive compensation processes, including quota allocation, attainment calculations, commission processing and payout reporting.
You will work closely with Sales Operations, Finance, Business Intelligence and technology teams to translate complex business rules into scalable, accurate and auditable data solutions.
Why This Opportunity Stands Out
This opportunity will allow you to:
Own the design and development of enterprise Databricks solutions
Influence the wider Databricks architecture and technical roadmap
Build data pipelines supporting business-critical compensation processes
Work with Delta Lake, Delta Live Tables, Unity Catalog and Databricks Workflows
Translate complex incentive-plan rules into reliable transformation logic
Improve automation, integration, data quality and platform performance
Partner directly with senior Finance, Sales Operations and technology stakeholders
Deliver solutions that directly influence commission and compensation accuracy
What You’ll Be Doing
Databricks Architecture and Engineering
Design, develop and implement scalable Databricks-based data solutions
Build and maintain data pipelines supporting ingestion, transformation, reporting and analytics
Develop data models, Delta Lake tables, notebooks and reporting datasets
Take ownership of the overall Databricks architecture
Ensure new pipelines and enhancements integrate effectively with existing workflows
Maintain and optimize Databricks workspaces, pipelines and orchestration processes
Identify opportunities to improve automation, performance and data integration
Design reliable ETL and ELT processes across multiple enterprise data sources
Sales Incentive Compensation
Build data pipelines supporting Sales Incentive Compensation processes
Translate incentive-plan rules into accurate and auditable data transformations
Support quota allocation and quota-management processes
Develop logic for attainment tracking and performance calculations
Support commission calculations and payout processing
Produce reliable datasets for compensation reporting and reconciliation
Work closely with Sales Operations and Finance to understand plan structures and business rules
Ensure calculation logic is transparent, documented and capable of being audited
Integrate incentive-compensation platforms with enterprise data and analytics solutions
Performance and Data Quality
Test, debug and optimize Databricks pipelines and notebooks
Investigate and resolve data-quality, performance and integration issues
Improve the reliability and efficiency of existing workflows
Implement appropriate controls and validation processes
Monitor data pipelines and troubleshoot failures or inconsistencies
Deliver a high-quality experience for analysts and other data consumers
Stakeholder and Technical Collaboration
Work with business stakeholders to understand complex data requirements
Translate functional requirements into scalable technical architectures
Challenge requirements constructively when a more effective solution is available
Partner with Finance, Sales Operations, IT and Business Intelligence teams
Provide technical support to analysts and end users
Communicate technical concepts clearly to both technical and non-technical audiences
Manage multiple priorities within a fast-paced enterprise environment
Documentation and Platform Development
Document data architectures, models, pipelines and configuration decisions
Produce clear technical specifications and development documentation
Maintain documentation to support knowledge transfer and long-term platform ownership
Stay current with Databricks capabilities and platform enhancements
Identify opportunities to use features such as Unity Catalog, Delta Live Tables and Photon
What You’ll Bring
Essential Databricks and Data Engineering Experience
At least seven years of professional data-engineering experience
A minimum of three years of hands-on Databricks experience
Extensive experience building and managing production Databricks solutions
Strong knowledge of:
Delta Lake
Delta Live Tables
Unity Catalog
Databricks Workflows
Databricks notebooks and workspaces
Advanced proficiency in Python and/or Scala
Strong SQL development expertise
Experience with data ingestion, transformation, ETL/ELT and data modelling
Experience designing and orchestrating enterprise data pipelines
Strong understanding of data quality, scalability and performance optimization
Sales Incentive Compensation Experience
Functional and technical experience with Sales Incentive Compensation processes
Understanding of incentive-plan design and compensation rules
Experience supporting quota allocation and management
Knowledge of sales-attainment calculations
Experience with commission calculation and payout processing
Ability to model complex incentive rules within data pipelines and transformation logic
Experience partnering with Sales Operations and Finance stakeholders
Cloud and Integration Experience
Experience integrating Databricks with at least one major cloud platform:
Microsoft Azure
Amazon Web Services
Google Cloud Platform
Experience working with integration and ingestion technologies such as:
Azure Data Factory
Azure Event Hubs
Apache Kafka
Comparable cloud-based data-integration tools
Ability to create holistic, end-to-end data solutions rather than isolated pipelines
Professional Capabilities
Strong stakeholder-management and consulting skills
Ability to translate complex business requirements into scalable data architectures
Excellent analytical and problem-solving capabilities
Strong written and verbal communication skills
Ability to work effectively with cross-functional teams
Resourceful and comfortable working with minimal direction
Strong technical-documentation skills
Ability to manage competing priorities and deadlines
Bachelor’s degree in Computer Science, Data Engineering, Information Technology or a related discipline
Preferred Experience
Experience integrating Databricks with platforms such as:
Anaplan
Oracle Incentive Compensation
Varicent
Comparable Sales Performance Management platforms
Databricks Certified Data Engineer Associate certification
Databricks Certified Data Engineer Professional certification
Experience using Photon to improve query and workload performance
Previous experience within insurance, financial services or another large regulated enterprise
Experience delivering auditable financial or compensation-related data solutions
The Ideal Candidate
The ideal candidate will be an experienced Databricks engineer who understands that compensation data requires exceptional accuracy, traceability and control.
You will be comfortable moving between detailed technical development and strategic conversations with Finance, Sales Operations and technology leaders. You should be willing to challenge requirements constructively, propose stronger architectural solutions and take ownership of delivery from initial design through testing, deployment and production support.
Most importantly, you will be able to combine enterprise data-engineering expertise with a practical understanding of how incentive plans, quotas, attainment calculations, commissions and payouts operate.
Contract Details
Initial Contract: Six months
Hourly Rate: Approximately $75 per hour
Location: New York, New York
Working Arrangement: Hybrid
Visa Sponsorship: Not available
Relocation Assistance: Not available
Ready to Build Business-Critical Data Solutions?
If you have deep Databricks expertise and experience translating complex Sales Incentive Compensation processes into scalable, accurate and auditable data solutions, this position offers the opportunity to make an immediate impact within a major enterprise environment.