Principal Data Scientist - Experimentation & Causal Inference

ByLabs ยท Seattle, WA
Pay ยท estimate~$135k โ€“ $185kMarket rate ยท not set by employer
PostedNov 17, 2026
LevelSenior
In short We are looking for a Data Scientist - Experimentation & Causal Inference to serve as the statistical brain behind our rapidly growing experimentation program.
SQLPYTHON

๐Ÿ“‹ About the job

About the Role

We are looking for a Data Scientist - Experimentation & Causal Inference to serve as the statistical brain behind our rapidly growing experimentation program. You will own the methodology layer that ensures every experiment we run is trustworthy, sensitive, and correctly interpreted.

You will NOT be building infrastructure or writing backend services โ€” our platform engineering team handles that. Your job is to design, validate, and continuously improve the statistical frameworks that sit on top of the platform. You will be the person the team turns to when they ask "Can we trust this result?" โ€” and you will build the automated systems that answer that question before anyone needs to ask.

Team context: You will join a team of data scientists and platform engineers within the Big Data group, reporting to the Head of Big Data. The experimentation platform engineering team implements your specifications into production systems.

Responsibility

  • Guardrail Metrics & SRM Detection
  • Design and maintain automated anomaly detection for live experiments โ€” including Sample Ratio Mismatch (SRM) checks and traffic split validation.
  • Define alerting thresholds and circuit-breaking criteria so compromised experiments are flagged or stopped before polluting decisions.
  • Define and validate guardrail metrics (sensitivity, directionality, interpretability) that protect the business during every experiment.
  • Variance Reduction & Sensitivity
  • Implement and iterate on CUPED and related pre-experiment covariate adjustment methods to reduce metric variance.
  • Develop techniques to remove noise from user historical behavior, enabling faster detection of true treatment effects โ€” especially in limited-traffic or high-priority scenarios.
  • Example of impact we are targeting: Shorten average experiment duration from 14 days to 9 days, unlocking 40%+ more experiments per quarter.
  • Continuous A/A Experiment Monitoring
  • Design and run continuous A/A experiments as an always-on health check for the data pipeline (from client-side event reporting through message queues to the real-time data warehouse).
  • Monitor metric baseline volatility and pipeline stability, ensuring the instrumentation layer remains trustworthy over time. You define what "healthy" looks like; the platform team implements the monitoring infrastructure.

Other responsibilities

  • Partner with product, engineering, and growth teams on experiment design: sample size calculations, metric selecโ€ฆ
๐Ÿ“„ Read the full description โ†’ The rest of this listing is on the hiring partnerโ€™s site.
Interested in this role?Apply in one click โ€” no JobQ account needed.
Apply now โ†’
Principal Data Scientist - Experimentation & Cau...ByLabs
Apply now