Scaling Model Evaluation at Pinterest with Ray (Part 4)
Pinterest's feature-importance and multi-experiment evaluation used to run Ray inference, write predictions to S3, then start a separate metrics job that reread everything. They defined a ReducibleMetric contract (dataset_preprocessor, compute_partials, reduce_partials) so MAE, threshold-based ROC-AUC, and group-wise Precision@K can all compute from batches or segment-preserving buckets, then fused metric reduction into the Ray inference pass via the shared object store. On a Pin Ranking Model, fusion cut end-to-end wall-clock time by up to 32% (25–32% at 100 experiments across ≈30K–3M row sets) and kept Large-set runtime roughly linear (~43 min fixed plus ~41s per experiment up to 300), absorbing a separate ~11s-per-experiment metrics stage.
- Published
- 2026-11-04
- Scale
- up to 32% faster end-to-end evaluation, Large eval set ≈3M rows (also ≈30K / ≈300K), at 100 experiments, fusion cuts wall-clock 25–32%, fused Large runtime ~43 min fixed + ~41s/experiment as experiments scale to 300
- Techniques
- fused inference and metrics reducible metrics batch inference feature importance
- Stack
- Ray Apache Spark S3