A quota, a decaying curve, and the feature I almost missed

Here's a problem, stripped to its bones. You have a batch of claims you can submit to an external decision-maker. Each claim is worth some amount of money if approved. The decision-maker only approves a limited number per period — a quota — and, crucially, the more you've already spent against that quota, the less likely the next one is to land. You can choose which claims to submit, and in what order. What's the policy that maximizes expected recovery?

It sounds like a tidy textbook exercise. In practice it was a chain of wrong assumptions, one genuinely dangerous data bug, and a final answer that was both simpler and stranger than where I started. This is the walk-through, because the lessons are portable to any constrained-approval or scarce-slot problem.

Wrong assumption #1: I could defer claims to a fresh budget

My first instinct was obvious: if the quota resets each period, then hold the low-value claims and submit them next period against a fresh budget. Spread the load, waste nothing. Clean.

The data killed it flat. The quota didn't attach to when I submitted — it attached to the period each claim inherently belonged to. Deferring a claim didn't move it to next period's budget; it just spent the same budget later, after the easy wins were gone. An entire class of 'smart scheduling' strategies evaporated the moment I understood that the budget follows the item, not the action. Assumption one, dead — and it would have quietly cost money if I'd shipped it.

Before you optimize the policy, make sure you actually understand what the constraint is attached to. I'd have built an elegant scheduler for a rule that didn't exist.

The join that lied to me

To learn the true shape of the approval curve, I had to join two datasets: the record of what I'd submitted, and the record of what came back approved or denied. I joined them on what looked like the obvious identifier — a short ID present in both — and got a clean-looking result.

It was clean-looking and wrong. That short ID wasn't unique; it collided across different records, so the join silently stitched together rows that had nothing to do with each other. Every downstream number was quietly poisoned, and nothing threw an error — the most dangerous kind of bug, because it produces confident, plausible garbage. The fix was to join on the true globally-unique key instead. The moment I did, the numbers changed and the real curve appeared.

A join is an assertion that two keys mean the same thing. If the key isn't unique, the join isn't a join — it's a random number generator with good manners.

The curve, and what it demanded

With an honest join, the approval curve by position-within-budget was brutal and clear. Roughly: the first claim against a period's budget landed ~99% of the time, the second ~96%, the third crashed to ~33%, and everything after sat near a ~5% floor. This isn't a gentle decay you can average over — it's a cliff. The first two slots are almost free money; the rest is almost a coin flip you usually lose.

That shape forces the question: if only the earliest slots are valuable, and my claims are worth wildly different amounts, which claim gets which slot?

The answer is a 100-year-old inequality

This is the rearrangement inequality, and it's the whole game. To maximize the sum of products of two sequences, you pair the largest of one with the largest of the other. Here: pair the biggest-dollar claims with the highest-probability (earliest) slots. Sort claims by value, descending, and pour them into the slots from the top. Same family of result as the classic cμ rule in scheduling — when service is scarce, serve the jobs where value × urgency is highest, first.

slots (by position):   p1 ≈ 0.99   p2 ≈ 0.96   p3 ≈ 0.33   p4+ ≈ 0.05
claims (by value):     $$$   $$   $$   $   ...

optimal: biggest value → earliest slot
  $$$  → p1      (0.99 · $$$   — capture the whale while it's near-certain)
  $$   → p2
  $$   → p3      (marginal; only if EV still beats the effort)
  ...  → p4+     (usually not worth a slot at all)

The trap that nearly inverted my logic

Here's where I almost shot myself in the foot, and it's the subtlest point in the whole thing. I'd built a model to predict the probability a claim gets approved. The tempting move is to sort your claims by that predicted probability and submit the most-likely-to-win first. It feels right. It's backwards.

The model was predicting the probability of a slot — position 1 is ~99% no matter what you put in it — not the desirability of a claim. Sorting claims by that score just re-derives the position curve and tells you nothing about which claim belongs there. Value decides the ordering; the probability curve only tells you how many slots are worth spending at all. Confusing 'how likely is this slot to succeed' with 'how good is this claim' would have had me feeding small claims into the golden early slots because the model happily said they'd win. They would have won — and wasted the slot.

Know what your model is actually predicting. Mine predicted the slot, not the claim — and the entire policy hinges on not confusing the two.

The feature beat the model

When I did model the win probability properly, the interesting result wasn't the algorithm — it was the feature. The single most predictive variable was position within the depleting budget. On its own it carried an AUC around 0.94, comfortably beating the intuitive proxy I'd started with (the item's own timestamp, ~0.90) and the naive 'which submission period' feature (~0.87). A small three-feature model — position, budget remaining, and recent loss streak — hit ~90% accuracy; throwing a gradient-boosted model at it inched to ~92%. With the wrong headline feature, everything had plateaued several points lower no matter how fancy the model got.

That's the meta-lesson, and it's the one I keep relearning: the leverage was almost entirely in framing the right variable, not in the sophistication of the model on top of it. Choosing to measure 'position in a depleting budget' instead of 'time the item was created' moved the needle more than any modeling choice downstream of it. The judgment about what to measure dominated the machinery that measured it.

What travels beyond this problem

• ▸Pin down what the constraint attaches to before optimizing — I nearly built a scheduler for a rule that didn't exist.
• ▸A join on a non-unique key is silent corruption. Verify the key is actually unique, or the analysis is fiction with a spreadsheet's confidence.
• ▸When a success curve is a cliff, not a slope, order is everything — and the rearrangement inequality tells you the order: biggest value into the best slots.
• ▸Know precisely what your model predicts. A score for the slot is not a score for the item, and mixing them up inverts your policy.
• ▸The right feature beats the fancier model. Spend your judgment on what to measure; the algorithm is usually the easy part.

None of the individual pieces here are exotic — a textbook inequality, a join bug, a calibration curve. The work was in the sequence: distrusting the obvious strategy, distrusting the clean join, and distrusting my own model's score long enough to ask what it was really telling me. The math was the easy part. Knowing which math, and which variable, was the job.