| Chance of the best, rejecting k | |
|---|---|
| Best cutoff, large n | |
| Its chance |
Lesson
The theory β The 37% Rule
In the secretary problem, n candidates arrive in random order, each is compared only with those already seen, and each must be accepted or rejected on the spot, for good. The cutoff rule rejects the first k and then accepts the first candidate better than all of them.
Where the formula comes from
- Split by where the best is. Write B for the position of the best candidate and W for ending with it. B is equally likely to be any of 1 to n.
- A best among the first k is lost. The rule never looks back, so those positions contribute nothing.
- A best at position i > k is reached only if nobody between k + 1 and i β 1 was taken. That happens exactly when the best of the first i β 1 sits among the first k, since only then does nobody in between beat the benchmark. Any of the i β 1 is equally likely to be that one.
- Add over the positions.
- Let n grow. With x = k/n the sum becomes an integral and the chance becomes βx ln x. Its derivative, βln x β 1, is zero at x = 1/e, where the chance is 1/e β 36.79%.
- Assumes
- Only ranks are seen, never scores; the order is uniformly random; a rejected candidate cannot be recalled and an accepted one always says yes; and anything short of the very best counts as a loss.
Problems solved in full
-
Three candidates, every order listed 3 steps
Three candidates arrive in random order. Write each order as the ranks in arrival order (1 = best). Rejecting the first and then taking the first who beats everyone before, in how many of the six orders do you end with the best? Check the answer with the formula.
-
The six orders:
-
The first candidate sets the benchmark. In 213 and 231 the best beats the 2; in 312 the 1 arrives second and is taken. In 123 and 132 nobody beats the 1 that was rejected, so the last is taken; in 321 the 2 beats the 3 and is taken before the 1 arrives:
-
The formula with n = 3 and k = 1:
Answer
3 of the 6 orders, a chance of 1/2, against 1/3 for taking the first candidate.
-
-
Four candidates, every cutoff 2 steps
With four candidates, find the chance of ending with the best for each cutoff k = 0, 1, 2, 3, and the best cutoff.
Show solution
-
k = 0 takes the first candidate, who is the best with chance 1/4. For k β₯ 1 use the formula:
-
Compare the four:
Answer
Reject one, then take the first who beats it: 11/24 β 45.83%.
-
-
Kepler's eleven 3 steps
In 1611 Johannes Kepler set about choosing a second wife and, over nearly two years, considered eleven candidates. With the cutoff rule and n = 11, which cutoff is best, what is its chance, and which candidate is the first the rule can take?
Show solution
-
Evaluate the formula around n/e = 11/e β 4.05:
-
k = 4 is the largest; k = 3 and k = 5 are both lower:
-
With four rejected, the fifth candidate is the first the rule can accept. Kepler did marry the fifth, but not by this rule: Ferguson's account of his letter has him persuaded to propose to the fourth after meeting the fifth, being turned down, and going back. The rule forbids going back.
Answer
Reject four, chance 39.84%; the fifth is the first candidate the rule can take.
-
-
Many candidates: where 1/e comes from 3 steps
For large n, write the share rejected as x = k/n and find the best x and the chance it gives. Then check against n = 100.
Show solution
-
The sum is a Riemann sum for 1/t on [x, 1]:
-
Set the derivative to zero:
-
For n = 100, n/e β 36.79, and the exact best cutoff is 37:
Answer
Reject a share 1/e β 36.79%; the chance tends to 1/e. At n = 100 the best cutoff is 37, with chance 37.1%.
-
-
Left with the last candidate 3 steps
A hundred candidates, the first 37 rejected. Find the chance that the best is among the rejected, the chance that the search reaches the last candidate, and the chance that the last candidate is then the best.
Show solution
-
The best is equally likely to be anywhere, so it is among the first 37 with chance:
-
The search reaches candidate 100 exactly when nobody from 38 to 99 is taken, that is when the best of the first 99 is among the first 37:
-
The difference is the case where the search reaches the end and the last candidate is the best:
Answer
37%, 37.37% and 0.37%: more than one search in three ends with the last candidate, and almost never because the last is the best.
-
-
The lowest expected rank, three candidates 3 steps
Three candidates, and the aim is now the lowest expected rank (1 = best), not the best itself. Find the optimal rule by backward induction and its expected rank.
Show solution
-
A candidate seen i-th whose rank among those seen is j has expected overall rank j(n + 1)/(i + 1). The third candidate must be taken, and has expected rank (n + 1)/2:
-
At the second candidate, take it if its expected rank is no more than 2, otherwise go on:
-
At the first, taking it gives 2, more than 5/3, so go on:
Answer
Pass the first, take the second if it beats the first, otherwise take the third: expected rank 5/3 β 1.67. With three candidates this is exactly the cutoff rule with k = 1.
-
-
Two candidates, scores seen 2 steps
Two candidates have scores drawn independently and uniformly from 0 to 1, and you see each score as it arrives. Find the rule that most often ends with the higher score, and its chance. Compare it with ranks alone.
Show solution
-
If you pass the first, whose score is x, you must take the second, which wins when it is higher: chance 1 β x. If you take the first, it wins when the second is lower: chance x.
-
So take the first when x β₯ 1/2. Average over x:
Answer
Take the first when its score is at least 1/2: chance 3/4. With ranks alone, two candidates give 1/2 whatever you do.
-
Supplementary problems, answers only
-
With five candidates, find the best cutoff and its chance.
Show answer
Reject two: 13/30 β 43.33%.
-
With two candidates and ranks only, show that every rule ends with the best half the time.
Show answer
Taking the first wins with 1/2; passing it means taking the second, also 1/2.
-
With ten candidates at the best cutoff, find the chance the best is among the rejected and the chance the search reaches the last candidate.
Show answer
k = 3: 30% and 3/9 β 33.33%.
-
In the large-n limit, find the chance of the best when a fifth of the candidates are rejected, and when three fifths are.
Show answer
β0.2 ln 0.2 β 32.19% and β0.6 ln 0.6 β 30.65%.
-
With four candidates, find the lowest expected rank any rule can reach.
Show answer
15/8 = 1.875.
Learning path
Choosing on what you have seen
References (4)
- The problem as now studied (n known, random order, relative ranks only, no recall, only the best counts), the cutoff rule and its chance, and the history the page uses: Kepler, after losing his first wife in 1611, spent nearly two years over at least eleven candidates, described in a letter to Baron Strahlendorf of 23 October 1613; friends persuaded him to propose to the fourth after he met the fifth, the fourth turned him down, and he chose the fifth. Ferguson notes that for n = 11 the rule's chance is largest when the fifth is the first candidate it may take, and that Kepler's recall breaks the problem's assumptions: T. S. Ferguson, Who solved the secretary problem? Statistical Science 4(3), 282β289 (1989). doi:10.1214/ss/1177012493
- The classical secretary problem solved by backward induction, the limits the page quotes in insight blocks 2 and 3: the least expected rank tends to 3.8695 (Chow, Moriguti, Robbins and Samuels), and with the values revealed and their distribution known the chance of the best tends to 0.58016 (Gilbert and Mosteller), 'to be compared to the limiting value eβ»ΒΉ when the intrinsic values are not revealed': T. S. Ferguson, Optimal Stopping and Applications, Chapter 2, Finite Horizon Problems, Β§Β§2.1β2.3. Mathematics Department, UCLA (online text).
- The rank-minimising rule of insight block 2 and solved problem 6: accept a candidate whose relative rank is within a threshold that rises as the candidates run out, with expected rank tending to 3.8695: Y. S. Chow, S. Moriguti, H. Robbins and S. M. Samuels, Optimal selection based on relative rank (the "secretary problem"). Israel Journal of Mathematics 2(2), 81β90 (1964). doi:10.1007/BF02759948
- The full-information problem of insight block 3 and solved problem 7: values seen, independent with a known distribution; the optimal rule takes a best-so-far above a threshold that depends on how many remain, and the chance of the best tends to 0.5802: J. P. Gilbert and F. Mosteller, Recognizing the maximum of a sequence. Journal of the American Statistical Association 61(313), 35β73 (1966). doi:10.1080/01621459.1966.10502008