Which pages should an SEO editor review first?

1Abstract

2The problem: a to-do list 195× longer than the day

Where this comes from

FlyRank treats content as infrastructure: it researches, writes and publishes content directly into a client's website, then watches the search data and optimises — the whole cycle run by algorithms rather than by people working page by page. That model runs into one persistent difficulty. Content that gets found in search decays quietly. A page ranks, holds, and then slips: positions drift down, clicks fall away, and because nothing breaks visibly, most teams notice only after the traffic is gone. Across thousands of pages per client, the valuable decision is not is this page healthy? — it is which page should a human fix first?

FlyRank's product already answers a version of that question today, using hand-written threshold rules: a health score, quick-win tags, needs-attention flags. If-this-then-that, with the numbers chosen by hand. They work, and they run in production. But rules of that shape run out exactly where the signals get many, tangled, and shifting — and at the time of this study no trained model had replaced them. This capstone lives in that gap: it asks, on real production search data, whether a learned ranking can order the candidate pool better than the rule that produced it.

One consequence of that framing shaped the whole study, and it is worth stating plainly: product flags are the thing to beat, never a thing to learn from. A health score or a needs-attention tag encodes a decision someone has already made. Feeding it in as a feature would produce a model that predicts the company's own past choices and calls the result a finding. So the hand-written rule appears in this paper exclusively as the baseline, and every column that carries a trace of a prior editorial decision is in the exclusion table in §3.

What that looks like on a Monday morning

An editor sits down with time to properly review about fifty pages. The pipeline hands them a list of 9,759 — every page that clears a fixed threshold of at least 500 impressions, a visible position between 1 and 20, and a click-through rate under 0.5%. Each of those pages genuinely qualifies. The list is still useless, because it is roughly 195 times longer than the week.

That gap is the entire problem, and it is a ranking problem rather than a classification one. Nobody needs a correct yes/no verdict on all 12,023 items; they need a trustworthy ordering of the fifty that will actually be opened. So quality is measured at the top of the list — precision@50 — and that metric was named before any model existed, so it could not be shopped for afterwards.

A fixed rule cannot produce that ordering, because the same CTR number means opposite things depending on where the page sits. A 0.10% click-through rate is above the median for a page ranking on results page three, and less than half the norm for a page sitting at position four. In this dataset, at the same page-one position, comparison articles averaged ~0.14% CTR against ~0.35% for keyword articles — a 2.5× difference that position alone does not explain. Hand-writing thresholds for every (position tier × content type × volume bucket) cell means dozens of numbers, each drifting as the inventory changes. That is the shape of problem where a learned ranking earns its place — conditionally, and only if it beats the transparent rule on held-out clients.

3Data

Everything below comes from the FlyRank ML Internship dataset, a pseudonymised release of roughly 79 million rows of production search and engagement data. Three surfaces of it are involved, and they are kept distinct on purpose.

Population, and the grain

One row is one pseudonymised content item for one client, described by its trailing 90-day search behaviour. Not a page-day, not a client — the editor's decision happens at the content-item level, so the model's rows live there too. The study population is visible pages only: at least 500 trailing-90-day impressions, at a real position between 1 and 20, GSC-measured. That is 12,023 content items across 28 clients, 40.1% of the starter release. One client alone accounts for 36.6% of those rows, a fact that shapes every uncertainty estimate later in this paper.

What was excluded, and why

Exclusions are the part of a data section that is worth reading, so here is the whole list.

One structural feature of the inventory is worth showing before any modelling, because it is the pattern the queue is built to exploit and it is visible without a model at all.

4Methodology

Assumptions, stated so they can be attacked

1. Review capacity is roughly fifty pages per cycle, so quality is measured at the top of the list, not averaged over 12,023 rows nobody will open.
2. Position-conditioned comparison is mandatory. A 0.3% CTR is ordinary at position 40 and alarming at position 3; every CTR claim here is conditioned on position tier.
3. Content items within a client share a site, a template and an audience, so they are not independent draws. Splits group by client, and uncertainty is clustered by client.
4. The queue orders review attention. It has never been validated against the outcome of actually refreshing anything, so "worth looking at first" is the only verb it earns.

The label, and its honest name

The target is is_declining_label: the item's trailing 30-day impressions fell versus the previous 30 days. Base rate 0.599 on this population. This is a proxy, and its weakness is named in the limitations rather than buried: it records a current-window movement, not a future outcome. The genuine past → future design — features from one fortnight, outcome observed in the next — exists separately, on the warehouse, and is reported in the results.

The label was also deliberately not defined by me. An early version of this project would have scored each page's CTR gap against its position peers and then called anything past a threshold an "underperformer". Training on that would teach a model to reproduce my own threshold and call the result a finding. The CTR gap is used as a transparent score in the baseline; the trained label comes from movement the world produced.

The baseline the model has to beat

A transparent hand-written rule: ctr_gap × log(impressions) × confidence — how far below the position-tier median an item's CTR sits, weighted by volume leverage, because a small CTR improvement on a high-traffic page yields more absolute clicks. No fitted weights, one reason code, fully explainable to a non-technical editor. Its expected-CTR lookup table (position tier × content type median) is refit on training rows only, so the rule never peeks at test data either. If the model cannot beat this on the same rows and the same metric, the rule ships.

The model, chosen by search rather than by taste

Sixteen configurations — four feature sets crossed with four estimators — were compared under nested cross-validation. Two things came out of that search, and the second is more interesting than the first.

The end-to-end ROC AUC spread across all sixteen configurations was only 0.067, and the inner folds selected four distinct configurations across five outer folds. In other words the choice of model is not stable, which is itself a result: on this problem the feature set matters more than the estimator, and neither matters very much. That is the argument for shipping the smallest thing that works.

The scorer I recommend — called shipped throughout this paper as shorthand for "the one this study selects", not "running in production", which nothing here is — is a logistic regression on eight features: six numeric — ctr, log_clicks_90d, days_with_sessions, log_impressions_90d, content_age_days, ctr_gap — and two categorical, position_tier and content_type. The Week-5 model used eleven features, two of which (avg_position, days_since_last_update) were later measured at 0.502 and 0.506 single-feature AUC. Those are coin flips, and they are gone.

Validation, in three layers

• Client-grouped 5-fold. Whole clients are held out, so the model is always scored on clients it has never seen. The same pipeline was also run on random folds, and the gap between the two schemes measures memorisation directly.
• Client-clustered bootstrap (2,000 resamples). Every comparison gets both a row-resampled interval and a client-resampled one. Where they disagree, the client interval wins and the claim is retracted.
• Time-aware 2×2 on the warehouse: new clients × later period, so the cost of unseen clients and the cost of elapsed time are separated instead of confounded.

Leakage checks that were actually run

• A planted label-derived feature drove AUC to 0.993 — proof the harness can see leakage when it is there. Then it was removed.
• A planted overlapping-window feature (a 61-day CTR spanning the outcome window) got the same treatment.
• Single-feature AUC on every survivor: the strongest is ctr at 0.598, the weakest avg_position at 0.502. Nothing is remotely in the range of the planted leak. This replaced a Pearson correlation scan, which ordered the features differently and would have missed a threshold-shaped leak entirely.
• Permutation importance on held-out rows replaced impurity importance, which had over-rated scroll_rate — the same methodological correction I had asked of someone else's chart, applied to my own.
• Adversarial validation AUC = 0.961. Clients are almost perfectly separable from one another. This is the number that makes a random row split indefensible on this data.
• Categorical proxy screen — and the one result I kept anyway. Each candidate categorical was scored on how well it alone predicts which client a row belongs to. content_type scores 0.913, and it is one of the two categorical features in the shipped model. That is high: it means content type is close to a partial stand-in for client identity in this inventory, because different clients publish different mixes of content. I kept it, because it is a genuine attribute of the page rather than an identifier, and dropping every feature correlated with client would leave almost nothing. The honest position is that this is a disclosed trade-off, not a clean result — and it is part of why the headline number is a client-clustered interval rather than a point estimate.

5Results

(a) Model versus baseline, same rows, same folds

Every ranker below was scored out-of-fold on client-grouped folds, against a base rate of 0.599. Lift is precision@K divided by the base rate: a lift of 1.00 means "no better than reading the lane in any order at all".

(b) What the honest split cost

The same pipeline, the same rows, the same estimators, five folds each — the only thing that changes is how rows are assigned to folds.

(c) The result worth carrying: the simpler model travels better

Trained on one fortnight, tested on the next, with unseen clients and later period separated into their own cells. This is the only genuinely past → future evaluation in the study, and it changes which model you would deploy.

(d) What the model leans on, and where it is wrong

The importances below are permutation importances measured on the six held-out clients the model never saw, with one-hot blocks permuted together. That matters: an importance ranking taken on rows the model trained on answers a different question. They are measured on the Week-5 eleven-feature set, because that is the set the error analysis was run on.

Reading it in plain words: click-through rate is the model's main signal, content age is second, and volume is third. Everything below the top three is close to noise — positions 7 through 11 all sit under 0.004 AUC drop, which is the same scale as their own standard deviations. This is also the check that corrects an earlier mistake of mine: Week 5 read scroll_rate as the top feature using impurity importance, which is biased toward high-cardinality continuous variables. On held-out rows it ranks fourth, and ctr takes first. Two of eleven features moved by two or more ranks between the two methods.

Is ctr too important? The drop test

When one feature dominates, the honest question is whether it is a leak. Refitting without ctr moves precision@50 from 0.820 to 0.700 and ROC AUC from 0.656 to 0.630 — the largest single-feature effect in the set, and it lands at the lower edge of the full model's confidence interval. "Inside the interval" is a weak pass when the interval is 0.22 wide, so I am not concluding ctr doesn't matter. It clearly does.

What the test does rule out is the thing it was built to rule out. A leaked feature has two signatures and ctr has neither: it would rank near-perfectly alone (ctr alone scores 0.598, and nothing in this feature set exceeds 0.60), and removing it would collapse the model toward the base rate of 0.537 (without it the model still ranks at 0.700, a lift of 1.30×). A strong legitimate feature and a leaked one both hurt when dropped; only the leak scores near 1.0 on its own.

Where it is wrong — the 9 misses

On the six held-out clients the top-50 queue is 41 right, 9 wrong. The interesting part is what the wrong picks look like, compared against both reference classes — against the test-set average alone it is easy to describe a pattern that is really just "top of the queue".

The surprise worth keeping

The clearest directional finding in this study is one I did not expect and did not go looking for: decline rate falls monotonically as content ages — 0.711 in the 31–90 day tier down to 0.494 past a year, against a 0.599 base rate (chart in §3). The intuition that old pages are the decaying ones is backwards in this inventory. A plausible reading is survivorship in the ordinary sense — pages that have held their position for a year have already proven they can — and the effect direction is unchanged on all 30,000 rows before filtering, so it is not an artifact of the visible-pages filter. It remains an observed association on one inventory, with no causal claim attached.

6Limitations & honest framing

1. Two claims were retracted, and here they are

Every comparison was bootstrapped twice. Resampling rows assumes 12,023 independent draws; there are really 28 clients, one of which is 36.6% of the data. Resampling clients instead widens every interval, and two results that looked measurable stopped being measurable.

2. The label is a proxy, and a current-window one

is_declining_label records that trailing 30-day impressions fell against the prior 30 days. It is not "this page will decline", and it is emphatically not "refreshing this page will help". In the warehouse fortnight-to-fortnight design, the base rate moved from 0.349 to 0.633 between two consecutive fortnights — a shift larger than every model-versus-baseline gap in this entire study. Any deployment must measure its own base rate before trusting a precision number.

3. Two shipped features drift by construction

days_since_update shows a population stability index of 8.49 between fortnights, and content_age_days 0.177, against a retrain trigger of 0.25. These features encode "how long since a timestamp", so they drift as the calendar moves, whatever the content does. They are in the shipped feature set. That is a named liability, not an oversight, and it is the first thing a monitoring system here should watch.

4. Twenty-eight clients is a small n, and one is a third of it

The client-clustered intervals above are wide because they should be. With 28 clusters and one dominating, a single unusual client moves the estimate. Every headline in this paper is an interval, not a point.

5. No experiment exists anywhere in this work

Every number is observational and cross-sectional. Nothing here shows that rewriting a title raises click-through rate — that would need a controlled test that was never run. Nothing here observes a search engine's ranking function either; this data sees one inventory's outcomes, never the algorithm that produced them. The queue has never been validated against the result of actually refreshing anything, so "worth looking at first" is the strongest verb it earns.

6. The model is a ranker, not a decision — and it cannot see the page

The scorer sees eight numbers about a page. It has never seen the page. All eight reason codes name something it structurally cannot check — that is why each one has a written reviewer instruction attached rather than an action: stale_90plus means a timestamp moved, not that anything changed; decay_zone_age means old, not stale; thin_for_tier means short, not thin; ctr_gap_vs_peers means the click rate is below the peer group's, not that the snippet actually mismatches the query; striking_distance means position 11–20, not that the gap is realistically closable. A concise definition page is not a thin page, and evergreen content is not dated content. Competitive difficulty and query intent are not in the feature set at all. Every item is a question for a human.

7. Not production

There is no API, no scheduler, no model registry and no automated action anywhere in this work. The deliverable is a ranked queue a person reads, plus the numeric rules for when to stop trusting it.

7Ranked recommendations

Ordered by how much evidence sits behind each one, strongest first, with the confidence stated rather than implied.

1. Review the top 50 each cycle — and know where the queue stops paying (measured)

Precision holds at 0.90 through K=50, 0.88 at K=100, 0.84 at K=400 and 0.79 at K=1,600. But cumulative precision hides the real story; marginal precision by decile shows it. Decile 1 runs at 0.809 (+0.211 over base rate) and decile 6 has already fallen below base rate. Past roughly rank 6,010, working down the queue is worse than reading the inventory in no particular order. Stop there.

2. Ship the eight-feature logistic regression, not the forest (measured, with a stated overlap)

It beats the transparent rule by a margin that survives client clustering, and it degrades at roughly half the forest's rate under time shift. It is not measurably better than the larger Week-5 model — so the honest argument for it is parsimony and stability, not accuracy.

3. Keep the transparent rule running alongside it (measured)

The rule is the revert target, and the trigger is already a number rather than a feeling: if precision@50 falls to 0.672 or below, the rule is matching the model, and the rule is cheaper and explainable to anyone. Below 0.599 — the base rate — retire the queue entirely.

4. Route by archetype, not by score alone (observed, directional)

Position tier crossed with freshness band splits the queue into six archetypes with genuinely different decline rates, and each implies different work. Same score, different job.

5. Trust four reason codes; treat two as context only (observed, directional)

Of the eight reason codes attached to queued items, six sit above base rate — but only four by a margin worth acting on. stale_90plus (+0.023) and striking_distance (+0.019) clear it by less than a third of what the weakest of the four manages, which is inside the noise these buckets carry; treat them as weak signals rather than evidence. The remaining two sit measurably below base rate, and reading those as evidence of risk would be a straightforward mistake.

6. Re-measure the base rate before trusting any precision number (measured)

The observed base rate ranges 0.485–0.747 across folds and 0.349–0.633 across fortnights. A precision@50 of 0.75 is excellent against a base rate of 0.49 and mediocre against 0.63. Report them together, or not at all.

7. Keep a human in front of every item (by design)

The reviewer's job is exactly the part the model cannot do: open the live snippet and judge whether the title genuinely mismatches the query intent; decide whether a page is dated or simply old; decide whether a competitor gap is realistically closable or whether position 11 is the ceiling for that query. Competitive difficulty is not in the feature set. Intent is not in the feature set. The queue orders the questions; the editor answers them.

When to stop trusting the queue

Every threshold below is a measured number, not a round one. A trigger set at "if performance drops a lot" fires never or always.

Back to the content problem

The question this started from was whether a learned ranking can beat a hand-written threshold rule at deciding which decaying page a human should fix first. On this data, the measured answer is yes, but by less than it first appears, and for a different reason than expected.

The gain over the rule is real and survives client-clustered resampling — a content operation reviewing fifty pages a cycle would find roughly 45 genuinely declining pages instead of 34. Across a year of weekly cycles that is a material difference in where editorial time lands. But the gain over a one-line heuristic is not measurable at that budget, and the gain from a bigger feature set is not measurable at all. The honest recommendation to a team in FlyRank's position is therefore narrower than "ship a model": replace the threshold rule with the smallest learned ranking you can defend, keep the rule live as the revert target, and spend the saved effort on the label rather than on the estimator.

That last point is where the study actually landed. The largest number in this entire investigation is not a model gap — it is the label's own base rate moving 0.349 to 0.633 between two consecutive fortnights, and a date-anchored feature drifting at PSI 8.49. For content decay specifically, when you measure moves the answer more than what you fit to it. A team investing here would get more from defining decay on a stable future window, and from running one controlled test of whether refreshing a queued page changes its trajectory, than from any further modelling. No such test exists in this work, which is why nothing in this paper claims a refresh causes recovery — only that these pages are the ones worth opening first.

8Reproducibility

Everything on this page traces to a committed file. Random seed 42 throughout; scikit-learn 1.6.1; dependencies in requirements.txt with no additions beyond the starter set. From a fresh clone:

git clone https://github.com/mdsvr/flyrank && pip install -r requirements.txt, then run the notebooks in order. The capstone notebook re-renders every table on this page from the committed metrics files in seconds, without refitting anything — so the page and the notebook cannot silently disagree.

9Acknowledgments & data credit

Built on the FlyRank ML Internship dataset — https://flyrank.ai — a pseudonymised release of roughly 79 million rows of production search and engagement data. Working on real production data, rather than a tidied teaching set, is what made most of the findings in this paper possible: the client imbalance, the measurement gaps, the drifting base rate and the near-perfect client separability are all properties of real inventory that a clean benchmark would have hidden.

Thanks to the FlyRank ML Internship programme for the dataset, the weekly structure, and the research paper whose exploratory appendix gave me two methodology questions to ask — and then made me ask the same two questions of my own model, which is where the retractions in §6 came from.