In a deep CTR analysis of 30,000 query/URL pairs, we looked for relationships between query and basic snippet text features including title, description and URL. All combined feature attempts resulted in signal noise and weak correlations. In response, we trained four separate classifiers, each specialising in a single aspect of the analysed SERP snippet. Each classifier was used to process the entire dataset and the results were combined.
This lead to clearer correlations, better feature interpretability and the discovery of search query as the dominant factor in CTR delta prediction. This finding highlights the importance of query intent comprehension with significant implications on traffic modelling and projection tasks in search.
The rest of this article outlines our process and concludes with practical application in the context of SEO analysis and strategy design.
Read on or skip to the end for TLDR.
In our study we slice the data to a single dimension group, for example:
Google Search Console data was pulled in via API with the following dimensions for a 90 day range:
Since each query can have multiple target URLs we determine the canonical URL for each query by selecting the top URL by clicks for that query.
Example:
This step is key in reducing bias and noise in CTR average calculations.
User behaviour differs vastly between different query types and so we employ our custom-made query intent classifier to label the query dataset into:
LABEL_0: ‘Commercial’
LABEL_1: ‘Non-Commercial’
LABEL_2: ‘Branded’
LABEL_3: ‘Non-Branded’
LABEL_4: ‘Informational’
LABEL_5: ‘Navigational’
LABEL_6: ‘Transactional’
LABEL_7: ‘Commercial Investigation’
LABEL_8: ‘Local’
LABEL_9: ‘Entertainment’
This step is key because it allows for intent-specific CTR average calculations. A query “seo by dejan” on #1 will have a 30% CTR while query “seo testimonials” on #1 will have a 1.8% CTR. If you treat them equally you’ll end up with noisy CTR averages. Splitting calculations into branded and non-branded queries addresses the biases associated with brand-related query intent.
With clean CTR statistics it’s possible to do three levels of traffic projections:
With the above it’s then possible to add conversion metrics such as conversion rate and conversion rate value and provide rudimentary ROI estimates signalling which area of SEO improvement will lead to which financial outcomes.
Important: When projecting clicks for rank increase scenarios it’s advisable to scale the projection proportionately to the current performance of the query. In other words if a query underperforms by 10% on rank 2, it will likely continue to underperform by 10% on rank 1. Similarly if a query performs better than average, moving it up in rank and factoring in site-average would indeed reduce its calculated click gains.
The next step is to scrape and extract meta data for each page and store it in a database. We use a combination of trafilatura and beautiful soup for the task. One for full page content and another for more precise extraction of titles and descriptions. Later down the track, the above mentioned metadata fits into the following structure:
Query [SEP] Title [SEP] Description [SEP] URL [SEP] CTR Delta
Equipped with clean and unbiased CTR statistics we’re able to calculate CTR Deltas for each query/URL pair. In the example provided we see a non-branded query page receiving a 20% lower CTR than expected for that site resulting in a loss of 500 clicks. This holds true after we filter for that exact country, device and query type.
With a reliable CTR Delta metric we can calculate the click loss and flag that query for investigation.
Is the cause for bad CTR in the snippet itself or is there something else?
We won’t be going into special SERP features and their impact on CTR as it is not the focus of this particular workflow.
And as is with anything related to machine learning, most of the time is spent on getting your data clean and ready. And now we’re ready. We have the CTR stats, the deltas, queries, titles, descriptions and URLs.
We fine-tune four small transformers on separate tasks and combine the predictions of each specialist model:
Each model provides classification for whether the query / snippet combination will result in better or worse CTR than average. It also provides a handy softmax value which can be used to scale predictions when calculating final score.
Here’s the full setup:
├── 2024-08-17T14-24_export.csv
├── albert-ctr-description-model
│ ├── config.json
│ ├── model.safetensors
│ ├── special_tokens_map.json
│ ├── spiece.model
│ └── tokenizer_config.json
├── albert-ctr-query-model
│ ├── checkpoints
│ │ ├── checkpoint-1553
│ │ │ ├── config.json
│ │ │ ├── model.safetensors
│ │ │ ├── optimizer.pt
│ │ │ ├── rng_state.pth
│ │ │ ├── scheduler.pt
│ │ │ ├── trainer_state.json
│ │ │ └── training_args.bin
│ │ ├── checkpoint-1582
│ │ │ ├── config.json
│ │ │ ├── model.safetensors
│ │ │ ├── optimizer.pt
│ │ │ ├── rng_state.pth
│ │ │ ├── scheduler.pt
│ │ │ ├── trainer_state.json
│ │ │ └── training_args.bin
│ │ ├── checkpoint-3106
│ │ │ ├── config.json
│ │ │ ├── model.safetensors
│ │ │ ├── optimizer.pt
│ │ │ ├── rng_state.pth
│ │ │ ├── scheduler.pt
│ │ │ ├── trainer_state.json
│ │ │ └── training_args.bin
│ │ ├── checkpoint-3164
│ │ │ ├── config.json
│ │ │ ├── model.safetensors
│ │ │ ├── optimizer.pt
│ │ │ ├── rng_state.pth
│ │ │ ├── scheduler.pt
│ │ │ ├── trainer_state.json
│ │ │ └── training_args.bin
│ │ ├── checkpoint-4659
│ │ │ ├── config.json
│ │ │ ├── model.safetensors
│ │ │ ├── optimizer.pt
│ │ │ ├── rng_state.pth
│ │ │ ├── scheduler.pt
│ │ │ ├── trainer_state.json
│ │ │ └── training_args.bin
│ │ └── checkpoint-4746
│ │ ├── config.json
│ │ ├── model.safetensors
│ │ ├── optimizer.pt
│ │ ├── rng_state.pth
│ │ ├── scheduler.pt
│ │ ├── trainer_state.json
│ │ └── training_args.bin
│ ├── config.json
│ ├── model.safetensors
│ ├── special_tokens_map.json
│ ├── spiece.model
│ └── tokenizer_config.json
├── albert-ctr-title-model
│ ├── checkpoints
│ │ ├── checkpoint-1582
│ │ │ ├── config.json
│ │ │ ├── model.safetensors
│ │ │ ├── optimizer.pt
│ │ │ ├── rng_state.pth
│ │ │ ├── scheduler.pt
│ │ │ ├── trainer_state.json
│ │ │ └── training_args.bin
│ │ ├── checkpoint-3164
│ │ │ ├── config.json
│ │ │ ├── model.safetensors
│ │ │ ├── optimizer.pt
│ │ │ ├── rng_state.pth
│ │ │ ├── scheduler.pt
│ │ │ ├── trainer_state.json
│ │ │ └── training_args.bin
│ │ └── checkpoint-4746
│ │ ├── config.json
│ │ ├── model.safetensors
│ │ ├── optimizer.pt
│ │ ├── rng_state.pth
│ │ ├── scheduler.pt
│ │ ├── trainer_state.json
│ │ └── training_args.bin
│ ├── config.json
│ ├── model.safetensors
│ ├── special_tokens_map.json
│ ├── spiece.model
│ └── tokenizer_config.json
├── albert-ctr-url-model
│ ├── checkpoints
│ │ ├── checkpoint-1582
│ │ │ ├── config.json
│ │ │ ├── model.safetensors
│ │ │ ├── optimizer.pt
│ │ │ ├── rng_state.pth
│ │ │ ├── scheduler.pt
│ │ │ ├── trainer_state.json
│ │ │ └── training_args.bin
│ │ ├── checkpoint-3164
│ │ │ ├── config.json
│ │ │ ├── model.safetensors
│ │ │ ├── optimizer.pt
│ │ │ ├── rng_state.pth
│ │ │ ├── scheduler.pt
│ │ │ ├── trainer_state.json
│ │ │ └── training_args.bin
│ │ └── checkpoint-4746
│ │ ├── config.json
│ │ ├── model.safetensors
│ │ ├── optimizer.pt
│ │ ├── rng_state.pth
│ │ ├── scheduler.pt
│ │ ├── trainer_state.json
│ │ └── training_args.bin
│ ├── config.json
│ ├── model.safetensors
│ ├── special_tokens_map.json
│ ├── spiece.model
│ └── tokenizer_config.json
├── raw_training_data.csv
├── test.py
├── train_description.py
├── train_query.py
├── train_title.py
└── train_url.py
All four models were driven to the point of overfitting and the checkpoint before the overfit was selected as the best model.
Overfitting is when model is trained to the point where its weights are adjusted to best fit training data features. This makes it perform exceptionally well on training data but less likely to generalise on new and never-before-seen data.
Our approach is to test the limit of the model and see at which point it overfits and select the checkpoint saved just before it happens.
On inference we chain all four models in a single pipeline, each one making predictions for a single SERP snippet feature:
Output includes:
We’ve developed a CTR delta prediction framework based on a pipeline of four small transformers, each dedicated to a single task.
The framework predicts whether CTR for a page query pair will have a better or worse CTR than expected. We input a query, the models return scores, we combine the scores and return a prediction. We do this in a batch-processing pipeline.
Output of all models is combined and balanced in proportion to their softmax values. This allows more confident predictions to have a stronger vote.
It predicts CTR delta correctly 8 out of 10 times. This is most likely as good as it gets without adding new biases such as special SERP features.
Regression. CTR is not a binary value and while our method predicts “better or worse” outcomes it would be more helpful to understand “by how much”. A regression model will solve that problem for us as the fifth node in the model chain.
Its input will be the classifier quartet output combined with the snippet features and its output will be a percentage of how much the CTR will deviate from the norm.
Stay tuned.
Dan Petrovic, the managing director of DEJAN, is Australia’s best-known name in the field of search engine optimisation. Dan is a web author, innovator and a highly regarded search industry event speaker.
ORCID iD: https://orcid.org/0000-0002-6886-3211
