Systematic review is the gold-standard of medical evidence if it’s done properly. But not all health researchers know how to do a systematic review, so I see the same mistakes being made over and over again.
I’m here to show you how to do it properly so that you can avoid those mistakes and save yourself oodles of time!
I’ll kick off with some background, take you through 5 vital steps, and then finish with a list of handy resources.
(I’ll also confess the biggest systematic review mistake that I’ve ever made as a reminder that we all start somewhere when we’re learning!)
Why you should do a systematic review
Before we dig into the juicy details of doing a systematic literature review, let’s get pumped up about why we want to do one in the first place.
Systematic reviews are the best medical evidence…
You might have seen this evidence hierarchy before because it’s well known. The higher up the triangle you go, the more trustworthy the evidence is.
See that point at the top? We’re aiming for that.
Systematic reviews are at the top because they are:
- Objective (not opinion based)
- Comprehensive (include all research on a topic)
- Precise (answer a specific question)
- Reproducible (if you did the exact same thing again then you should get the same result)
…and they’re great for career development…
Give yourself a career boost by publishing your own research. It establishes you as an expert and gives you wider reach, deeper knowledge and extra skills.
It’s also really interesting to research something you feel passionate about!
Literature reviews are an ideal starting point. They’re low cost and don’t need participant recruitment, a big team, or regulatory approval.
Plus they can often be published in high impact journals because they’re the crème de la crème of evidence.
…but only if they’re done right
If you don’t know what you’re doing, it’s easy not to hit the objective, comprehensive, precise and reproducible parts of a systematic review or spend much longer than needed getting there.
So we’re going to go over 5 of the most important steps (which also happen to be the most common mistakes that I see!).
Step 1. Plan like a boss
Here’s an often-repeated life lesson from my Dad…
“Failing to prepare is preparing to fail”
Dad said this a lot in my teens when I hadn’t got my school stuff ready the night before. It probably explains why I’m now super organized.
So how will failing to prepare affect your review?
It will take you soooo much longer to do your review if you don’t plan it well upfront. You’ll repeat steps and retrieve papers and data that you then don’t use.
Worst case scenario is you’ll have to bin everything and start from scratch. I’ve seen this happen more than once and it usually ends in tears and frustration.
We can avoid this by putting together a solid plan before you start
It’s best practice to write a review protocol.
A protocol is really helpful for keeping you on track and making sure you and your collaborators all have the same understanding of what’s happening.
Most journals now also want to see your protocol when you submit your review for publication.
Check out the PRISMA website for useful information on how to write a systematic review protocol.
You’ll need to include these steps in your protocol
These are the steps you’ll work through to complete your review and what to cover in your protocol.
| Review step | What to include in your protocol |
| Decide on your research question | Primary and secondary research questions (more on this tomorrow) |
| Decide on your overall approach to your literature review and who will do what | Overall literature search strategy |
| Decide on what sources you’ll search and what search terms (usually a combination of key words and MeSH terms) and limits (such as a date range or specific languages) you’ll use for each source | Data sources plus exact search terms and limits |
| Decide on your inclusion/exclusion criteria for selecting studies | Selection criteria |
| Carry out a scoping review and use it to refine your research question, sources, search terms, limits and selection criteria. You can also check that the same review hasn’t already been done/being done | Results of any scoping review (optional but really helpful) |
| Finalise your protocol | |
| Register your review on PROSPERO (highly recommended and required by most journals) | Plans for PROSPERO registration |
| Run your searches | Details on sources, search terms and limits |
| Combine all of the identified publications and remove duplicates | Criteria for removing duplicate research, for example duplicate publications can be identified via most reference management software. You’ll also need to think about what you’ll do if there are multiple papers reporting on the same paper. How will you choose which one to keep? |
| Identify any other relevant published and grey (unpublished) literature | Details on how you will you attempt to retrieve all relevant literature, for example hand-searching reference lists, contacting prominent leaders in the field and internet searches |
| Screen titles and abstracts for inclusion (first pass) | Inclusion/exclusion criteria which should reflect your research question and plans for independent screening or quality control (more on this in a couple of days time) |
| Retrieve full papers of publications that remain potentially eligible | Include details on whether you will still include abstracts if the full paper can’t be retrieved |
| Screen full papers for inclusion (second pass) | Inclusion/exclusion criteria which should reflect your research question and plans for independent screening or quality control (more on this in a couple of days time) |
| Extract data (you can also contact study authors for any missing data) | Plans for data management and quality control of data extraction (more on this coming up) |
| Assess the quality of each publication | How you will do the quality assessment. There are lots of published measures you can use. |
| Summarise and analyse the data | Analysis plan |
| Write the report | Plan for reporting |
Step 2. Nail your research question
Us medical statisticians LOVE you to drill down on exactly what your research question is
That’s because being crystal clear on that will make your research easier to do and easier for other people to understand.
So it massively increases your impact.
This blog post walks you through how to write your research question.
PICO is a tool to make your research question is clear
This stands for:
P – Population or Patient
I – Intervention or Exposure
C – Comparison
O – Outcome you would like to measure
Finding it hard to choose only one research question?
For a systematic review, it’s often best to stick with one research question, but it’s okay to have a few. I usually recommend keeping it to 3 at most.
If you have more than one then always choose one as the primary research question. The others will then be secondary research questions.
It’s best to have as much overlap as you can in the PICO for each question. For example, only change the outcome between questions. Otherwise you’ll have to run multiple searches and screenings.
Step 3. Get the experts in
You can technically do an end-to-end evidence review by yourself.
But you’ll end up with a better review if you involve other people.
Here’s my run down of who to get involved and when.
Clinician
If it’s not your area of clinical expertise then you’ll want to involve a clinician who knows the subject area well.
They’ll help to refine your research question and give you the background knowledge you need to plan your search criteria.
Librarian
Clinical librarians are invaluable in helping you to write your search terms.
They’ll help you with general search strategy approaches, like Boolean operators and wild cards. They’ll also be able to help with alternative words, phrases and spellings for your search terms.
If you’re associated with a hospital or university then there’s likely to be a librarian who specializes in literature reviews.
If you don’t have access to a clinical librarian then get in touch as I can support you with this.
Statistician
If you’re doing a review of quantitative data (numbers-based) then you’re going to need to summarise and synthesise the data you find.
You can do this yourself if it’s a straightforward meta-analysis or a statistician can do it for you (picture me jumping up and down with my hand up saying “Pick me! Pick me!”)
Either way, it’s always a good idea to involve a statistician at the protocol writing stage.
We can help you avoid collecting data you don’t need and make sure you collect the data you do need. Then you won’t have to go back and re-do your data extraction whilst your search is getting out of date.
Qualitative researcher
If you’re doing a qualitative review (words-based) instead and you’re not experienced with qualitative analysis then it’s a good idea to bring an expert in here too.
Like a statistician, they’ll be able to help you with the planning, conduct and analysis stages.
Second (and third) reviewer
To hit the gold standard of systematic review, you’ll need 2 reviewers to independently screen articles for inclusion. Any articles where you disagree can then be discussed and agreed between you, or a third reviewer can make the decision.
You’ll then go through the same process of independent review and agreement for the data extraction.
Second reviewer not got time for all of that? They could instead do a quality control of your search and data extraction where they check what you’ve done. Again, you can discuss disagreements or go to a third reviewer.
Step 4. Avoid screening twice by recording everything as you go
I promised to tell you the biggest mistake that I made
That way, you can avoid doing your literature search twice like I did with my first review.
Once you know, it’s obvious.
So I feel embarrassed confessing this but we’re friends now right.
And I know others have made the same mistake.
Let’s start with the end…
Checklists help you to write a better paper
When you publish a research paper, use a checklist to make sure you include everything you need.
The PRISMA checklist is the best one for systematic reviews.
I’d recommend using it in the planning stage as well to make sure you’re covering everything that you’re going to need to report on later. It’s no coincidence that PRISMA have tools for both protocol writing and reporting.
The PRISMA statement also has a flow chart
The flow chart shows each stage of your literature review, such as how many initial articles you found from each source, how many were excluded at the title/abstract screening stage (first pass), how many were excluded at the full paper screening stage (second pass), and more.
Can you guess what my mistake was yet?
That’s right. I didn’t track those screening stages so I had no way of completing the PRISMA flow chart.
I had to do ALL of the searching again.
This is such a simple mistake to avoid with some basic data management. Save each step of your search as you go.
You don’t need a fancy tool for this. Excel or a reference management software, like EndNote, work well. And keep a note of any inclusion/exclusion decisions you make because you’ll need to summarise them in your paper too.
Step 5. Do your data extraction the right way
You’ve ran your searches, retrieved the papers and decided which ones are eligible. Now we’ve got to get the relevant data out of the paper so it can be analysed.
This is called data extraction
You can extract straight into Excel or onto a paper copy of your data extraction form and then type that into Excel. I find it easiest to extract onto paper first, but try both and go with whichever works best for you.
There are 3 types of information you’ll need to extract
- Basic information about the paper (author, data, study type, location etc)
- Data needed for your planned analysis
- Information for quality/risk of bias assessment
Side note…
I’m focusing on quantitative data because I’m a statistics lady.
A lot of this also applies to qualitative data, but please check with a qualitative expert before you start extracting to save yourself headaches down the line!
Data extraction is a big topic
So here I’m going to focus on common mistakes.
Mistake 1… The gigantic spreadsheet
The point is to extract relevant data. Not to re-arrange the contents of the paper into your spreadsheet. Re-visit your research question and consult with your statistician to check what you actually need. Otherwise you might spend hours extracting data that is never used.
Mistake 2… Missing information
The opposite of this is where you don’t extract vital information and have to go back to the paper to pull it out. Broken record shout out: Your statistician can help with this!
Mistake 3… Forgot to pilot the data extraction form
Once you’ve written your data extraction form, use it with 2 or 3 papers to see how well it works. You’re looking to see how logical the flow is, whether you’re able to capture all the information you need, and whether it’s clear what should be extracted.
Then go back and revise your data extraction form based on how you got on. Use your revised data extraction form for all of the data extraction, including the papers you piloted.
Mistake 4… The un-analysable spreadsheet
I might have made that word up?! But the concept is real. It’s a spreadsheet that’s such a mess, you have to spend days tidying it up before you can do anything with it. I’ve got a blog post on how to avoid that.
Super useful resources
Here’s a quick recap of the things we’ve talked about that you should do to avoid common mistakes when doing a systematic review…
- Write a protocol before you start
- Write down a super clear research question using PICO
- Get expert help in where you need it
- Record details of your screening process as you go
- Make sure you extract all of the data that you need (and avoid data you don’t need) into a well-thought out spreadsheet
There’s lots out there to help you
Here’s a list of some of my favourite resources.
Planning – PRISMA Protocol, PICO, PROSPERO
Searching – Cochrane Library, PubMed, Google Scholar, Embase
Storing – EndNote Web
Appraising – Cochrane Collaboration’s tool for assessing risk of bias (there are lots of different tools out there so find one that meets your needs)
Reporting – PRISMA Checklist
I’d love to work with you on your systematic review
If you need a systematic review completing as a standalone project or a Clinical Evaluation Report then get in touch to chat through the options.