Authors of a systematic review ask a specific clinical question, perform a comprehensive literature search, eliminate irrelevant or poor quality studies, and attempt to make practice recommendations based on the combined evidence from relevant, good quality research.
This guide indicates required stages in the process of conducting a systematic review.
It does not cover the structure and layout of the review. See the box labelled Systematic review sources to find examples of structure and layout in existing reviews.
Systematic reviews must include all stages of this process to be regarded as systematic.
Not all reviews need to be systematic reviews. There are other useful types of reviews that might be more appropriate for your research.
You can still take a systematic approach in another type of review, without resulting in a systematic approach.
Quantitative systematic reviews are usually designed to answer a yes/no or A/B question - is this therapy better than placebo, or is therapy A better than therapy B? If your research question is not in this form, another type of review might be more relevant.
Qualitative systematic reviews can have some leeway, but should also have a similar strong focus e.g. has this change made a beneficial difference, or is this approach more beneficial than that approach?
A preliminary protocol (or plan) document can address and plan for these required stages.
Many systematic reviews include this protocol stage.
The PRISMA-P guidelines and documents are a good guide for preparing this protocol.
PRISMA-P = Preferred Reporting Items for Systematic Reviews and Meta-Analyses - ProtocolsThe question must be: Explicit, specific, clearly defined, answerable.
Note: Before settling on a final systematic review question, preliminary scoping searches can help to clearly define the specific topic and research question. These preliminary searches can help review the terminology and examine the breadth of existing research.
Although not absolute, systematic reviews are typically defined by a question with a simple binary answer: Yes or no, true or false, A or B. This is true of both quantitative and (some) qualitative systematic reviews, only the types of studies and evidence will differ.
For example:
An unacceptable question:
"Better" is not clearly defined and thus not answerable. Does this mean fewer side effects, fewer problematic interactions with other therapies, less costly, more effective outcomes, or something else?
Changed to be acceptable:
This version has defined what can be measured (explicit, clearly defined), and the question is now answerable (and hopefully the research exists to be reviewed - see the scoping link above).
For example:
For the objectivity required by a systematic review, these criteria must be defined before reviewing the search results.
Preliminary scoping searches can help to define these inclusion/exclusion criteria, but for the final systematic review these criteria should be clearly defined before reviewing the final defined search results.
The search stage of a systematic review must be: sensitive, repeatable, verifiable, and transparent.
The explicit search strategy must be published as part of the systematic review.
Remember: Preliminary scoping searches can help to define search terms and alternatives.
For example:
Don't use "internet search" to search for a phrase.
Instead, use internet N4 search* (in an EBSCO database). This will find the words internet and search within 4 words of each other (in either direction).
(The asterisk at the end of search will find other word endings such as searching, searched, searches.)
This is better than "internet search" because it will still find internet search, but it will also find internet keyword search, internet site search, searching the internet, and similar phrases.
Adjacency operators vary considerably between databases (they do not all use N). The Library has a table of adjacency operators and wildcards for your searching convenience .
Search filters are not the limits and refinements you will encounter in databases.
In a systematic review context, the phrase "search filters" refers to pre-tested search strategy fragments that have been refined to produce all results of a specific type.
Search filters are sometimes known as "hedges".
A search filter is a pre-tested set of search terms and Boolean combinations that you can combine with your own search strategy for more sensitive search results.
Sensitive searches, preferred for systematic reviews, will find every article related to the search topic at the cost of including some articles that might be irrelevant.
For example, you might use a search filter to find all randomised trials. You can then combine that with your topic keywords to be sure you will find all randomised trials on your topic.
Some sources for search filters are included further down on this page.
Controlled vocabulary refers to keywords added to an article by the publisher or database. These are also known as thesaurus terms, descriptors, subject headings, topic headings, and index headings. They are independent of the text of the article. Social media uses hashtags for the same reason.
For instance, even if the author only uses decubitus ulcers in an article, the standard subject heading pressure ulcers will be added by the publisher or database.
In theory, these terms alone should be sufficient, but in practice application of descriptors is neither complete nor consistent. Relevant material might be missed using descriptors alone.
Free-text searching refers to searching for words used by the author(s) in the article itself.
Free-text searching alone might miss some unknown alternative terms, so a mix of both types is required:
For example:
Pressure Ulcers [subject heading] OR pressure ulcers [free text search] OR [other free text search terms]
If you work in health, you may already be familiar with PICO . PICO is a framework for dividing the search terms into 4 categories. This can make it easier to formulate a search strategy.
Variations include PICOS (which also includes Study design as a search term category), PICOT (which includes Timeframe, such as the length of the study or the time for an intervention to produce outcomes), or PICOTS (which includes both timeframe and study design).
For qualitative or mixed method studies, another useful search framework is SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type).
Some additional, less-structured searching ( also known as "hand searching") might also be used in addition to your formal search strategy, such as:
Manually work through the search results and decide whether each result should be included or excluded from the review, using the pre-defined eligibility criteria from stage 2 to decide.
This stage should be repeatable/reproducible and transparent. Anyone with the same search results and the same criteria should be able to produce the same set of included/excluded results.
Therefore, to avoid bias and check for consistency, at least two researchers should be involved in this stage, and any differences should be reconciled, preferably by a third party, using the eligibility criteria.
Quantitative systematic reviews might be able to use a meta-analysis - a statistical technique for producing a combined analysis from several different studies.
Qualitative systematic reviews might use a different types of analysis, such as a meta-synthesis, but the aim is still to combine the different studies into one set of evidence for answering the specific review question.
Systematic reviews often include a meta-analysis, and sometimes confusion arises between systematic reviews and meta-analysis reviews.
A meta-analysis review tends to focus specifically on the statistical analysis stage.
A meta-analysis review might use more of a high specificity search rather than a high sensitivity search. This results in fewer results and a higher proportion of more relevant articles. However, it might miss some relevant sources that a high sensitivity search will find.
A meta-analysis review will not necessarily use explicit eligibility criteria for inclusion/exclusion.
A meta-analysis might not present recommendations for practice or further research.
The results of the analysis/synthesis should unambiguously support a conclusion that clearly answers the original specific question.
The conclusion should produce recommendations for practice as a result of answering the research question.
Alternatively, if the evidence is insufficient or unclear, the analysis might produce a conclusion of "more research needed" or "more specific research needed" as an answer to the systematic review question.
Be sure to reference correctly, including not only the selected studies that form part of the analysis but all other sources cited during the review (e.g. in the introduction, analysis, and discussion).
Systematic reviews should be carefully documented and tracked, and a report may be generated as part of the systematic review.
The PRISMA process (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) is one such method of reporting.
It includes:
Researchers should also retain other documentation, including the selection list with inclusion/exclusion decisions made against the eligibility criteria.
Search filters (sometimes known as "hedges") are pre-tested search fragments that you can include in your search strategy.
Search filters will find all articles of a specific type (methodology type, specific subject/discipline area, etc).
Combine one or more search filters with your own topic search terms for more sensitive (wider coverage) searching.
Here are some common sources for research filters (this is not a comprehensive list):
Cochrane Handbook for creating systematic reviews
If you are working on a systematic review, the Cochrane handbook will help you to produce a higher quality review.
The Cochrane Collaboration insists on rigorous standards for including systematic reviews in the Cochrane Library.
This can result in reviews that take longer to complete (possibly a couple of years), but these reviews are regarded as the gold standard for quality, reliability, and repeatability.
Part of this rigorous approach has been the development of "filters", standard search strategy fragments that find everything of a specific type e.g. every possible naming variant of a randomised trial.
Searches for Cochrane systematic reviews must use high sensitivity. This means using searches that find all the evidence that might have any possible relevance to the topic. Relevant items are then selected manually at the inclusion/exclusion stage.
Finding several thousand articles and including/excluding them individually to result in a final list of 20 to 40 articles is not unusual in a Cochrane systematic review.
A Cochrane systematic review cannot be done by a single researcher, because the selection/rejection process in Cochrane reviews specifies 2 or more researchers selecting/rejecting articles individually by pre-defined criteria then reconciling any differences.
The processes for qualitative systematic reviews are still developing.
The rigid processes used for quantitative systematic reviews are not always useful nor appropriate for qualitative review questions and research.
The searching, selection, and analysis/synthesis portions of qualitative systematic reviews might vary considerably due to the requirements to address different purposes and research questions.
Some potentially useful reading:
Individual consultations with a Liaison Librarian are available to all staff and HDR students.
1300 552 567 or Ask The Library
Systematic reviews use a careful approach to avoid bias and gather relevant evidence. This provides credible evidence to make valid conclusions that can change professional practice.
As a result, systematic reviews can hold more professional significance than other reviews or individual studies.
However, systematic reviews usually require longer timeframes and greater attention to the process.
Systematic reviews are best used for answering a specific narrow research question. They are not useful for wider coverage of a topic.
Some databases have an option to limit search results to only systematic reviews.
These operators are used to retrieve references that contain the specified search terms or phrases within a certain number of words from each other in the same indexed field.
For example:
In EBSCO databases: tertiary N3 education will find articles containing the phrase education in a tertiary institute as well as articles that contain the phrase tertiary education
Some more information about data analysis can be found on the Data Preparation & Analysis page of the Research Data Management guide.
Grant, M. J., & Booth, A. (2009). A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26(2), 91–108. https://doi.org/10.1111/j.1471-1842.2009.00848.x
Paré, G., Trudel, M.-C., Jaana, M., & Kitsiou, S. (2015). Synthesizing information systems knowledge: A typology of literature reviews. Information & Management, 52(2), 183–199. https://doi.org/10.1016/j.im.2014.08.008