The package `metarep`

is an extension to the package `meta`

, which allows incorporating replicability-analysis tools to quantify consistency and replicability of treatment effect estimates in a meta-analysis. The tool was proposed by Jaljuli et. al. (submitted) for the fixed-effect and for the random-effects meta-analyses, whit or without the common-effect assumption.

Regardless of the type of meta-analysis applied, `metarep`

allows to perform replicability analysis with or with out the common-effect assumption of the fixed-effects model. We recommend applying the replicability analysis free of the common-effect assumption to guard from a possibly faulty assumption. At this case, the replicability analysis is performed based Fishers’ combining function using truncated-Pearson’s’ test. If the user finds the common-effect assumption supported and wishes to incorporate it in the replicability analysis, they might as well do that. Whereas with this assumption, the replicability analysis is performed using the test statistic of the fixed-effects model, for the combining of every set of \(n-u+1\) studies.

Currently, both `meta`

and `metarep`

packages can be downloaded from `GitHub`

, therefore make sure that the package `devtools`

is installed. `metarep`

also requires the latest version of `meta`

(>= 4.11-0, available on `GitHub`

)

Run the following commands in to install the packages:

Here we demonstrate the approach by implementation with `metarep`

, using examples from systematic reviews Cochrane library. These examples are detailed in the paper as well, along with a demonstration of a way to incorporate our suggestions in standard meta-analysis reporting system.

We demonstrate the with an example based on a fixed-effects meta-analysis. This example was included in Jaljuli et. al. 2020, found in review number CD002943 in the Cochrane library. This analysis explores the effect of mammogram invitation on attendance during the following 12 months.

```
library(metarep)
library(meta)
Loading 'meta' package (version 4.12-0).
Type 'help(meta)' for a brief overview.
Attaching package: 'meta'
The following object is masked from 'package:metarep':
forest
data(CD002943_CMP001)
m2943 <- metabin( event.e = N_EVENTS1, n.e = N_TOTAL1,
event.c = N_EVENTS2, n.c = N_TOTAL2,
studlab = STUDY, comb.fixed = T , comb.random = F,
method = 'Peto', sm = CD002943_CMP001$SM[1],
data = CD002943_CMP001)
m2943
OR 95%-CI %W(fixed)
Sutton-1994 1.2836 [0.9933; 1.6589] 32.3
Somkin-1997 1.8739 [1.5372; 2.2844] 54.2
Turnbull-1991 3.5709 [1.9225; 6.6326] 5.5
Mohler-1995 1.8764 [0.5272; 6.6788] 1.3
Bodiya-1999 1.0758 [0.6110; 1.8941] 6.6
Number of studies combined: k = 5
OR 95%-CI z p-value
Fixed effect model 1.6564 [1.4317; 1.9164] 6.78 < 0.0001
Quantifying heterogeneity:
tau^2 = 0.0882 [0.0000; 1.5129]; tau = 0.2970 [0.0000; 1.2300];
I^2 = 70.3% [24.4%; 88.3%]; H = 1.84 [1.15; 2.93]
Test of heterogeneity:
Q d.f. p-value
13.47 4 0.0092
Details on meta-analytical method:
- Peto method
- DerSimonian-Laird estimator for tau^2
- Jackson method for confidence interval of tau^2 and tau
summary(m2943)
Number of studies combined: k = 5
OR 95%-CI z p-value
Fixed effect model 1.6564 [1.4317; 1.9164] 6.78 < 0.0001
Quantifying heterogeneity:
tau^2 = 0.0882 [0.0000; 1.5129]; tau = 0.2970 [0.0000; 1.2300];
I^2 = 70.3% [24.4%; 88.3%]; H = 1.84 [1.15; 2.93]
Test of heterogeneity:
Q d.f. p-value
13.47 4 0.0092
Details on meta-analytical method:
- Peto method
- DerSimonian-Laird estimator for tau^2
- Jackson method for confidence interval of tau^2 and tau
```

In this meta-analysis, the effect of sending invitation letters was examined in five studies. The authors main result is that: “The odds ratio in relation to the outcome, attendance in response to the mammogram invitation during the 12 months after the invitation, was 1.66 (95% CI 1.43 to 1.92)”.

We suggest reporting the replicability-analysis results alongside: the r-value and lower confidence bounds on the number of studies. Results of complete replicability analysis can be added to the contents of a `meta`

object or using the function `metarep(...)`

, as well as to its summary using `summary( metarep(...) )`

.

To perform assumption-free replicability-analysis requiring replicability in at least `u = 2`

(default) studies, we calculate \(r(2)-value\) using truncated-Pearson’s’ test with truncation threshold `t=0.05`

(default):

```
(m2943.ra <- metarep(x = m2943 , u = 2 , common.effect = F ,t = 0.05 ,report.u.max = T))
OR 95%-CI %W(fixed)
Sutton-1994 1.2836 [0.9933; 1.6589] 32.3
Somkin-1997 1.8739 [1.5372; 2.2844] 54.2
Turnbull-1991 3.5709 [1.9225; 6.6326] 5.5
Mohler-1995 1.8764 [0.5272; 6.6788] 1.3
Bodiya-1999 1.0758 [0.6110; 1.8941] 6.6
Number of studies combined: k = 5
OR 95%-CI z p-value
Fixed effect model 1.6564 [1.4317; 1.9164] 6.78 < 0.0001
Quantifying heterogeneity:
tau^2 = 0.0882 [0.0000; 1.5129]; tau = 0.2970 [0.0000; 1.2300];
I^2 = 70.3% [24.4%; 88.3%]; H = 1.84 [1.15; 2.93]
Test of heterogeneity:
Q d.f. p-value
13.47 4 0.0092
Details on meta-analytical method:
- Peto method
- DerSimonian-Laird estimator for tau^2
- Jackson method for confidence interval of tau^2 and tau
- replicability analysis (r-value = 2e-04)
- out of 5 studies, at least: 2 with increased effect and 0 with decreased effect.
```

The bottom two lines report the \(r(2)-value\), lower bound on the number of studies with increased effect (\(u^L_{max}\)) and decreased effect (\(u^R_{max}\)) , respectively. The evidence towards an increased effect was replicable, with \(r(2) − value = 0.0002\). Moreover, with \(95\%\) confidence, we can conclude that at least two studies had an increased effect. For higher replicability requirement, compute \(r(u')-value\) for \(u'>2\) using `metarep(u = u' , ... )`

.

The two-sided \(r(u)-value\) of the model can be accessed via `r.value`

:

The replicability-analysis reported was performed with an assumption free test, based on truncated-Pearson’s’ test with truncation level set at the nominal hypothesis testing level (i.e., `t=0.05`

, default). For ordinary Pearson’s’ test, use `t=1`

.

Although the fixed-effect model assumes that all studies are estimates of the same common effect \(\theta\), we recommend applying assumption-free replicability-analysis for protection against an (perhaps) unsupported assumption. Despite that, we extend our suggested method with the common-effect incorporation in section 7. This analysis can be performed via `metarep( ... , common.effect = TRUE )`

.

`metarep`

also allows adding replicability results to the conventional forest plots by `meta`

. This can be done by simply applying `forest()`

on a `metarep`

object.

The lower bounds \(u^L_{max} \, \text{and} \, u^R_{max}\) are calculated with \(1-\alpha = 95\%\) confidence level ( default), meaning that each of the null hypotheses \[H^{u^L_{max}/n}(L) \;\; \text{and}\;\; H^{u^L_{max}/n}(R)\] is tested at level \(\alpha /2 = 2.5\%\), resulting in bounds in overall type error rate \(5\%\). Type I error rate can be controlled for any desired \(\alpha\) using the argument `confidence = 1 -`

\(\alpha\).

The calculation of \(u^L_{max} \, \text{and} \, u^R_{max}\) can also be calculated directly using the function `find_umax()`

with the option to specify one-sided alternative, confidence level, truncation threshold and common-effect assumption. For example, let’s compute \(u^L_{max}\) with the same confidence level as produced by `m2943.ra.bounds`

.

```
find_umax(x = m2943 , common.effect = F,alternative = 'less',t = 0.05,confidence = 0.975)
$worst.case
[1] "Sutton-1994" "Somkin-1997" "Turnbull-1991" "Mohler-1995"
[5] "Bodiya-1999"
$side
Direction of the stronger signal
"less"
$u_max
u^L
0
$r.value
r^R
1
$Replicability_Analysis
[1] "out of 5 studies, 0 with decreased effect."
```

Note that this function produces 2 main types of results:

Worst-case scenario studies: A list of \(n-u_{max}^L+1\) studies names yielding the maximum \[\max_{\forall \{i_1,\dots , i_{n-u+1}\} \subset \{1,\dots , n\} } \{\,p^L_{i_1,\dots , i_{n-u+1}}\,\}\]

Replicability-analysis results, including:

\(u^L_{max}\) or \(u^R_{max}\). If

`alternative='two-sided'`

, then \(u_{max}=\max\{u^L_{max}\, , \, u^R_{max}\}\) is also reported.\(r(u_{max}^L)-value\) or \(r(u_{max}^R)-value\) if setting

`alternative='less'`

or`alternative='greater'`

, respectively. If`alternative='two-sided'`

, then \[rvalue = r(u_{max}) = 2\cdot\min\{r^R(u_{max}^R) , r^L(u_{max}^L) \}\] is also reported.

For demonstration, see the following example.

```
find_umax(x = m2943 , common.effect = F,alternative = 'two-sided',t = 0.05,confidence = 0.95)
$worst.case
[1] "Sutton-1994" "Turnbull-1991" "Mohler-1995" "Bodiya-1999"
$side
Direction of the stronger signal
"greater"
$u_max
u_max u^L u^R
2 0 2
$r.value
r.value r^L r^R
2e-04 1e+00 1e-04
$Replicability_Analysis
[1] "out of 5 studies: 0 with decreased effect, and 2 with increased effect."
```

```
find_umax(x = m2943 , common.effect = T,alternative = 'two-sided', confidence = 0.95)
Performing Replicability analysis with the common-effect assumption
$worst.case
[1] "Sutton-1994" "Mohler-1995" "Bodiya-1999"
$u_max
[1] 3
$side
[1] "greater"
$r.value
[1] 0.0468
(m2943.raFE <- metarep(x = m2943 , u = 2 , common.effect = T ,report.u.max = T))
Performing Replicability analysis with the common-effect assumption
Performing Replicability analysis with the common-effect assumption
OR 95%-CI %W(fixed)
Sutton-1994 1.2836 [0.9933; 1.6589] 32.3
Somkin-1997 1.8739 [1.5372; 2.2844] 54.2
Turnbull-1991 3.5709 [1.9225; 6.6326] 5.5
Mohler-1995 1.8764 [0.5272; 6.6788] 1.3
Bodiya-1999 1.0758 [0.6110; 1.8941] 6.6
Number of studies combined: k = 5
OR 95%-CI z p-value
Fixed effect model 1.6564 [1.4317; 1.9164] 6.78 < 0.0001
Quantifying heterogeneity:
tau^2 = 0.0882 [0.0000; 1.5129]; tau = 0.2970 [0.0000; 1.2300];
I^2 = 70.3% [24.4%; 88.3%]; H = 1.84 [1.15; 2.93]
Test of heterogeneity:
Q d.f. p-value
13.47 4 0.0092
Details on meta-analytical method:
- Peto method
- DerSimonian-Laird estimator for tau^2
- Jackson method for confidence interval of tau^2 and tau
- replicability analysis (r-value = 0.0011)
- out of 5 studies, at least: 3 with increased effect
```

```
find_umax(x = m2943 , common.effect = T,alternative = 'two-sided', confidence = 0.95)
Performing Replicability analysis with the common-effect assumption
$worst.case
[1] "Sutton-1994" "Mohler-1995" "Bodiya-1999"
$u_max
[1] 3
$side
[1] "greater"
$r.value
[1] 0.0468
(m2943.raFE <- metarep(x = m2943 , u = 2 , common.effect = T ,report.u.max = T))
Performing Replicability analysis with the common-effect assumption
Performing Replicability analysis with the common-effect assumption
OR 95%-CI %W(fixed)
Sutton-1994 1.2836 [0.9933; 1.6589] 32.3
Somkin-1997 1.8739 [1.5372; 2.2844] 54.2
Turnbull-1991 3.5709 [1.9225; 6.6326] 5.5
Mohler-1995 1.8764 [0.5272; 6.6788] 1.3
Bodiya-1999 1.0758 [0.6110; 1.8941] 6.6
Number of studies combined: k = 5
OR 95%-CI z p-value
Fixed effect model 1.6564 [1.4317; 1.9164] 6.78 < 0.0001
Quantifying heterogeneity:
tau^2 = 0.0882 [0.0000; 1.5129]; tau = 0.2970 [0.0000; 1.2300];
I^2 = 70.3% [24.4%; 88.3%]; H = 1.84 [1.15; 2.93]
Test of heterogeneity:
Q d.f. p-value
13.47 4 0.0092
Details on meta-analytical method:
- Peto method
- DerSimonian-Laird estimator for tau^2
- Jackson method for confidence interval of tau^2 and tau
- replicability analysis (r-value = 0.0011)
- out of 5 studies, at least: 3 with increased effect
forest(m2943.raFE, layout='revman5',digits.pval = 2 , test.overall = T )
```