Soil handling methods should be selected based on research questions and goals

There is rapidly increasing interest in the contribution of plant–soil interactions to aboveground and belowground community composition and ecosystem processes (Bardgett & Wardle, 2010; Van der Putten et al., 2013). It is well recognized that soils can be highly heterogeneous, which can have consequences for soil communities and the processes that they drive (Ettema & Wardle, 2002; Gundale et al., 2006). A recent Letter by Reinhart & Rinella (2016) criticized a large number of published glasshouse studies that tested the effects of soil biota on plant performance for failing to consider and properly incorporate soil heterogeneity into experimental treatments (Van der Putten et al., 1993; Nijjer et al., 2007; Felker-Quinn et al., 2011; Pendergast et al., 2013; Rodríguez-Echeverría et al., 2013; Yang et al., 2013; Gundale et al., 2014a; Pizano et al., 2014; Hilbig & Allen, 2015; Larios & Suding, 2015). Specifically, they argued that the practice of combining soils from plots, stands, or sites to create experimental treatments (referred to as 'Mixed Soil Sampling' or 'MSS') eliminates important residual error that dramatically increases Type I statistical error rates (i.e. the finding of differences when they are not present) compared to the practice of keeping soils separate (referred to as 'Independent Soil Sampling' or 'ISS'). A subsequent dialogue between Cahill et al. (2017; in this issue of New Phytologist pp. 11–14) and Rinella & Reinhart (2017; in this issue of New Phytologist pp. 15–17) articulated substantial disagreement regarding the effectiveness and applicability of MSS and ISS approaches. Here, we identify several additional concerns regarding Reinhart & Rinella's (2016) suggestion that studies using MSS are always flawed, and that ISS should always be used instead. Reinhart and Rinella's simulation model introduced a very small amount of experimental noise compared to most real glasshouse experiments focused on soil biotic effects on plants. In their study, an experimental subject consisted of a pot filled with a mixture of commercial potting soil and sand, and plants grown from a commercial seed source, neither of which likely represented the variation in soil physical properties or plant genetic diversity found in the geographic range of their study sites (Reinhart et al., 2010). Thus, in their simulation, as well as the experimental work underpinning it (Reinhart et al., 2010), the main source of noise is the soil biota treatment itself and a small amount of random glasshouse variation, whereas other sources of ecologically relevant noise were excluded. By contrast, a researcher conducting real glasshouse experiments intentionally or unintentionally introduces other sources of noise into experiments, such as by creating experimental subjects consisting of genetically diverse individuals (Felker-Quinn et al., 2011; Gundale et al., 2014a; Hilbig & Allen, 2015) or soils with variable physical properties (Gundale et al., 2014a) such as is found in nature. Fig. 1 illustrates four different approaches researchers can take to incorporate or exclude experimental noise into glasshouse experiments focused on soil biota. Experimental scenarios B and D represent experiments where plot-to-plot, stand-to-stand, or site-to-site variability in soil communities are incorporated into soil biota treatments (i.e. the ISS approach), whereas experimental scenarios C and D represent experiments where a researcher has actively decided to incorporate ecologically relevant variation into the experimental subjects (i.e. the plant–soil unit receiving the inoculum treatment). We sought to understand how sensitive Type I error rates estimated by Reinhart & Rinella's (2016) model were to the inclusion of additional experimental noise (Supporting Information Notes S1, S2). First, we ran their model 1000 times both with pathogen presence set to zero (to simulate a sterile soil control) and with pathogen presence set according to Reinhart and Rinella's simulation (i.e. 0.3). This showed that the MSS approach had a coefficient of variation (CV) of 0.12 and 0.26 without and with pathogens, respectively, and the ISS had a CV of 0.13 and 0.30 without and with pathogens, respectively. We then compared these values to the results obtained from a range of studies that have used either MSS or ISS approaches, including several MSS studies that were criticized by Reinhart and Rinella (Table 1). Studies that included a sterile control showed CVs on average about three-fold greater than the sterile control we generated within Reinhart and Rinella's model (i.e. representing soils without any pathogens present). We also found that both the MSS and ISS approaches on average added a relatively small amount of noise to studies (on average 2% and 11% additional noise introduced by the MSS and ISS approaches, respectively), compared to a c. 125% and 131% increase in noise caused by the MSS and ISS methods, respectively, in Reinhart and Rinella's model (Table 1). Because Reinhart and Rinella appeared to underestimate the importance of background noise, and overestimate the importance of noise derived from soil biotic treatments, we sought to understand how sensitive their conclusions were to adding experimental noise. We found that incrementally increasing experimental noise in the simulation caused Type I errors of the MSS approach to sharply decline, converging with the ISS approach when noise was equivalent to approximately three times (i.e. 300%) the original standard deviation (Fig. 2a,b). While this analysis does not necessarily provide a justification for using the MSS approach, it suggests that Reinhart and Rinella's conclusion that all studies that have used MSS are 'fatally flawed' or have reached erroneous conclusions is exaggerated. Concerned with the generalization that the MSS approach is 'fatally flawed' (Reinhart & Rinella, 2016), Cahill et al. (2017) proposed that the ISS approach also has major limitations that should be considered. Cahill et al. (2017) altered Reinhart and Rinella's model by setting mean pathogen densities in two regional populations to be different, and found that the MSS approach was more successful at detecting those known differences, whereas the ISS approach was rarely able to find differences. Rinella & Reinhart (2017) responded by suggesting that the ISS approach had sufficient power to detect differences between regions if differences were large enough to be biologically meaningful. We performed an alternative approach to that employed by Cahill et al. (2017) to evaluate Type II errors of MSS and ISS by creating two regions that were unambiguously different (Fig. 2c). In contrast to Cahill et al. (2017), we altered pathogen presence probabilities instead of pathogen densities. Specifically, we randomly generated two regions each consisting of 10 000 individual soils from which we could sample, and with pathogen presence probability in the first and second region of 0.3 and 0, respectively (Notes S1, S3). Complementary to the result of Cahill et al. (2017), our analysis showed the MSS approach was more successful at detecting differences between the two regions (Fig. 2d), indicating that the MSS approach had a substantially lower Type II error rate (i.e. false negatives). We also ran the model by setting pathogen presence probabilities of both regions to be equal (i.e. 0.3 in both regions), in order to repeat the Type I error analysis done by Reinhart & Rinella (2016). This analysis, together with the Type II error analysis, showed that both the MSS and ISS approaches have disadvantages when random sampling is used to make an inference about populations. The MSS approach too often showed differences when they were not present, and the ISS approach only infrequently detected differences when they were clearly present (Fig. 2d), highlighting limitations of both approaches within the context of Reinhart & Rinella's (2016) model. When a researcher selects a few individuals or sites to represent an entire population of individuals or sites, there is always the possibility that the selected individuals are not representative, especially when sample sizes are small, as noted by Cahill et al. (2017), and the population that is sampled is heterogeneous. We agree with Reinhart & Rinella (2016) and Rinella & Reinhart (2017) that the MSS approach is more likely to show a difference between the sampled groups even when the populations that are sampled from are not different because of lower variance (although as we conclude earlier, this is a less severe problem than they claim). However, an alternative interpretation of Reinhart and Rinella's analysis is that the MSS approach is actually more sensitive at identifying differences between sample groups (as opposed to identifying differences between entire populations), and that the Type I errors identified within Reinhart and Rinella's model only occur when the results of the sample group comparison are extrapolated to the larger population that was sampled. We assert that not all studies or research goals focused on plant–soil interactions involve randomly sampling a larger population, so the mechanism that generates Type I errors identified by Reinhart and Rinella's model is not always relevant, as we now discuss. We suggest that the field of plant–soil feedback interactions would benefit from clarifying which questions can best be addressed by the use of different types of experimental design (Fig. 1). We agree with Reinhart and Rinella that the ISS approach is necessary for understanding how soil biotic effects on plant growth vary across space, and when samples are meant to be representative of a broader population that was sampled. But, as noted in numerous studies that have used MSS approaches, spatial variability of soil community effects is often not the particular question of interest (Pizano et al., 2011; Rodríguez-Echeverría et al., 2013; Gundale et al., 2014a; Kardol et al., 2014; De Long et al., 2015; Hilbig & Allen, 2015; Larios & Suding, 2015), and not all hypothesis testing has the goal of inferring beyond their specific study system. For studies where spatial variability of effects is not a focal question, and a researcher does not wish to extrapolate results beyond a particular study system, experimental scenarios A or C (Fig. 1) are preferred approaches because MSS is more successful at revealing actual differences between two groups. For example, we believe MSS is highly appropriate for comparing a specific set of sites or stands in a species native range that are the known source of a specific set of sites or stands in a species new range, because such a comparison does not involve random sampling a population or extrapolating results beyond the specific study system (see Gundale et al. (2014a) as an example). By contrast, if nothing is known about an exotic or invasive species exact origin, it would be necessary to randomly sample native and introduced regions in order to establish study sites. Such a study design would thus require the ISS approach (as well as a large sample size), to reveal actual differences if they were present in the sampled populations (but see Gundale et al. (2014b) for a discussion on problems associated with random sampling for making biogeographical comparisons). Contrary to the assertions by Reinhart & Rinella (2016), we maintain that the MSS approach is not 'fatally flawed' as long as the experimental subjects (i.e. the plant–soil units subjected to the treatment) within the actual experiment are sufficiently independent and randomly dispersed (Hurlbert, 1984). Further, contrary to claims by Reinhart & Rinella (2016), in the study by Van der Putten et al. (1993), treatment variability was not included (i.e. MSS) when providing proof of principle (i.e. the first phase of investigation), but was included (i.e. ISS) when providing further evidence. This study provides a good example of how MSS and ISS can both be effectively used at different stages of an investigation, where a different goal of inferring beyond or within the experimental system may exist. A researcher also can decide whether to exclude (experimental scenarios A or B) or include (experimental scenarios C or D) ecologically-relevant variation among experimental subjects (i.e. the plant–soil unit receiving the inoculum treatment). These decisions can likewise impact upon whether results are relevant and transferable to a broader population of subjects, or only to the subjects that were used in the particular experiment. Experimental scenario A is not necessarily technically incorrect, but provides a researcher with the least information, where nothing is learned about how soil biota effects vary across space, or the variation in subjects in which those effects are likely to be relevant (i.e. plants or soil abiotic properties that are relevant to the defined geographic study area). Experimental scenario B (such as used by Reinhart et al., 2010), can provide inference to a broader population of soil communities in the defined study area, but provides no information on the range of subjects (i.e. the plant–soil unit receiving the treatment) for which those effects are relevant. Experimental scenario D has the potential to provide the most information, by revealing both the spatial variation of soil biota effects and a greater contextual breadth for the types of subjects for which those effects are relevant. However, we emphasize that experimental scenario D would likely require a substantially greater number of experimental units relative to the other experimental approaches (and therefore time and money) in order to minimize Type II errors because of the large amount of experimental noise that it incorporates, which is more likely to result in incorrect conclusions that soil biotic composition is unimportant for plant performance. We therefore maintain that researchers should not be coerced into a specific experimental design that includes the largest amount of imaginable environmental variation when that approach is not always justified by a specific research question or goal and when it is likely to inflate Type II errors. The authors thank N. Fanin and J. U. Diez and four anonymous reviewers for helpful comments on a previous draft of this manuscript. We also thank Pella Brinkman for compiling literature presented in Table 1. All authors contributed to ideas developed in the manuscript. M.J.G wrote the manuscript, with input and discussion from co-authors. R.W.L and M.J.G. performed the model analysis. Please note: Wiley Blackwell are not responsible for the content or functionality of any Supporting Information supplied by the authors. Any queries (other than missing material) should be directed to the New Phytologist Central Office. Notes S1 Methods: addition of noise to Reinhart and Rinella's model, and Type II error rates of the ISS vs MSS approaches. Notes S2 R code used to add experimental noise to Reihart & Rinella's (2016) model. Notes S3 R code used to compare two regions with different pathogen presence probabilities. Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.