# An ecocultural model predicts Neanderthal extinction through competition with modern humans

See allHide authors and affiliations

Contributed by Marcus W. Feldman, December 21, 2015 (sent for review October 6, 2015; reviewed by Magnus Enquist, Stefano Ghirlanda, and Franz J. Weissing)

## Significance

Ecocultural niche modeling and radiocarbon dating suggest a causal role for interspecific competition in the extinction of Neanderthals. Most archaeologists argue that the advantage to modern humans lay in a higher culture level (a sizable minority dispute this view). Competition between the two species may have occurred when a modern human propagule entered a region occupied by a larger Neanderthal population. We present a model for this replacement, stressing the importance of the founder effect. Our findings shed light on the disappearance of the Neanderthals, showing that endogenous factors such as relative culture level, rather than such extrinsic factors as epidemics or climate change, could have caused the eventual exclusion of a comparatively larger population by an initially smaller one.

## Abstract

Archaeologists argue that the replacement of Neanderthals by modern humans was driven by interspecific competition due to a difference in culture level. To assess the cogency of this argument, we construct and analyze an interspecific cultural competition model based on the Lotka−Volterra model, which is widely used in ecology, but which incorporates the culture level of a species as a variable interacting with population size. We investigate the conditions under which a difference in culture level between cognitively equivalent species, or alternatively a difference in underlying learning ability, may produce competitive exclusion of a comparatively (although not absolutely) large local Neanderthal population by an initially smaller modern human population. We find, in particular, that this competitive exclusion is more likely to occur when population growth occurs on a shorter timescale than cultural change, or when the competition coefficients of the Lotka−Volterra model depend on the difference in the culture levels of the interacting species.

Neanderthals are a human species (or subspecies) that went extinct, after making a small contribution to the modern human genome (1, 2). Hypotheses for the Neanderthal extinction and their replacement by modern humans, in particular as recorded in Europe, can be classified into those emphasizing competition with modern humans and those arguing that interspecific competition was of minor relevance. Among the latter are the climate change (3) and epidemic/endemic (4) hypotheses. However, an ecocultural niche modeling study has shown that Neanderthals and modern humans exploited similar niches in Europe (5), which, together with a recent reassessment of European Paleolithic chronology showing significant spatiotemporal overlap of the two species (6), suggests a major role for interspecific competition in the demise of the Neanderthals.

Replacement of one species (or population) by another is ultimately a matter of numbers. One competing species survives while the other is reduced to, or approaches, zero in size. In the classical Lotka−Volterra model of interspecific competition, this process is called competitive exclusion (7). If Neanderthals were indeed outcompeted by modern humans, the question arises: Wherein lay the advantage to the latter species? Many suggestions have been made, including better tools (8), better clothing (9, 10), and better economic organization (11). These hypotheses share the premise that modern humans were culturally more advanced than the coeval Neanderthals.

The purpose of our paper is threefold. First, we extend the Lotka−Volterra-type model of interspecific competition by incorporating the “culture level” of a species as a variable that interacts with population size (12, 13). Here, culture level may be interpreted as the number of cultural traits, toolkit size, toolkit sophistication, etc. Although, as noted above, many anthropological and archaeological discussions invoke interspecific cultural competition, there is, to the best of our knowledge, no mathematical theory of this ecocultural process. A mechanistic resource competition model is difficult to justify at present, because there is a limited understanding of “what the species are competing for… [or] how they compete” (14). Second, we use our interspecific cultural competition model to explore, analytically and numerically, the possibility that a difference in culture level, or in underlying learning ability, may produce competitive exclusion of a comparatively (although not absolutely) large regional (Neanderthal) population by an initially smaller (modern human) one. Third, we assume the competition coefficients of the Lotka−Volterra model to depend explicitly on the difference in the culture levels of the interacting species (rather than to be constants) and ask how this modification affects the invasion and subsequent dynamics.

Dependence of the culture/technology level of a human population on its size has been the focus of many theoretical (15⇓⇓⇓⇓⇓–21) as well as psychological (22⇓–24), archaeological (25, 26), and ethnological (27⇓⇓–30) studies. However, the coupled dynamics of population size and culture level, where both quantities are treated as variables, has received less theoretical attention (12, 13, 31, 32).

Taking refs. 12 and 13 as the point of departure, we extend previous treatments by introducing two such populations in direct competition with each other in the Lotka−Volterra framework. The two populations are described in terms of their size, *N*_{i} (≥0), their culture level *z*_{i} (≥0), *i* (=1, 2), and parameters to be defined below. We ask whether a population can be replaced by an initially smaller one, which has an advantage in culture level or in learning ability. This ecological perspective on the competition between “size−culture profiles” may inform ongoing debate on the replacement of Neanderthals by modern humans.

## Theoretical Model

Our model is a four-variable system of coupled differential equations,

Eq. **1** entails logistic growth of population *i* subject to competition from population *j*, where *i* has carrying capacity **1** with **2** was originally introduced in ref. 12 as a minimal model for the effect of population size on culture level. Parameter

For the carrying capacity of population *i*, we assume the step function**3** can be made slightly more general by interpolating a gradual continuous transition between the lower carrying capacity, *SI Text*.

## Results

### Equilibrium Properties.

We formulate the question of replacement of Neanderthals by modern humans in terms of the instability of an equilibrium involving the size−culture profile of the former to an invading propagule of the latter that may be smaller in numbers but more advanced culturally or in underlying learning ability. Equilibria of the model are obtained by setting Eqs. **1** and **2** to zero. Local stability can be determined from the Jacobian. We introduce the shorthand notation *i* at equilibrium, which equals either **3**). Here, we summarize the kinds of equilibria and their local stability conditions for general parameter values when the competition coefficients are constants. Later, we consider the case where

First, an internal (i.e., coexistence) equilibrium *Local Stability with Constant Competition Coefficients*. In the Lotka−Volterra model, this condition ensures global stability of the (unique) coexistence equilibrium. Second, an edge equilibrium *Local Stability with Constant Competition Coefficients*. Finally, the corner equilibrium

### Case of Equal Parameter Values.

We first investigate the equilibrium properties of the model (Eqs. **1**−**3**) assuming cognitive equality of Neanderthals and modern humans. Suppose that the populations are equivalent with respect to the cognition parameters,

There are three different kinds of internal equilibria. If **3** entails **2**, we have **4** entails local stability of this equilibrium if

Similarly, a “high” symmetrical internal equilibrium**2**.

The third kind of internal equilibrium is asymmetrical. Suppose, without loss of generality, **8**—will exist when inequalities **9** and **10** are satisfied. Thus, four stable coexistence equilibria are possible in this special case of our model. Moreover, they may exist and be locally stable at the same time.

The edge equilibria can also be identified. The “low” edge equilibrium,**5**, both are locally stable if **10** is reversed.

Thus, the equilibrium structure of our model is quite complex. An edge equilibrium corresponds to the presence of just one of the populations (population 1, say), and is (locally or globally) stable in the absence of the other (i.e., population 2) (13). However, it may or may not be stable when population 2 is introduced. In the replacement problem, we equate populations 1 and 2 with Neanderthals and modern humans, respectively. We show in *Numerical Analysis with Equal Parameter Values* that the subsequent dynamics could result in coexistence or in competitive exclusion of population 1, either of which can happen when the initial value of

### Numerical Analysis with Equal Parameter Values.

To illustrate the dynamics of competition between the size−culture profiles of Neanderthals and modern humans, we carry out a numerical analysis under the above simplifying assumptions. We assume the step function model with equal parameter values in the two populations. Because the dynamical system in Eqs. **1**−**3** is four-dimensional, it is difficult to generate a simultaneous 2D pictorial representation of the basins of attraction for all of the equilibria described above. For simplicity, we focus on the case where the system is initially at the low edge equilibrium (Eq. **11**), corresponding to our premise that the Neanderthals in isolation would have formed a comparatively small population at a relatively low culture level. Figs. 1 and 2 illustrate the basins of attraction in the **2** with

Positive deviations [**14**) but whose culture level has not yet declined to a value commensurate with its reduced size. Using a “serial founder effect” model, it has been argued that such a propagule may be about 0.09–0.18 the size of the parental population (34). Figs. 1 and 2 show that population 1 can be competitively excluded by an initially smaller population 2, provided the latter maintains an advantage in culture level.

Each colored region in Figs. 1 and 2 corresponds to the basin of attraction for a distinct equilibrium. Seven equilibria exist given the parameter values in Fig. 1. The values of the parameters *K* and *D* in the numerical analysis do not represent known carrying capacities; rather, they should be regarded as being measured in some unspecified units. The two symmetrical coexistence equilibria (Eqs. **6** and **7**) and the two high edge equilibria (Eqs. **13** and **14**), are locally stable, but only two of these—Eqs. **6** and **14**—are attractors from the initial conditions considered here. The low edge equilibrium **11**, representing the state of the system before the introduction of population 2, is unstable. Hence, population 2 will necessarily invade, and the question is whether convergence occurs to equilibrium **6**, resulting in coexistence, or to equilibrium **14**, resulting in competitive exclusion. The purple and blue regions in Fig. 1*B* denote the basins of attraction of the low symmetrical coexistence equilibrium **6** and the high edge equilibrium **14**, respectively.

To the left of the white broken line in Fig. 1*B*, population 2 is initially smaller than population 1. Fig. 1*B* shows that competitive exclusion of population 1 by population 2 is possible even when the latter is initially smaller than the former. For example, convergence to the high edge equilibrium **14** (black disk in the upper right-hand corner) can occur from an initial population ratio of *B*), then convergence to the low symmetrical internal equilibrium **6** (black disk in the lower left-hand corner) results. Importantly, our ecocultural model—in contrast to the standard Lotka−Volterra model—predicts the simultaneous existence of locally stable internal and edge equilibria, where the former may act as “traps” hindering competitive exclusion. Fig. 1*A* shows the corresponding trajectories.

Fig. 1 *A* and *B* illustrates the default case where the timescales of demographic and cultural change are equal, specifically *B* suggests that *C* shows that the minimum *γ* and *δ* together, subject to the constraint

Fig. 2 shows the effects of varying the competition coefficient, *b*. Fig. 2*A* (*Middle*) is equivalent to Fig. 1*B*. For low values of *b* satisfying inequality **10**, the two asymmetrical internal equilibria, **8** and its counterpart, exist and are locally stable. Thus, all nine corner, edge, and internal equilibria exist for the parameter values in Fig. 2*A* (*Left*). However, only two attractors are found, and the green region of Fig. 2*A* (*Left*) denotes the basin of attraction of the asymmetrical internal equilibrium **8** (where *b* increases, the asymmetrical coexistence equilibria undergo transcritical bifurcation into the high edge exclusion equilibria, as shown in the bifurcation diagram Fig. 2*B*.

Eventually, at even larger values of *b*, as in Fig. 2*A* (*Right*), all four coexistence equilibria are eliminated, while the four edge equilibria become locally stable. Here, the attractors are the low edge equilibria **11** and **12**, and the high edge equilibrium **14**; the basins of attraction are colored in light yellow, light blue, and blue, respectively. The light yellow region corresponds to noninvasion, whereas the light blue and blue regions both correspond to competitive exclusion of population 1 by population 2. The sizes of the populations at equilibrium, *b*, as shown in Fig. 2*B*.

### Difference in Learning Ability.

The jury is still out on whether or not Neanderthals and modern humans differed in cognition (35, 36). Aspects of cognition incorporated into our model are the infidelity of social learning, *Effect of a Large Difference in Learning Ability*, inequality **15** together with**15** ensures a globally stable small population−low culture level equilibrium for population 1 in isolation and a globally stable large population−high culture level equilibrium for population 2 in isolation (13).

### Feedback of Culture Level on Competition.

Ecological competition coefficients between populations with different culture levels may be functions of this difference, rather than simple constants. We return to our assumption that the two populations are equivalent with respect to the cognition, competition, and growth parameters [*ε* are positive constants, and *i* and *j* (**1** to be meaningful, *ε* must be chosen small enough that

In Eq. **16**, **16** into Eq. **1** induces positive feedback, whereby a population (2, say) that enjoys a small initial cultural advantage (i.e., **2**, which in turn further increases its competitive edge. Hence, we expect and find that this feedback enhances the effect of an initial cultural advantage (Fig. 3*A*).

The feedback model has the same edge and symmetric coexistence equilibria as occur in the model without feedback. The conditions for the existence of these equilibria are unchanged. The conditions for their local stability are given in *Local Stability with Feedback*. Fig. 3*A* shows that the minimum *ε* increases. Fig. 3*B* is a phase diagram illustrating when different solutions exist and are stable for different values of *ε* or *B*) where coexistence becomes unstable when

Somewhat surprising is the existence and stability of a new class of asymmetrical internal equilibria. These equilibria occur in pairs, either both above or below the step (*Asymmetrical Internal Equilibria with Feedback* derives analytical conditions for their existence and reports a perturbation analysis suggesting that existence implies local stability. However, determination of local stability proves difficult in general, and we check for local stability by numerically obtaining the eigenvalues of the characteristic polynomial. We find that these solutions are, in fact, locally stable whenever they exist (at least for the parameter values assumed in Fig. 3*B*, blue and red regions).

## Discussion

It is likely that Neanderthals went extinct and were replaced by modern humans due to interspecific competition for overlapping resources. Modern humans are believed to have gradually expanded their range into areas inhabited by Neanderthals (and other archaic humans) by a process of iterative propagule formation. This serial founder scenario receives support from a genetic study showing a regular reduction in heterozygosity with distance from a putative origin in Africa (37). The size of each such propagule may have been about 0.09–0.18 that of the parental population (34). These considerations suggest that a modern human group would have been smaller than a Neanderthal one at initial contact. Our paper reports the theoretical conditions under which such a numerical disadvantage could have been more than compensated by an advantage in culture level or learning ability. Note that in ref. 5, the term “ecocultural” refers to the relationship between environmental niches occupied by Neanderthals and modern humans, overlap in which may have led to competition.

In deriving these theoretical conditions, we have made two critical assumptions, which we now discuss. First, we have assumed that the culture level of the invading modern human population at first contact would have shown systematic positive deviations from the quasi-equilibrium value. Specifically, we assume that propagule formation does not entail a significant loss of cultural traits due to the founder effect, as it may for genetic variation. The fundamental difference between genetic and cultural transmission in humans is that an individual inherits only a small subset of the genetic variation present in the population into which he/she is born (unless that population is highly inbred), but there is no such intrinsic limitation on the acquisition of cultural traits (38).

A caveat that immediately comes to mind here is division of labor. However, division of labor may not be pronounced in hunter−gatherer societies except between the sexes, and a viable propagule that comprises both reproductive males and females should initially lack few of the cultural traits in its parental population, and its culture level may therefore approximate that of the parental population. This situation may be maintained if time between propagule formation and first contact is short. (Eventually, quasi-equilibrium will be reached according to Eq. **2**, but, by then, the propagule will also have grown in size.) On the other hand, expert makers of sophisticated artifacts are likely few, so the cultural founder effect (and hence random cultural drift) cannot be completely neglected (39).

Second, we have assumed that the timescales of demographic and cultural change may differ, specifically that the potential rate of population growth may be an order of magnitude higher than the potential rate of cultural change. Richerson et al. suggest that this may be true for the Late Pleistocene and most of the Holocene (32). In extant hunter−gatherers, the intrinsic growth rate (**2** were perhaps large but of similar magnitude—or because the cultural decay rate itself was small.

The other observation concerns the variation in Acheulean handaxe dimensions produced (by hominids ancestral to both Neanderthals and modern humans) over a time span of 1.2 million years. Based on a simulation model, Kempe et al. suggest that this is consistent with a copying error rate of 0.17% per generation, which is low compared with the intrinsic population growth rate and supports our assumption (43). The error rate associated with simple social learning has been estimated in laboratory transmission chain experiments (with extant humans as subjects). In one such experiment (43), each participant in a transmission chain was tasked to view and faithfully reproduce, on an iPad screen, the size of a handaxe image made by the previous participant. The estimated error rate was of the order of 3% per act of copying. Kempe et al. suggest possible reasons for this major discrepancy; for example, unlike the Paleolithic knappers, the experimental subjects could not actually handle their artifacts and thereby reduce the copying error rate.

These assumptions (minimal cultural founder effect, different timescales of demographic and cultural change) were not invoked in our analysis of a difference in learning ability. We found that replacement of Neanderthals by modern humans was assured, given a large advantage in learning ability to the latter (inequality **15**) and provided an auxiliary condition was also met. However, it may be unrealistic to assume an innate cognitive difference of this magnitude between these two closely related species. A smaller difference in learning ability may be sufficient to drive the replacement process, if the assumptions on cultural dynamics discussed immediately above apply even partially.

Although our model defined by Eqs. **1**−**3** is simple, it preserves the qualitatively important features arising from the interaction between population size and culture level that are seen in more detailed models (13). However, two other assumptions also made in the interest of simplicity require comment here. First, we have neglected cultural/technological transfer between the competing populations. Acculturation of European Neanderthals by modern humans has been a hotly debated issue (44, 45), and, in eastern Eurasia, there is strong evidence for archaeological continuity, perhaps due to the reverse acculturation of incoming modern humans by the resident archaic humans (46, 47). Second, we have ignored the demographic and cultural consequences of interbreeding. Introgression of Neanderthal genes is small in scale (1, 2) but may have affected male fertility (48). Clearly, these considerations must be kept in mind.

The replacement process predicted by our ecocultural model can be “self-perpetuating.” That is, given moderate competition (*A*, *Middle*). Hence, it will be in a position to generate a new propagule to invade the next area still occupied by Neanderthals. The resulting spatial effects could also have important consequences.

## SI Text

### Local Stability with Constant Competition Coefficients.

Assume the ramp function**3**, and reduces to it when

For an internal equilibrium, the Jacobian is**3**.

For an internal equilibrium

For an edge equilibrium

### Effect of a Large Difference in Learning Ability, γ i / δ i .

Assume that the two populations have identical competition and growth parameters, **3**; however, they differ in their learning abilities, specifically in the ratio

The low symmetrical internal equilibrium **6**. The high symmetrical internal equilibrium **7**. The asymmetrical internal equilibria **10**. The low edge equilibrium **12**. The high edge equilibrium **13**. Finally, the high edge equilibrium

### Local Stability with Feedback.

Assume the ramp function model for the carrying capacity, equal parameter values in the two populations, and competition coefficients given by Eq. **16** that depend on the difference in culture levels. The characteristic polynomial is*λ*.

For a symmetrical internal equilibrium where *ε* is a hyperbolic function of

We deal with asymmetric internal equilibria in *Asymmetrical Internal Equilibria with Feedback*.

The Jacobian for an edge equilibrium of the ramp function model

### Asymmetrical Internal Equilibria with Feedback.

As before, we assume the ramp function model for the carrying capacity, equal parameter values in the two populations, and competition coefficients given by Eq. **16** that depend on the difference in culture levels. For an internal equilibrium, we have, in general,*ε* is small. An internal equilibrium with *ε* remains small. Hence, the asymmetrical internal equilibrium is locally stable to first order in *ε* if

At equilibrium, we have

Local stability is difficult to show, in general. Here, we use a perturbation argument to prove that an asymmetrical internal equilibria of this class is locally stable, provided it is located in the parametric neighborhood of either a symmetrical internal equilibrium or an edge equilibrium. Substituting for

Next we note that when *ξ* so that

Similarly, when *η*.

## Acknowledgments

This research was supported in part by a National Science Foundation Graduate Research Fellowship (to W.G.), the John Templeton Foundation (M.W.F.), and Monbukagakusho Grant 22101004 (to K.A.).

## Footnotes

- ↵
^{1}To whom correspondence should be addressed. Email: mfeldman{at}stanford.edu.

Author contributions: W.G., M.W.F., and K.A. designed research, performed research, contributed new reagents/analytic tools, analyzed data, and wrote the paper.

Reviewers: M.E., University of Stockholm; S.G., Brooklyn College; and F.J.W., University of Groningen.

The authors declare no conflict of interest.

This article contains supporting information online at www.pnas.org/lookup/suppl/doi:10.1073/pnas.1524861113/-/DCSupplemental.

Freely available online through the PNAS open access option.

## References

- ↵.
- Shunkov MV,
- Molodin V

- Viola B,
- Pääbo S

- ↵
- ↵
- ↵
- ↵
- ↵
- ↵.
- Murray JD

- ↵.
- Akazawa T, et al.

- Bar-Yosef O

- ↵
- ↵
- ↵
- ↵
- ↵.
- Aoki K

- ↵
- ↵
- ↵
- ↵
- ↵
- ↵.
- Lehmann L,
- Aoki K,
- Feldman MW

- ↵
- ↵.
- Mesoudi A,
- Aoki K

- Fogarty L,
- Wakano JY,
- Feldman MW,
- Aoki K

- ↵
- ↵
- ↵
- ↵
- ↵.
- Klein RG,
- Steele TE

- ↵.
- Collard M,
- Kemery M,
- Banks S

- ↵
- ↵.
- Kline MA,
- Boyd R

- ↵
- ↵.
- Coleman D,
- Schofield R

- Lee RD

- ↵
- ↵
- ↵.
- Deshpande O,
- Batzoglou S,
- Feldman MW,
- Cavalli-Sforza LL

- ↵
- ↵.
- Zilhão J, et al.

- ↵.
- Ramachandran S, et al.

- ↵.
- Cavalli-Sforza LL,
- Feldman MW

- ↵
- ↵.
- Cavalli-Sforza LL

- ↵.
- Excoffier L,
- Schneider S

- ↵
- ↵
- ↵
- ↵
- ↵
- ↵.
- Mesoudi A,
- Aoki K

- Kobayashi Y,
- Kadowaki S,
- Naganuma M

- ↵

## Citation Manager Formats

## Article Classifications

- Biological Sciences
- Evolution

- Social Sciences
- Anthropology