Modeling the unpredictable: lessons from developing decision support tools for COVID19 in Mexico

Series Fortune Fortuna Painting By Tatiana Siedlova

When will the outbreak of COVID19 end? Have we flatten the curve? When is it safe to reopen the economy? These are questions that are hunting Mexico right now and which have spurred a heated debate all over the country. To answer these inquiries in a policy relevant way, we need to produce analysis that is fundamentally different to what we are used to.

This note summarizes the work that some colleagues and I at Tecnologico de Monterrey have done for developing a simulation model for studying COVID19 in Mexico. The purpose of this model is to help state authorities and the public understand how the pandemic is progressing in their region, estimate plausible near-term trajectories and understand in more detail the risks of reopening the economy. During the development of this tool, we have come across a wide range of talented analysts and decision makers and, as a result, we have learnt important lessons that are worth sharing.

Any pandemic outbreak is uncertain. However, in the case of COVID19, the inherent epidemiological uncertainty of the virus is exacerbated by the unpredictable nature of human behavior, the difficulties of explaining complex information to the public and by politics. Attempting to keep up with the pandemic is one of the most complicated analytical challenges we have faced.

Our model (EgobiernoyTP-SIR) is a classic SIR model, expanded to describe the dynamics of five population groups: 1) susceptible, 2) infected, 3) recovered, 4) hospitalized and 5) deceased. The model has two more important features: 1) it is designed to model a wide range of policy containment options (e.g., social distancing, contact tracing, isolation, testing and vaccination), and 2) the core assumption in the model is that we treat the reported number of confirmed cases and deceased population as delayed information on real developments. You can learn more about the model by visiting the mexicovid19 project website and the github repository where a public version of the model resides.

The assumptions described above sound reasonable. However, is our model perfect? NO, IT IS NOT. No model is. Simulation models are perfectible and can always be improved. Ours in particular can be improved by disaggregating population cohorts and by increasing the granularity of the policy interventions (e.g. wearing masks). Areas in which we are working on right now.

The above does not preclude the model from being useful as a decision support tool. In my view, to be able to develop useful and effective decision support tools. The first order of business is to test the model against historical conditions. If a simulation model is not able to reasonably replicate baseline conditions, it is very difficult to trust the analysis derived from the model. Calibrating the model in a rigorous way helps improve the model, understand its limitation and increase confidence on its output.

Given the importance and public sensitivity of COVID19, we chose to test our model against both data on confirmed cases and confirmed fatalities across Mexico’s 32 states. We did this to ensure that our model provided a plausible description of how the pandemic is occurring in Mexico. This is no small feat, replicating both records at a regional level means the our mathematical description of the pandemic (i.e., system of differential equations) needs to conform with both data sets, and thus provide a harmonized description of what is happening in every region.

The graph below shows an example of this exercise. It compares our model’s behavior (orange lines) against the historical number of confirmed cases and fatalities (blue lines) in Mexico City. You can see that our model is able to replicate reasonably well how the pandemic has behaved in this region. Once we find the combination of parameters that is able to do this, we tag this run as “baseline condition”. Getting the model to be able to do this for 32 states required many late nights of coding. We ultimately accomplished this by expanding and improving the mathematical structure of the model several times, using mobility data records from cellular devices, and by implementing a multivariate version of the Maximum Likelihood (MLE) method with Theil’s decomposition criteria, and we are not done yet, as this process of calibration is constantly pointing at ways in which we can improve our work.

This feature of the model does not mean the model is perfect, or that it can be used a crystal ball for predicting what is going to happen with the pandemic. Meeting this minimum threshold of quality, fundamentally means the model can be used as a laboratory for experimentation (i.e., it is partially valid). After this, a lot improvements are derived from socializing the results with experts and stakeholders, whose criticism has helped us resolved inconsistencies in our work. The combination of both: rigorous statistical calibration and critical feedback has helped us improved the model a lot.

The lessons derived from experimenting with the model are particularly useful for understanding where we are in the pandemic, how we got here and what are the critical tradeoffs forward. I cannot emphasize this point enough, especially for the case of COVID19. The progression of the pandemic is sufficiently complex and volatile that any prediction endeavor is destined to fail. Yet, we see in our tweeter time lines and newspapers constant prophesies about COVID19. In the worse cases, we see public officers publicizing exact days at which the pandemic will end. This creates a lot of anxiety among the public and decision makers, and, of course, frustration when these prophecies do not come to past.

The model I described in the previous paragraphs has several applications. The first is that it can be used to estimate range values for the real number of confirmed cases and the deceased population (i.e., using bootstrapping over the historical record). See for instance the figure below. The blue line shows the historical record for confirmed cases (left pane) and deceased population (right pane). The orange lines and box plots describe the range of values for both variables.

It should not come as a surprise that there is a difference between the historical record and the estimated real values. The value of this exercise resides on the fact that we can estimate a value range for both figures with some degree of statistical rigor. In this example, for Mexico City, by June 7 the estimated real number of cases ranged between 151 and 234 thousand, this is approximately five times higher than the historical record. For the deceased population, the estimated real value ranged between 13 and 21 thousand fatalities, this is four times higher than the historical record -also a testament of the dire human cost of COVID19-. There are a few nuances of these figures that are worthwhile commenting.

Can these estimates change? Absolutely, as new data becomes available, our estimations are deemed to change, it would be ludicrous to expect otherwise. Once more, we are not dealing with forecasting whether or not is going to rain tomorrow, we are dealing with a complex, rapidly changing socio-environmental phenomenon. Shall the public and decision makers be given these figures? Yes, I believe that is the correct course of action, the public should be informed as much as possible as we all need to make personal and professional decisions to adapt to COVID19. Is the difference between the estimated real and the confirmed number the result of a conspiracy? No, I do not think so. I have interacted over the past few months with several analysts and public officials that are in the frontlines dealing with this pandemic, and that interaction only confirms to me the heroism and professionalism of many of them. Yet, the reality is that without large scale testing, the uncertainty on the real number of deaths and cases increases, and the difference between confirmed and real cases becomes higher. The former point has not been explained clearly enough to the public.

One indicator that is commonly used for monitoring what is happening with the pandemic is the reproduction rate-R0-. In summary, this indicator describes the number of infectious contacts per time period each infectious person generates (i.e., cid), controlling by the probability that infected people encounter a susceptible person (S/N). The first component of this index depends on the average contacts between people (c), the probability of being infected after being in contact with an infectious person (i) and the average duration of the infection (d). The second component of this index depends on the total population (N) and the susceptible population (S). If R0 is greater than one, the pandemic is accelerating, if is lower, the pandemic is on a trajectory towards stabilization. It is important to note that when R0 is approximately one, the progression of the pandemic is extremely fragile, as changes in either direction can result in complete different outcomes (this is a tipping point). Thus, a clear policy objective is to first drive R0 towards one, once this milestone is achieved, the policy objective is either a) keep R0 close to one or b) continue driving R0 below one.

The figure below uses the results for Mexico City as an example of how we can experiment with this model to understand in more detail the situation of COVID19 in Mexico. The blue shaded area (left) describes the simulated baseline historical conditions for three variables: R0 (top row), confirmed cases (middle row) and mobility index (bottom row). This part of the analysis is mainly descriptive. It indicates that the reproduction rate of the virus decreased substantially during March, stayed stable during April and May, and began to grow during the first week of June. It is possible to see that the R0 is positively correlated with the mobility index we used for calibrating the model- the addition of mobility data has helped us reduce significantly the variance of the calibration process-. Thus, monitoring the mobility of the population seems to be a good proxy for the speed at which the virus is reproducing.

The white area (right) displays results of the simulation experiment we use to understand in more detail the status of the virus in the region. In this experiment we simulate the model 200 times (using Latin hypercube sampling) to explore how things can change in the coming days if the baseline parameters are negatively or positively biased. The purpose of this experiment is to study the implications of a higher/lower infectivity of the virus, higher/lower information delays and higher/lower population mobility rates on the three displayed variables.

These results show that the progression of the pandemic is still highly uncertain, yet some interesting patterns begin to emerge. In the top row we see that in a few cases the reproduction rate of the virus reaches a value below one, while in the majority of cases, the reproduction rate remains above this threshold. Further inspection, figure below, shows that the case under which the reproduction rate declines the most in this experiment is associated with a substantially lower infectivity rate of the virus (54% lower than that estimated in the baseline calibration) and a substantially lower level of mobility of the population (66% lower than the mobility rates before the beginning the outbreak). Yet, these conditions are associated with a higher number of confirmed cases (i.e., 61,521).

As shown in the figure below, since the confirmed number of cases is a delayed version of the real number of cases, this creates a discrepancy that is very difficult to digest for the public and for decision makers. In this case, the output is filtered to display only the results of the most optimistic simulation run in the experiment. The top row now displays the estimated real number of cases. We can clearly see that in this case, while the estimated real number of cases stabilizes, the delayed confirmed number of cases continues growing.

This discrepancy is a tremendous policy challenge, if we only use the confirmed number of cases to inform the public about the progress being made with the pandemic. The public is certainly going to be disappointed and frustrated, as the results of containment actions will only become apparent weeks after this took effect. This goes back to the point of testing. It has been argued that mass testing can help us cope and manage the pandemic more effectively for various reasons, yet in Mexico, and other countries, public officials are still reluctant to make mass testing available. I think this is mistake. The appeal of cutting down on testing resides on the fact that less testing, results in less confirmed cases, which may give the impression that the pandemic is under control. The downside of doing this is that the more you cut on testing, the longer it will take to publicly register the effect of containment actions, which induces more frustration in the public.

The previous scenario is also interesting because it shows that it is possible to reduce the reproduction rate of the virus and stabilize the number of infections if infectivity and mobility of the population are kept under control. Keeping people at home longer is going to be very difficult as many people are in dire need of getting back work, but reducing the infectivity of the virus can be achieved to some degree by wearing masks in public, washing our hands more frequently, keeping safe distance when we go out and by considering the level of risks of different economic activities (if you are interested on the latter point, this article provides useful analysis: https://tinyurl.com/y9nffef5 ). In Mexico City, as of July 2nd, the number of confirmed cases totaled approximately 48 thousand, a number lower than this best case scenario. This is a sign that the reproduction rate of the virus reduced during June, which is definitely good news, yet we need to update the analysis to know how fragile the situation is Mexico City really is.

We have done the previous analysis for all states in Mexico. The figure below summarizes the results. Columns denote months in the calendar and rows indicate states. The number noted in the cells indicates the percent of simulations that achieve stabilization of the outbreak (R0<=1) for the indicated month. For example, for Mexico City, this analysis shows that 20% of the simulations achieve stabilization in June, 30% in July and 37% in August. Thus, across the 200 simulations, by September 2020, 87% of these achieve stabilization.

When we look at other states, we can see that other regions that may be in a similar phase as Mexico City include Jalisco, Veracruz, Sinaloa, Yucatan and Guerrero. In contrast, states that may need longer times for stabilization include Colima, Baja California Sur, Morelos and Zacatecas.

At this moment, the majority of states are lifting restrictions, thus is essential we understand the level of risk that each of these regions faces during this phase of the pandemic. To look into this, we use the experiment to estimate the impact of higher or lower mobility rates on the reproduction rate of the virus at a regional level.

The figure below shows this last piece of the analysis. Columns denote changes in ranges for mobility rates with respect to the last known data point (June 6th). Columns of the right, denote scenario sets in which mobility rates increase with respect to the beginning on June. To the left, there are scenarios in which mobility rates decline. The color legend and value in the cells denote the mean reproduction rate from June 6th to July 7th.

These results give us an idea of how likely is that a state experiences an acceleration in the reproduction rate of the virus as restrictions are lifted (i.e., allows higher mobility rates). In this case, states that are able to absorb higher mobility rates are those with a lower risk of reopening. In this respect, the only state that seems to be in this situation is Quintana Roo, which shows that the mean reproduction rate of the state remains below 1.4 as long as mobility rates do not increase more than 50% compared to the baseline condition. Other states, such as Mexico City, should be careful about relaxing restrictions too abruptly or too soon. As shown in this table, an increase in mobility rates higher than 50% -compared baseline conditions- can result in a substantial increase in the reproduction rate of the virus. Finally, states such as Zacatecas, San Luis Potosi, Puebla, Estado de Mexico, Colima and Guanajuato appear to face the highest risk in reopening their economies as modest increases in mobility could result in substantial increases in the reproduction rate of the virus.

The results discussed in this note show that we are in very fragile moment. The majority of states face a difficult road ahead to contain the virus. Keeping social distancing restrictions will become more and more difficult as many people are in urgent need of getting back to work. What happens next? It is impossible to predict. Using advanced statistical tools to speculate with the tragedy is a waste of time. In a moment like this, analyses should better focus on understanding who is making progress and how the lessons learned in these places can be put to good use somewhere else.