The COVID-19 pandemic is the largest public health crisis in more than a century and it spread across the world in a matter of weeks. Similarly, the amount of data and science around COVID-19 has risen exponentially, leading to everyday discussions among experts and laypeople about cases, deaths and where we are headed. This week we highlight 11 misconceptions about COVID-19 data and its interpretation to better inform our use of data for decision-making.
Myth 1: Case trends are enough to monitor COVID-19 spread. Trends in the cases counts, even those adjusted for population numbers, are not enough to fully understand the disease situation. The absolute burden of disease is also important: a 10% decline in 10,000 cases is much different than a 10% decline in 100 cases. The level of testing is also an important consideration, as case trends may artificially increase or decrease if the level of testing is changing significantly. Lastly, the number of susceptible people over time should be considered when using case rates. If cases per capita decline by 10% in a place where half of the people moved away in the same time period, the actual spread of disease is probably rising, not declining. To fully understand the spread of disease, additional information from other metrics should be considered.
Myth 2: Case incidence is always a good indicator of community risk. The number of new cases (incidence) in a population does not always reflect the risk of transmission in a community. The main reason is that the composition or distribution of these cases may be very different, even while the overall total is the same. There also can be significant clustering from “superspreader” events that drive local transmission. For example, consider the following two hypothetical communities, A and B. Both have four new cases in a population of 20 people, so the case incidence rate is the same (one in five people). Community A however has a more demographically and geographically diverse distribution of cases than Community B, where three of four cases are located in the same long-term care facility. The risk of coming into contact with an infectious person in Community B is much lower than in Community A.
To accurately capture community risk, we must consider additional information on the cases and patterns of transmission, where they are located, and the current status of control measures such as isolation of cases and quarantine of contacts. This information varies by location and should be used to fine-tune public health and social measures at local levels.
Myth 3: COVID-19 deaths are an indicator of the current situation. The time lag in reporting deaths makes it less useful for understanding what is happening today. In the absence of consistent information on cases, hospitalizations or other metrics, deaths can be a more reliable indicator of the burden of illness in a population, especially when the risk of death is not expected to change substantially over time. Deaths can also be used to obtain a rough estimate of the number of cases in a population. A simple calculator can also account for other factors such as age distribution of a population and the time interval between cases and deaths. This interval, or lag, makes deaths a reflection of the situation several weeks prior to when they are reported. According to current CDC modeling parameter best estimates, it takes on average about six days to develop symptoms after an exposure, 15 days from symptom onset to death, and seven days from death to reporting, for a total of about 28 days from symptom onset to death. It is important to remember that these are rough estimates and in practice this timeline can vary a great deal from one individual to the next.
Given this lag, deaths are not a useful early indicator for monitoring whether the disease situation is worsening or improving. Practically, this means that to assess the impact of changes (e.g. loosening or tightening) of public health and social measures, one would have to wait several weeks before any change is detected.
Myth 4: COVID-19 deaths are the only indicator of COVID-19-related mortality. COVID-19 deaths are underestimated for several reasons: limited testing leads to under-detection of those infected, and a large proportion of deaths occurring in the community—as opposed to in hospitals—may not be attributed to COVID-19. Excess mortality, or the gap between current deaths and the historical average, is a better measure of the overall impact of COVID-19. This includes those who directly or indirectly die due to the pandemic. This gap can be very large in some locations, such as Italy:
In the above figure we can see that only about half of the excess deaths (gray curve above the dotted line) from March to April were attributed to COVID-19. Hence official COVID-19 death counts would not fully reflect the mortality associated with COVID-19. The other deaths likely include a mix of people who died from COVID-19 (but were not tested for or presumptively diagnosed with it) outside of health facilities, and additional deaths from other conditions such as heart attacks due to disruptions in routine and emergency care.
Myth 5: Mobility indicators are a direct indicator of risk. Mobility data, typically generated using location data from mobile devices (e.g. data from Apple, Facebook, or Google), have been referred to extensively in this pandemic as a leading indicator of risk. This data can provide timely and granular information at the local level to assess broad adherence with physical distancing measures. It is important to recognize that this is a proxy for risk, and there are significant limitations and considerations that should be kept in mind when using this data. Mobility data can tell you about trends in frequency and distance of travel but does not tell you much about high-risk behaviors, such as being in close contact with someone for an extended period of time indoors. It also cannot tell you about personal behaviors, such as good hand hygiene and mask wearing, which can reduce transmission even if there is increased mobility. It is possible that these behaviors, if widely adopted, can blunt the disease transmission impact of increased mobility. Mobility data also may not fully represent the population in areas with low mobile phone penetration across demographic groups. To accurately measure changes in risk and the impact of public health and social measures, mobility data should be interpreted in concert with information on personal behaviors and additional epidemiological information.
Myth 6: “R” is all you need to know about disease transmission. As previously covered in a Resolve to Save Lives Science Review, the effective reproduction number (Rt) is simple to understand but difficult to estimate with accuracy. This figure represents the expected number of secondary infections arising from a single infected person. If the value of Rt is greater than 1, then disease spread is increasing. If Rt is less than 1, then disease spread is decreasing. The idea that one number can be updated daily to reflect current transmission is extremely appealing. It has been used by officials to communicate with the public and by the media, for example, from San Francisco Chronicle, “A simple score shows how fast the coronavirus is spreading. Here’s what it is for the Bay Area.” However, as presented in the science review, there is no one standard way to estimate R, estimates typically have a large amount of uncertainty and wide confidence intervals, and, because of delayed case reporting, estimates typically have a lag of one to two weeks before they are stable. These factors make them less useful for decision-making. For example, consider the Rt estimates for the states in the U.S.:
In the above figure, each U.S. state is represented by a circle and the bands above and below indicate the confidence intervals around the estimate, with the lighter band representing the 90% confidence interval of the estimate. As depicted, if we were to use the point estimate of R to inform decision-making, the R in many states (38 to be exact) would be below 1, indicating declining transmission, and several (10 states) would be above 1, indicating growing transmission. But if we consider the uncertainty of the Rt estimate, only three states have a value that is statistically different from 1; New York, New Jersey and Illinois are the only states with an R below 1. There are also no standard cutoffs for when to make decisions. For example, is it OK to reopen if Rt is 0.95, 0.9 or 0.8? How long does it have to be that way, one day or one week? If decisions to tighten or loosen public health and social measures are based on Rt point estimates alone, different decisions might be made depending on the model used and whether uncertainty was considered. If Rt is used for decision-making, it should be considered along with other epidemiological data to fully understand disease spread.
Myth 7: Symptom-based screening is adequate to protect every population. Symptom-based screening for COVID-19 is common, and is used in places such as airports, offices and hospitals. People who have a higher probability of infection can be identified by inquiring about fever, cough and other symptoms of COVID-19. Symptom screening can inform who should be tested and when to isolate individuals. Although easy to do, this method of screening is not sufficient to protect every population from COVID-19. Based on early evidence, an estimated 20-50% of those infected with COVID-19 never develop symptoms. In addition, a proportion of infected people will not be symptomatic at screening but go on to develop symptoms later and are infectious one to two days before developing symptoms, so the lack of symptoms does not imply a lack of infectiousness. According to U.S. CDC, asymptomatic cases are 50% to 100% as infectious as symptomatic cases. Clearly, symptom-based screening will miss a proportion of infections. In settings with vulnerable populations (such as nursing homes) where many people are at higher risk for illness and death, it is important that symptom-based screening be supplemented by laboratory testing to rapidly identify most infected people in order to facilitate early isolation and stop ongoing transmission.
Myth 8: Hospital and ICU bed capacity are the most useful metrics for capturing health care system readiness. The ability to safely manage COVID-19 cases, including the critically ill, is an important means to prevent deaths. However, the proportion of current hospital and intensive-care unit (ICU) beds that are available is not a robust indicator of this capability. Hospital occupancy changes with seasonality, rising during influenza season and falling in other months. At or near the peak of seasonal influenza, hospital beds and ICU beds are likely to be at or near capacity, which is appropriate and expected. More informative data would be the proportion of hospital beds or ICU beds occupied by patients with COVID-19, and the ability of a health care system to expand the number of beds, procure and maintain sufficient equipment (e.g. oxygen and ventilators), and most importantly, trained staff, to accommodate a surge in COVID-19 patients. Another essential health care system indicator is the number of infections among health care workers. Tens of thousands of health care workers have been infected in the current pandemic, and many were not properly equipped to protect themselves, their patients or their families. Currently, many locations do not have data on the number of health care worker infections, which means they cannot accurately assess their ability to safely treat patients.
Myth 9: COVID PCR test positivity is all you need to know about the state of testing: PCR tests are used to detect active infections; test positivity is the number of positive tests out of the total number of tests performed. This metric is useful in understanding whether or not a location is testing enough people to detect cases, regardless of the size of the outbreak. For instance, a test positivity rate of 5% indicates that there is one confirmed case out of every 20 people tested. Generally, the lower the test positivity rate, the more robust the testing program. However, this number alone is not sufficient to understand testing because it does not indicate anything about whether high-risk people are getting tested. Consider this hypothetical example of two communities:
Both communities have 100 people and three people with COVID-19 infections. Community A is letting anyone who needs a test get a test, and many of these tests have been done in lower-risk groups (people who might have better access to testing). Community B has a testing strategy which includes prioritized testing for higher-risk groups (e.g. nursing home residents, health care workers, symptomatic contacts of known cases). Since Community B is testing people more likely to have disease, they have detected more of the cases (two of three) in the community as compared to Community A (one of three). Both Communities A and B have a test positivity rate of 10%. Therefore, if this was the only metric monitored, important insights would be missed. Ideally, communities have metrics on testing of priority groups, but this can be hard to measure. In the absence of this one might monitor the number of tests performed as a rough measure of whether sufficient numbers of people (including high-risk groups) are being tested. Another important consideration when interpreting testing is whether the number of tests refers to the number of people tested, or the number of tests performed. In some instances, one individual can receive multiple tests, so the number of tests performed is larger than the actual number of people tested. For instance, many protocols for hospital discharge require that a COVID-19 patient have two negative tests at least 24 hours apart. This means that most hospitalized COVID-19 patients who recover have been tested at least three times (once on admission, twice for discharge). Indicators such as the time interval between symptom onset and positive test result, and time interval between symptom onset and isolation, are more informative and more relevant to determining if testing is being well targeted, and more useful to improving strategies for testing and follow-up of test results.
Myth 10: The most significant health impacts of the pandemic are directly related to COVID-19. COVID-19 can be a severe disease, particularly in older people and in those with underlying conditions. As in prior disease outbreaks, in many communities, the largest health impacts of COVID-19 are likely not from those directly affected with the disease, but in the secondary disruptions of essential health services and public health programs. In the West African Ebola epidemic, more than 11,000 people died directly from Ebola, but an estimated 11,000-26,000 additional deaths occurred from HIV/AIDS, tuberculosis, malaria and measles alone, due to interruptions in treatment and vaccination. In the COVID-19 pandemic, disruptions have undermined communicable disease control programs, immunization activities, reproductive and maternal and child health activities, and noncommunicable disease management. WHO estimates that COVID-19 disruptions could double malaria deaths in Sub-Saharan Africa and disrupt the vaccination of 80 million children under the age of 1. In order to mitigate the overall health impacts of COVID-19, rapid resumption of these activities in a safer way is essential, such as with catch-up immunization campaigns. To fully understand the total health impact of the pandemic, it is important to monitor non-COVID metrics to understand both the direct and indirect effects on population health.
Myth 11: Everyone is an epidemiologist. The COVID-19 pandemic has resulted in more people engaging with and using epidemiological data than ever, to try to understand how the pandemic is progressing and gain insight into their personal risk. Terms that were previously known only to scientists, such as R and test positivity, are suddenly part of everyday conversations in many circles. Media articles and guidance suggesting metrics and targets are released regularly. Data on COVID-19 can be found to support nearly any viewpoint or message, whether scientifically valid or not. In the midst of this fog of information, it is important to understand that epidemiology, which is the study of the distribution and determinants of disease, is a field of science that goes beyond cases and deaths. Just as a health care provider is an expert in medicine and a virologist is an expert in viruses, an epidemiologist is an expert in how to collect, analyze and disseminate data and take other steps to prevent and control diseases. Epidemiologists have insights that can inform sound actions and accelerate progress in pandemic response. They also adapt their recommendations based on evolving information, which is often an indication of good science and not that prior recommendations were wrong. The public needs to hear from these disease experts directly and often in order to understand the state of the pandemic.