Why deep learning will probably never improve malaria mapping…
… or SDMs or prognostic models or …
Every couple of months I start thinking about deep learning again. A few days and a few headaches later I conclude that it is not useful for most of the work I do. So I thought I’d write down my thought process this time to try and save my poor aching brain from another period of thought.
I’ve tried to be careful with the title wording.
Why - I’ll give my reasons, it’s not just a hunch.
deep learning - Deep neural nets but also other methods. Anything where the learning involves a transformation of a transformation of a transformation of a … of the data. But importantly not including other machine learning methods which I use all the time because they often improve predictive accuracy in my work.
probably - an uncertain forecast
never - even if data size and compute increases 100x and even after 50 postdoc years of effort
improve - predictive accuracy. Other elements of statistical modelling are not the topic of this post.
malaria mapping or SDMs or prognostic models or… - I fully acknowledge the sucesses of deep learning. But I don’t think they’ll help in my work.
So the first thing to note is that there’s two ways that depth in neural networks and other methods is commonly used. The first is the case of having multiple dense hidden layers (i.e. all nodes in each layer is connected to all nodes in the next layer). As far as I understand this type of architecture is not that important to the success of deep learning. I don’t quite understand the benefits of this architecture compared to a shallow but very wide neural network (one hidden layer with a lot of nodes) but they are commonly used so they must be useful. However, the important thing here is that ultimately, the only thing that this architecture provides is increased flexibility, or nonlinearity, in the model. However, something like a RandomForest or boosted regression trees can have unlimited nonlinearity. So this architecture isn’t providing anything particularly unusual. Furthermore I’ve had a pretty good go at using deep, dense neural networks to map malaria with very little success. Tree based methods are very efficient with the data provided. Due to their greedy estimation, they put all their focus on areas of parameter space that determine the output. Neural networks are much less good at this. So overall, I’m fairly confident that dense multilayer neural networks will never give much better predictions than tree based methods. As we get more data, these architectures may do about as well as tree based methods, but not significantly better.
The architectures that have really driven the success of deep learning as we know it are convolutional neural networks. These are the neural networks used for image and video analysis including image classification, image segmentation, self driving cars etc. These methods generally exploit the spatial (and/or temporal) structure of the images or videos used to train them. And specifically they make good predictions by learning good ways to represent the data. At the top of network, there will be nodes that learn that a line of low values next to a line of high values is an “edge”. In the middle layers, there will be nodes that learn that two horizontal edges and two vertical edges makes a rectangle. And at the bottom of the network will be nodes that learn that a rectangle and some circles is maybe a lorry. This type of data, and these types of representations of the data just don’t exist in most of the subjects I have worked on. In malaria mapping, having an “edge” between cold and hot areas tells you very little about the risk of malaria*. In prognostic modelling, there is often no covariates that have any sort of image structure at all. You might have covariates like age and pre-existing conditions, and these covariates might have interactions. But this idea of edges and rectangles just isn’t relevant.
Perhaps another way to think about this is that deep learning methods have mostly excelled in situations where humans can (more or less) easily perform the task but representing the problem in a way that the computer can usefully use is difficult. A three year old can identify a cat, but despite years of computer vision experts hand creating features such as “circles” and “triangles”, computers still couldn’t identify a cat in an image. We have to let the model learn how to represent the data.
Most of the problems I work on have the opposite situation. We can easily represent the data in a totally acceptable way; one column for age, one column for each pre-existing conditions, done. But even experts in the particular field often couldn’t effectively use these data to make good, quantitative predictions. How much malaria will there be in an area with a mean temperature of 28 degrees and 120 days of rain? Quite a lot, I guess, but that’s about the best I can do. So in these problems, the task for the machine learning model is much more about seperating signal from noise, and also about finding nonlinear relationships and relatively simple interactions, and using these to make accurate predictions. As far as I can see deep learning doesn’t provide anything for this task that tree-based methods can’t already do, and they’re just less efficient with the data.
This felt like it would be a much longer post as I was, once again, grappling with what a deep neural network is really doing (this time I was trying to think of good ways to fit convolutional RandomForests). But oh well. I’ll chuck it up on my webpage and see if it generates some disscussion.
One thing I’ll note is that I really know very little about the architectures used in language models like GPT-3. I’ve read a fair bit about long short term memory networks and they definitely aren’t relevant to most of the areas I have worked on. Maybe there’s something here that will be useful though. Similarly I don’t know about the deep neural networks used in reinforcement learning for robotics. Maybe I’ll come back and edit this is 6 months when I’ve done more reading.
*The fact that some of the areas are hot (high malaria) and some of the areas are cooler (low malaria) may well tell you that in aggregate there will be intermediate malaria risk in the area. But this simpler fact can be handled much more directly with disaggregation regression, which is precisely what I have been working on for five years.
A practical guide to squishing yearly ‘disag data’ objects
A practical guide to pooling annual data for Bayesian disaggregation regression in R
This post is a guest post by my predoctoral fellow, Samana.
Bayesian spatial disaggregation gives you a high resolution risk surface from low-resolution case counts ….. but only one year at a time. This post shows how to pool several years into a single model fit.
Hi all, my name is Samana and I’m a predoctoral fellow and I’ve been working on the problem of a multi-year disaggregation model!
In this blog I will be presenting a recipe of how to do a multi-year disaggregation model fit.
Currently, using the disaggregation package we can only really fit one model per time slice of data we might have, but this blog aims to change that and let you fit a disag model with multiple years of data.
Part A demonstrates the mechanics without a time trend; Part B adds a time spline to the six-year model. This blog is part A.
This post shows how to combine annual disag_data objects and fit one pooled spatial disaggregation model. It demonstrates shared effects across years and it is not a full spatio-temporal model with separate spatial field for each year.
The resolution mismatch problem:
Simply put, there is a resolution mismatch between satellite covariates (which are very high resolution) and public health data, like case counts of a particular disease in a specific county (this is low resolution).
Public health data is often reported as annual case counts for administrative areas such as counties and districts whereas environmental information that may explain risk, such as temperature, land cover, vegetation is often available on a fine spatial grid.
The problem is that we can’t just combine these together and then fit a model for risk predictions to get a high resolution risk surface. This is because of the ecological fallacy.
These figures above show the resolution mismatch; on the left we have the low resolution public health case counts and on the right we have the high resolution environmental satellite covariates (Nandi et al. 2023, https://doi.org/10.18637/jss.v106.i11).
What is disaggregation and why you’d want to do it:
So these resolutions are linked by modelling a latent disease risk at the pixel level and THEN aggregating the pixel level predictions to the polygon level so they’re consistent with the observed disease counts. all of this is done by Bayesian spatial disaggregation (implemented in the R package disaggregation )
The package fits a Bayesian disaggregation regression model (similar to a GLM) with an INLA style spatial random field via TMB, using the function prepare_data() to line up polygons, covariates and population, and disag_model() to fit the model.
The current single year workflow is like this:
Read in the inputs: a sf object containing the area boundaries and case counts, a SpatRaster of fine-scale covariates, and a population raster when modelling counts with population as exposure.
Prepare these inputs into a coherent data object with prepare_data().
Fit the model with disag_model(), choosing the likelihood and link and whether to include a spatial field and an independent (iid) effect.
Inspect the fit with summary() and plot(), then use predict() to produce fine-scale prediction and uncertainty maps.
For multiple years, the missing step is between 2 and 3: prepare each year separately, then combine the resulting disag_data objects carefully before fitting one pooled model.
Why multi-year disaggregation?
When you have many years of polygon level counts, a natural question is: fit each year separately, or pool several years into one model?
Pooling helps when you want the spatial field and covariate effects to borrow strength across years, while still letting a time trend move through a flexible term (we’ve used a natural spline on centered year in part B of this blog).
I’ve been calling this “squishing” years together, stacking each year’s prepared data into one long object before handing it to disag_model().
Bayesian spatial disaggregation is straight forward for one year of data but multi-year models are harder because the disag_data object contains many components that need to be in the right shape, format and size.
This object contains polygon-level outcomes, pixel-level covariates, aggregation weights, spatial coordinates, start end indices and the spatial mesh. These objects must remain correctly aligned when the years are combined.
This tutorial develops a reproducible workflow for combining yearly disag_data objects informally, “squishing” or “combining” them before fitting a multi-year Bayesian disaggregation model.
Model components
The disaggregation regression models demonstrated here include fixed effects of covariates and a number of random effects, for various different modelling purposes.
In this example, the pooled model estimates shared climate effects. We have different covariate observations for each year, but want to fit a model using all years of data in one go.
The spatial random effect operates at high resolution. If you model each year separately, then each time period has its own spatial field, defined over the mesh and projected to the pixels. It captures smooth residual spatial variation that remains after accounting for measured environmental covariates. However, we often want to pool estimates for this random effect across years.
The iid random effect operates at polygon level, the lower resolution. It gives each polygon-time observation an additional independent residual term, helping account for overdispersion and unmeasured area-level factors.
When yearly data are squished, the model is fitted jointly across all periods. The workflow must preserve the relationships between each polygon, its pixels, its aggregation weights and its appropriate spatial field.
The iid term allows an additional residual effect for each polygon-year observation. Stacking the data does not automatically create a separate spatial field for each year or model how fields evolve over time.
THE DATA:
The example region is Madagascar, deliberately chosen because it’s the same country used in the disaggregation package’s own founding methods paper (Nandi et al. 2020, malaria case counts across Madagascar, [Nandi et al](https://www.jstatsoft.org/article/view/v106i11)) so the mesh settings and covariate naming are the same as that original worked example.
The geometry, climate covariates AND population is real public data; only the case counts are simulated (real disease data can’t be shared here for data-permission reasons)
The dataset pairs real, public district geometry, real climate covariates and real population with simulated case counts:
Real public administrative level-2 (district) boundaries for Madagascar, from the GADM database (https://gadm.org), downloaded via the R geodata package.names.
Year varying covariates (temp_wc, precip_wc): annual mean temperature and annual total precipitation for each simulated year, built from REAL monthly TerraClimate records (Abatzoglou et al. 2018) not long-run climate normals, so these differ from one year to the next. Centred and scaled (mean 0, sd 1, using one mean/sd computed across all years combined so real year-to-year differences are preserved). Source: https://www.climatologylab.org/terraclimate.html.
Simulated case counts drawn from known truth parameters (shipped alongside the data as true_parameters.csv), using the real covariates and population above plus a simulated spatial random field and district-level noise. So this post can check whether the fitted model recovers the true coefficients, a much better teaching device than fitting to arbitrary noise.
None of the numbers in the dataset correspond to any real disease case count.
:::
### Step 4: prepare data for each year
Mesh args here match those used in the `disaggregation` package's own
Madagascar malaria worked example (Nandi et al. 2020) a nice side
benefit of picking the same country is that these don't need retuning
from scratch.
Always retune these to your own study area if you swap in a different
region.
The details for what these parameters mean can be found in the inla
tutorials for example.
::::::: cell
``` {.r .cell-code}
mesh_args_mdg <- list(max.edge = c(0.7, 8), cutoff = 0.05, offset = c(1, 2))
dis_2001 <- prepare_data(
polygon_shapefile = shapes_2001,
covariate_rasters = cov_stack_2001,
aggregation_raster = population_raster,
mesh_args = mesh_args_mdg,
id_var = "district_id",
response_var = "cases",
na_action = TRUE
)
::: {.cell-output .cell-output-stderr} Warning: [rast] CRS do not match :::
::: {.cell-output .cell-output-stderr} Warning: [extract] transforming vector data to the CRS of the raster :::
::: {.cell-output .cell-output-stderr}
Warning: [rast] CRS do not match
Warning: [extract] transforming vector data to the CRS of the raster
:::
``` {.r .cell-code}
dis_2003 <- prepare_data(
polygon_shapefile = shapes_2003,
covariate_rasters = cov_stack_2003,
aggregation_raster = population_raster,
mesh_args = mesh_args_mdg,
id_var = "district_id",
response_var = "cases",
na_action = TRUE
)
::: {.cell-output .cell-output-stderr} Warning: [rast] CRS do not match Warning: [extract] transforming vector data to the CRS of the raster ::: :::::::
Step 5: The squish: stack the 3 years of data into one disag data object
To stack these objects together correctly, we handle different parts of the data differently. polygon_data and covariate_data are just stacked with a year label attached. We offset start_end_index: each prepare_data() call starts its pixel-row numbering again, so the second and third years’ indices must be shifted to point to their new positions in the combined covariate table.
::: cell ``` {.r .cell-code}
polygon_data and covariate_data just stack with a year label attached
poly_0103 <- bind_rows( transform(dis_2001$polygon_data, year = 2001), transform(dis_2002$polygon_data, year = 2002), transform(dis_2003$polygon_data, year = 2003) )
cov_0103 <- bind_rows( transform(dis_2001$covariate_data, year = 2001), transform(dis_2002$covariate_data, year = 2002), transform(dis_2003$covariate_data, year = 2003) )
:::
### step 6: fit the pooled model
:::::: cell
``` {.r .cell-code}
fit_0103 <- disaggregation::disag_model(
data = dis_0103,
iterations = 1000,
field = TRUE,
iid = TRUE,
family = "poisson",
link = "log"
)
::: {.cell-output .cell-output-stderr} Fitting model. This may be slow. :::
Compare that against three separate single-year fits (disag_model() on 2001, 2002, 2003 individually, no squishing) to see whether pooling changes the covariate estimates, tightens their standard errors, or shifts the spatial field. This comparison is the actual point of the exercise the code above just gets you to a model you can compare against.
:::::: cell ``` {.r .cell-code} fit_2001 <- disaggregation::disag_model(dis_2001, iterations = 1000, field = TRUE, iid = TRUE, family = “poisson”, link = “log”)
::: {.cell-output .cell-output-stderr}
Fitting model. This may be slow.
:::
``` {.r .cell-code}
fit_2002 <- disaggregation::disag_model(dis_2002, iterations = 1000, field = TRUE,
iid = TRUE, family = "poisson", link = "log")
::: {.cell-output .cell-output-stderr} Fitting model. This may be slow. :::
``` {.r .cell-code} fit_2003 <- disaggregation::disag_model(dis_2003, iterations = 1000, field = TRUE, iid = TRUE, family = “poisson”, link = “log”)
::: {.cell-output .cell-output-stderr}
Fitting model. This may be slow.
:::
::::::
::::::: cell
``` {.r .cell-code}
summary(fit_2001)
Pooling the three years reduced uncertainty: the pooled model had smaller standard errors for both temperature and precipitation than any separate annual model. The temperature estimate was positive and the precipitation estimate negative, consistent with the values used to simulate the data.
Refrences
Nandi, A., Lucas, T., Arambepola, R., Python, A. (2023). disaggregation: An R Package for Bayesian Spatial Disaggregation Modeling. Journal of Statistical Software, 106(11). https://doi.org/10.18637/jss.v106.i11
Nandi, A., Lucas, T., Arambepola, R., Gething, P., Weiss, D. (2020). disaggregation: An R Package for Bayesian Spatial Disaggregation Modelling (original methods paper, Madagascar malaria worked example). https://arxiv.org/abs/2001.04847
GADM database of global administrative boundaries: https://gadm.org
Abatzoglou, J.T., Dobrowski, S.Z., Parks, S.A., Hegewisch, K.C. (2018). TerraClimate, a high-resolution global dataset of monthly climate and climatic water balance from 1958-2015. Scientific Data, 5, 170191. https://doi.org/10.1038/sdata.2017.191
These figures above show the resolution mismatch; on the left we have the low resolution public health case counts and on the right we have the high resolution environmental satellite covariates (Nandi et al. 2023, https://doi.org/10.18637/jss.v106.i11).
