Exploring Dengue Data with BigDataPE

Introduction

The BigDataPE package provides a simple and secure interface for accessing datasets published by the Big Data PE platform from the Government of the State of Pernambuco, Brazil.

In this vignette we walk through the full package workflow using the Dengue, Zika and Chikungunya dataset — a record of notified cases in health units (public and private) in Recife, shared by the State Health Department of Pernambuco (Secretaria Estadual de Saude de Pernambuco).

Field Value
Classification Broad (public)
Source institution State Health Department of Pernambuco
Ingestion type SQL
Records 1 009
LGPD level None
Last updated 2024-05-24

Access note: Although this dataset is public, you must submit an access request on the Big Data PE platform and be connected to the PE Conectado network or a VPN to query the data through the API. Without this connection, requests will time out or return a service-unavailable error.

Setup

Load the package and the helper libraries used in the analysis:

library(BigDataPE)
library(dplyr)
library(tibble)

1. Storing the access token

The first step is to store the authentication token associated with the dataset. This token is provided by the Big Data PE platform after your access request is approved.

bdpe_store_token("dengue", "your-token-here")
#> ✔ Token stored in environment variable: `BigDataPE_dengue`

You can verify that the token was stored correctly:

bdpe_list_tokens()
#> [1] "dengue"

2. Fetching data

Basic query

bdpe_fetch_data() is the most direct way to query the API. Let’s fetch the first 100 records:

data <- bdpe_fetch_data("dengue", limit = 100, offset = 0)
glimpse(data)
#> Rows: 100
#> Columns: 126
#> $ nu_notificacao           <chr> "3517726", "3613049", "3507055", ...
#> $ tp_notificacao           <chr> "2", "2", "2", ...
#> $ co_cid                   <chr> "A90", "A90", "A90", ...
#> $ dt_notificacao           <chr> "2020-07-07", "2020-06-27", "2020-01-03", ...
#> $ ds_semana_notificacao    <chr> "202028", "202026", "202001", ...
#> $ notificacao_ano          <chr> "2020", "2020", "2020", ...
#> $ co_municipio_notificacao <chr> "261160", "260790", "261160", ...
#> $ tp_sexo                  <chr> "M", "M", "M", ...
#> $ febre                    <chr> "1", "1", "1", ...
#> $ mialgia                  <chr> "1", "2", "1", ...
#> $ cefaleia                 <chr> "2", "1", "1", ...
#> $ ...

The dataset contains 126 columns with detailed information on each notification, including demographics, symptoms, laboratory results, warning signs, severity indicators and case outcome.

Using query parameters

The API supports additional filters through the query parameter. For example, to fetch only notifications from the year 2020:

dengue_2020 <- bdpe_fetch_data(
  "dengue",
  limit = 50,
  offset = 0,
  query = list(notificacao_ano = "2020")
)
nrow(dengue_2020)
#> [1] 50

You can also filter by municipality of residence. The IBGE code for Recife is 261160:

dengue_recife <- bdpe_fetch_data(
  "dengue",
  limit = 100,
  offset = 0,
  query = list(co_municipio_residencia = "261160")
)
nrow(dengue_recife)
#> [1] 100

Filters can be combined. To fetch female cases in Recife:

dengue_female_recife <- bdpe_fetch_data(
  "dengue",
  limit = 100,
  offset = 0,
  query = list(
    co_municipio_residencia = "261160",
    tp_sexo = "F"
  )
)
nrow(dengue_female_recife)

3. Fetching data in chunks

When the data volume is large, use bdpe_fetch_chunks() which paginates automatically. To fetch all 1,009 records in blocks of 500:

all_data <- bdpe_fetch_chunks(
  "dengue",
  total_limit = Inf,
  chunk_size = 500,
  verbosity = 1
)
#> ℹ Fetched 500 records (total: 500).
#> ℹ Fetched 500 records (total: 1000).
#> ℹ Fetched 9 records (total: 1009).
#> ✔ Fetching complete: 1009 records retrieved.

dim(all_data)
#> [1] 1009  126

4. Exploring the data

With the data in hand we can run quick exploratory summaries.

Distribution by sex

all_data |>
  count(tp_sexo, sort = TRUE)
#> # A tibble: 3 x 2
#>   tp_sexo     n
#>   <chr>   <int>
#> 1 F         561
#> 2 M         443
#> 3 I           5

Notifications by year

all_data |>
  count(notificacao_ano, sort = TRUE)
#> # A tibble: 1 x 2
#>   notificacao_ano     n
#>   <chr>           <int>
#> 1 2020             1009

Most frequent symptoms

Symptom fields use "1" for yes and "2" for no. We can compute the proportion of each symptom:

symptoms <- c("febre", "mialgia", "cefaleia", "exantema", "vomito",
              "nausea", "dor_costas", "conjutivite", "artrite",
              "artralgia", "dor_retro")

all_data |>
  summarise(across(all_of(symptoms), ~ mean(.x == "1", na.rm = TRUE))) |>
  tidyr::pivot_longer(everything(),
                      names_to  = "symptom",
                      values_to = "proportion") |>
  arrange(desc(proportion))
#> # A tibble: 11 x 2
#>    symptom      proportion
#>    <chr>             <dbl>
#>  1 febre             0.914
#>  2 cefaleia          0.531
#>  3 mialgia           0.512
#>  4 artralgia         0.194
#>  5 exantema          0.181
#>  6 dor_costas        0.155
#>  7 vomito            0.150
#>  8 nausea            0.139
#>  9 dor_retro         0.125
#> 10 conjutivite       0.053
#> 11 artrite           0.046

Distribution by neighbourhood (top 10)

all_data |>
  filter(no_bairro_residencia != "") |>
  count(no_bairro_residencia, sort = TRUE) |>
  head(10)
#> # A tibble: 10 x 2
#>    no_bairro_residencia     n
#>    <chr>                <int>
#>  1 IBURA                  46
#>  2 VARZEA                 37
#>  3 COHAB                  33
#>  4 BOA VIAGEM             30
#>  5 AGUA FRIA              27
#>  ...

Hospitalisations

all_data |>
  count(st_ocorreu_hospitalizacao) |>
  mutate(description = case_match(
    st_ocorreu_hospitalizacao,
    "1" ~ "Yes",
    "2" ~ "No",
    "9" ~ "Unknown",
    ""  ~ "Not reported"
  ))
#> # A tibble: 4 x 3
#>   st_ocorreu_hospitalizacao     n description
#>   <chr>                     <int> <chr>
#> 1                             330 Not reported
#> 2 1                            51 Yes
#> 3 2                           565 No
#> 4 9                            63 Unknown

5. Managing tokens

Retrieve a stored token

my_token <- bdpe_get_token("dengue")

Remove a token

At the end of your session, or when the token is no longer needed:

bdpe_remove_token("dengue")
#> ✔ Token successfully removed for dataset: "dengue"

Data dictionary (key variables)

Variable Description
nu_notificacao Notification number
co_cid ICD-10 code (A90 = Dengue)
dt_notificacao Notification date
notificacao_ano Notification year
co_municipio_notificacao IBGE code of the notifying municipality
dt_diagnostico_sintoma Date of first symptoms
tp_sexo Sex (M, F, I = Indeterminate)
tp_raca_cor Race/colour (1–5, 9 = Unknown)
no_bairro_residencia Neighbourhood of residence
febre … dor_retro Symptoms (1 = Yes, 2 = No)
diabetes … auto_imune Pre-existing conditions (1 = Yes, 2 = No)
st_ocorreu_hospitalizacao Hospitalisation occurred (1/2/9)
tp_classificacao_final Final case classification
tp_evolucao_caso Outcome (1 = Recovery, 2 = Death, …)

References