2.1 Reformat raw benthic cover data

Last updated

July 6, 2026

2.1 Reformat raw benthic cover data

On this page

This page reads the raw benthic cover exports from three monitoring programs (TCRMP, VINPS, CSUN) and reshapes them into one common long format so downstream pages can treat them together. Each program records cover with a different structure, so the page harmonizes column names, standardizes the survey date, labels the program and replicate type, and converts point counts to percent cover where needed. It is the first step in the section-2 benthic pipeline: it writes the reformatted per-program tables and the shared workspace (benthicCoverReformatData.RData) that pages 2.2 through 2.10 load. This page produces no analytical figures, so the percent cover values written here (per taxon, site, and year) flow forward unchanged into every benthic-cover page and into the publication analyses.

Data sources

This page combines three monitoring programs. TCRMP contributes transect-level cover by benthic group (the tcrmp entry). VINPS contributes the VIIS and BUIS CoralVideo Summary by Transect exports, whose transect-level taxon counts are converted to percent cover (the vinps entry). CSUN contributes the site-level random-sites, Yawzi/Tektite, and per-genus cover files (the csun entry). Shared site metadata (sitedat, for shallow-to-deep ordering and site-code matching) loads from the preamble, and every record is filtered to the embargo window (data ≤ 2023-12-31) before saving. The reformatted per-program tables and the shared workspace this page derives are described in the Downloads section below.

Benthic cover data

TCRMP

The TCRMP load reads the transect-level cover file, attaches site metadata, and renames columns to the shared schema (program, date, site, period, replicate, replicatetype, nopts, then the benthic-group cover columns from AA onward). TCRMP carries transect replicates, so replicatetype is set to transect.

Show code
tcrmp_benthicDat <- read.csv("../../../RRSdata/data_TCRMP/TCRMP_coverSpecificBenthicGroups_allSites_2001_2025pbl_data.csv")

# if want to filter sites by period or sampleyear. removed in summer 2024, can re-apply if adding the period column doesnt work.
# tcrmp_benthicDat <- tcrmp_benthicDat %>%
#   filter(Period == "Annual", 
#          SampleYear >= minyear & SampleYear <= maxyear)

# get site data 
tcrmp_benthicDat_siteinfo <- getSiteData(tcrmp_benthicDat, 4)
# tcrmp_benthicDat <- tcrmp_benthicDat[tcrmp_benthicDat_siteinfo$yearadded <= maxyearadded,]

# Reformat data
tcrmp_benthicDat <- tcrmp_benthicDat %>%
  mutate(
    program = "TCRMP",
    date = mdy(FilmDate), # Using actual filming date instead of constructed date
    site = Location,
    replicate = Transect,
    replicatetype = "transect",
    nopts = NoPts,
    period = Period
  ) %>%
  # select(program, date, site, period, replicate, replicatetype, nopts, everything())
  select(
    program,
    date,
    site,
    period,
    replicate,
    replicatetype,
    nopts,
    which(colnames(tcrmp_benthicDat) == "AA"):ncol(tcrmp_benthicDat)
  )

VINPS

The VINPS load reads the VIIS and BUIS transect summaries, matches site codes, sums taxon counts per transect, and converts those counts to percent cover. The block immediately below is the retired 1999–2020 loader, kept commented for reference; the active loader is the VIIS and BUIS pipeline that follows it.

Show code
# vinps_benthicDat <- read.csv(
#   "../../../RRSdata/data_VINPS/VINPS_coverSpecificBenthicGroups_allSites_1999_2020_data.csv"
# )
# 
# vinps_benthicDat <- vinps_benthicDat %>%
#   mutate(SiteFullName = paste(Park, SiteFullName, sep = "-")) %>%
#   filter(Purpose == "Annual", Year >= minyear & Year <= maxyear) %>%
#   select(Year, StartDate, SiteFullName, Transect, TaxonCode)
# 
# # Ensure site names match
# vinps_benthicDat <- vinps_benthicDat[vinps_benthicDat$SiteFullName %in% sitedat$site, ]
# vinps_benthicDat_siteinfo <- getSiteData(vinps_benthicDat, 3)
# 
# # Remove sites added after maxyearadded
# vinps_benthicDat <- vinps_benthicDat[vinps_benthicDat_siteinfo$yearadded <= maxyearadded, ]
# 
# # Calculate percentage cover
# vinps_benthicDat <- vinps_benthicDat %>%
#   group_by(Year, StartDate, SiteFullName, Transect, TaxonCode) %>%
#   summarise(count = n(), .groups = "drop") %>%
#   pivot_wider(names_from = TaxonCode,
#               values_from = count,
#               values_fill = 0)
# 
# npoints <- rowSums(vinps_benthicDat[, 5:ncol(vinps_benthicDat)])
# vinps_benthicDat[, 5:ncol(vinps_benthicDat)] <- vinps_benthicDat[, 5:ncol(vinps_benthicDat)] / npoints * 100
# 
# # Reformat data
# vinps_benthicDat <- vinps_benthicDat %>%
#   mutate(
#     program = "VINPS",
#     date = floor_date(mdy(StartDate), unit = "day"), # Standardize to first of month for consistency
#     site = SiteFullName,
#     replicate = Transect,
#     replicatetype = "transect",
#     nopts = npoints,
#     period = "NA" # added to match with TCRMP periods, but this program did not classify a period 
#   ) %>%
#   select(program,
#          date,
#          site,
#          period,
#          replicate,
#          replicatetype,
#          nopts,
#          everything())

VINPS: read VIIS and BUIS transect summaries

The current VINPS load reads the proofed VIIS (1999–2024) and BUIS (2000–2024) coral-video transect summaries and confirms the two files share the same columns before they are combined.

Show code
vinps_benthicDat_VIIS <- read.csv("../../../RRSdata/data_VINPS/misc/st_croix_nps_data_1/VIIS CoralVideo Summary by Transect 1999 - 2024 proofed.csv")
# vinps_benthicDat_SARI <- read.csv("../../../RRSdata/data_VINPS/misc/st_croix_nps_data_1/SARI CoralVideo Summary by Transect 2012 - 2024.csv")
vinps_benthicDat_BUIS <- read.csv("../../../RRSdata/data_VINPS/misc/st_croix_nps_data_1/BUIS CoralVideo Summary by Transect 2000 - 2024.csv")
NoteValidation note

The VIIS and BUIS transect files share identical column layouts, so they can be combined without re-mapping columns: column names match = TRUE.

What happens here: the proc_vinps() function filters each VINPS file to annual surveys, matches site codes to the shared site table, sums taxon counts per transect, and converts those counts to percent cover. It runs on the VIIS and BUIS files, then the two are harmonized to a common column set and stacked into one vinps_benthicDat table. An internal hist(npoints) call checks the distribution of points per transect during processing; its plot is not shown here because it is a processing check, not a result.

Show code
proc_vinps <- function(dat, sitedat) {
  dat <- dat %>%
    mutate(site_code = paste(ParkCode, Site, sep = "_")) %>%
    filter(Purpose == "Annual") %>%
    select(Year, Date, site_code, Transect, TaxonCode, CountOfTaxonCode)
  
  # Ensure site names match
  dat <- dat %>%
    left_join(sitedat %>% select(site, site_code), by = "site_code") %>%
    rename(SiteFullName = site)
  
  # Remove sites added after maxyearadded
  # vinps_benthicDat <- vinps_benthicDat[vinps_benthicDat_siteinfo$yearadded <= maxyearadded, ]
  
  # Calculate percentage cover
  dat <- dat %>%
    group_by(Year, Date, SiteFullName, Transect, TaxonCode) %>%
    summarise(count = sum(CountOfTaxonCode)) %>%
    ungroup() %>%
    pivot_wider(
      names_from = TaxonCode,
      values_from = count,
      values_fill = 0
    )
  
  npoints <- rowSums(dat[, 5:ncol(dat)])
  hist(npoints)
  
  dat[, 5:ncol(dat)] <- dat[, 5:ncol(dat)] / npoints * 100
  
  # Reformat data
  dat <- dat %>%
    mutate(
      program = "VINPS",
      date = mdy(Date),
      site = SiteFullName,
      replicate = Transect,
      replicatetype = "transect",
      nopts = npoints,
      period = "NA" # added to match with TCRMP periods, but this program did not classify a period
    ) %>%
    select(program,
           date,
           site,
           period,
           replicate,
           replicatetype,
           nopts,
           everything())
  
  return(dat)
}

vinps_benthicDat_VIIS <- proc_vinps(vinps_benthicDat_VIIS, sitedat)
Show code
vinps_benthicDat_BUIS <- proc_vinps(vinps_benthicDat_BUIS, sitedat)
Show code
vinps_benthicDat_BUIS <- vinps_benthicDat_BUIS %>% filter(!is.na(site))

# Get all unique column names across the three datasets
all_columns <- unique(c(
  colnames(vinps_benthicDat_VIIS),
  colnames(vinps_benthicDat_BUIS)
))

# Desired column order
desired_order <- c(
  "program",
  "date",
  "site",
  "period",
  "replicate",
  "replicatetype",
  "nopts",
  setdiff(all_columns, c("program", "date", "site", "period", "replicate", "replicatetype", "nopts"))
)

# Function to harmonize a single dataset (make column names equal)
harmonize_dataset <- function(dataset, all_columns, desired_order) {
  dataset[setdiff(all_columns, colnames(dataset))] <- 0  # Add missing columns as NA
  dataset <- dataset[, desired_order, drop = FALSE]       # Reorder columns
  return(dataset)
}

# Harmonize each dataset
vinps_benthicDat <- 
  rbind(harmonize_dataset(vinps_benthicDat_VIIS, all_columns, desired_order),
        harmonize_dataset(vinps_benthicDat_BUIS, all_columns, desired_order))

CSUN

CSUN records cover at the site level with no transect replicate, so every CSUN table sets replicatetype to site and a single replicate. The section reads three CSUN sources in turn: the random-sites cover file, the long-term Yawzi and Tektite cover file, and the per-genus Yawzi and Tektite files. The record starts in 1992.

The random-sites load reads the file, keeps the taxon columns through Millepora, drops rows flagged nd, coerces cover to numeric, and averages quadrats to one value per site and year.

Show code
csun_random_benthicDat <- read.csv(
  "../../../RRSdata/data_CSUN/CSUN_coverCoralMacroalgaeCTB_RandomSites_1992_2023_data.csv"
)

csun_random_benthicDat <- csun_random_benthicDat %>%
  select(1:which(colnames(csun_random_benthicDat) == "Millepora")) %>%
  # select(-cyanobacteria) %>% 
  filter(!if_any(4:ncol(.), ~ . == "nd")) %>%
  mutate(across(4:ncol(.), as.numeric)) %>%
  rename(year = Year, site = Site, quadrat = Quadrat) %>%
  rename(Montastrea = Montastrea.cav)

# Add month column (assuming June as default)
csun_random_benthicDat$month <- 6
csun_random_benthicDat$month[csun_random_benthicDat$year == "2017.07"] <- 7
csun_random_benthicDat$month[csun_random_benthicDat$year == "2017.11"] <- 11
csun_random_benthicDat$year[csun_random_benthicDat$year %in% c("2017.07", "2017.11")] <- "2017"
csun_random_benthicDat$year <- as.numeric(csun_random_benthicDat$year)

# Filter by year range
csun_random_benthicDat <- csun_random_benthicDat %>%
  filter(year >= minyear & year <= maxyear)

# Aggregate data
csun_random_benthicDat <- csun_random_benthicDat %>%
  group_by(site, year, month) %>%
  summarise(across(2:25, ~ mean(.x)), .groups = "drop")

# Reformat data
csun_random_benthicDat <- csun_random_benthicDat %>%
  mutate(
    program = "CSUN",
    date = ymd(paste(year, month, "01", sep = "-")), # Set to first of month since no day information
    replicate = "1",
    replicatetype = "site",
    nopts = 1,
    period = "NA" # added to match with TCRMP periods, but this program did not classify a period
  ) %>%
  select(program,
         date,
         site,
         period,
         replicate,
         replicatetype,
         nopts,
         everything())

The Yawzi and Tektite load reads the long-term summary cover file, derives year and month from the survey date, drops nd rows, and averages quadrats to one value per site and year.

Show code
csun_yztk_benthicDat <- read.csv(
  "../../../RRSdata/data_CSUN/CSUN_coverCoralMacroalgaeCTB_YawziTektite_1987_2021_data.csv"
)

csun_yztk_benthicDat <- csun_yztk_benthicDat %>% 
  mutate(month = month(Date)) %>%
  mutate(year = year(Date)) %>%
  select(site, quadrat, year, month, percentCover_allCoral:percentCover_CTB)

csun_yztk_benthicDat <- csun_yztk_benthicDat %>%
  filter(!if_any(5:ncol(.), ~ . == "nd")) %>%
  mutate(across(5:ncol(.), as.numeric))


# Aggregate data
csun_yztk_benthicDat <- csun_yztk_benthicDat %>%
  group_by(site, year) %>%
  summarise(across(contains("_"), ~ mean(.x, na.rm=T)), .groups = "drop")

# Reformat data
csun_yztk_benthicDat <- csun_yztk_benthicDat %>%
  mutate(
    program = "CSUN",
    date = ymd(paste(year, "06", "01", sep = "-")), # Set to first of month since no day information,
    replicate = "1",
    replicatetype = "site",
    nopts = 1,
   period = "NA" # added to match with TCRMP periods, but this program did not classify a period
  ) %>%
  select(program,
         date,
         site,
         period,
         replicate,
         replicatetype,
         nopts,
         everything())

CSUN per-genus Yawzi and Tektite data

The per-genus load reads the Yawzi and Tektite genus files, standardizes a misspelled genus name, averages to one value per year, labels each series with its site, and joins the matching all-coral, macroalgae, and CTB summary cover columns.

Show code
csun_yz_benthicDat <- read.csv(
  "../../../RRSdata/data_CSUN/CSUN_coverCoralGenera_Yawzi_1987_2023_data.csv"
)
csun_tk_benthicDat <- read.csv(
  "../../../RRSdata/data_CSUN/CSUN_coverCoralGenera_Tektite_1987_2023_data.csv"
)

csun_yz_benthicDat <- csun_yz_benthicDat %>%
  select(1:which(colnames(csun_yz_benthicDat) == "Millepora")) %>% 
  rename(year = Year) %>% 
  rename(Mycetophyllia = Mycetopohyllia)

csun_tk_benthicDat <- csun_tk_benthicDat %>%
  select(1:which(colnames(csun_tk_benthicDat) == "Millepora")) %>% 
  rename(year = Year)%>% 
  rename(Mycetophyllia = Mycetopohyllia)

# Add month column (assuming June as default)
csun_yz_benthicDat$month <- 6
# csun_yz_benthicDat$month[csun_yz_benthicDat$year == "2017.07"] <- 7
# csun_yz_benthicDat$month[csun_yz_benthicDat$year == "2017.11"] <- 11
csun_yz_benthicDat$year[csun_yz_benthicDat$year %in% c("2017.07", "2017.11")] <- "2017"
csun_yz_benthicDat$year <- as.numeric(csun_yz_benthicDat$year)

csun_tk_benthicDat$month <- 6
# csun_tk_benthicDat$month[csun_tk_benthicDat$year == "2017.07"] <- 7
# csun_tk_benthicDat$month[csun_tk_benthicDat$year == "2017.11"] <- 11
csun_tk_benthicDat$year[csun_tk_benthicDat$year %in% c("2017.07", "2017.11")] <- "2017"
csun_tk_benthicDat$year <- as.numeric(csun_tk_benthicDat$year)

csun_yz_benthicDat <- csun_yz_benthicDat %>% 
  group_by(year) %>%
  summarise(across(is.numeric, ~ mean(.x)), .groups = "drop")

csun_tk_benthicDat <- csun_tk_benthicDat %>%
  group_by(year) %>%
  summarise(across(is.numeric, ~ mean(.x)), .groups = "drop")

csun_yz_benthicDat <- csun_yz_benthicDat %>%
  mutate(
    program = "CSUN",
    date = ymd(paste(year, month, "01", sep = "-")), # Set to first of month since no day information,
    site = "Yawzi",
    period = "NA",
    replicate = "1",
    replicatetype = "site",
    nopts = 1,
    year = year,
    month = month,
  ) %>%
  select(program,
         date,
         site,
         period,
         replicate,
         replicatetype,
         nopts,
         everything())

csun_tk_benthicDat <- csun_tk_benthicDat %>%
  mutate(
    program = "CSUN",
    date = ymd(paste(year, month, "01", sep = "-")), # Set to first of month since no day information,
    site = "Tektite",
    period = "NA",
    replicate = "1",
    replicatetype = "site",
    nopts = 1,
    year = year,
    month = month,
  ) %>%
  select(program,
         date,
         site,
         period,
         replicate,
         replicatetype,
         nopts,
         everything())

csun_yz_benthicDat <- csun_yz_benthicDat %>%
  left_join(
    csun_yztk_benthicDat %>% select(
      site,
      year,
      percentCover_allCoral,
      percentCover_macroalgae,
      percentCover_CTB
    ),
      by = c("site", "year")
  )

csun_tk_benthicDat <- csun_tk_benthicDat %>%
  left_join(
    csun_yztk_benthicDat %>% select(
      site,
      year,
      percentCover_allCoral,
      percentCover_macroalgae,
      percentCover_CTB
    ),
    by = c("site", "year")
  )

The step below joins the Yawzi and Tektite series into the CSUN random-sites table, renames the three summary cover columns to the shared names, and fills genus columns that are absent from the long-term sites with zero.

Show code
csun_random_benthicDat <- csun_random_benthicDat %>%
  rename(
    percentCover_allCoral = Percent_coral_cover,
    percentCover_macroalgae = Percent_macroalgae_cover,
    percentCover_CTB = Percent_CTB_cover
  ) %>%
  bind_rows(csun_yz_benthicDat, csun_tk_benthicDat)


csun_random_benthicDat <- csun_random_benthicDat %>%
  mutate(Mycetophyllia = ifelse(is.na(Mycetophyllia), 0, Mycetophyllia)) %>%
  mutate(Acropora = ifelse(is.na(Acropora), 0, Acropora)) %>%
  mutate(Isophyllastrea = ifelse(is.na(Isophyllastrea), 0, Isophyllastrea)) %>%
  mutate(Helioseris = ifelse(is.na(Helioseris), 0, Helioseris)) %>%
  mutate(Solenastrea = ifelse(is.na(Solenastrea), 0, Solenastrea)) %>% 
  relocate(Millepora, .after = Solenastrea)

Downloads

This page writes four reformatted per-program CSV tables to a timestamped benthicCoverReformatData_<datetime> subfolder and saves the full workspace to benthicCoverReformatData.RData. The RData workspace is the object that every later section-2 page loads, so it is an intermediate rather than a standalone reader download. The section-2 pages that build the reader-facing cover products (2.3 major categories, 2.4 genera, 2.5 species, 2.6 trait groups, and 2.7 through 2.10) each offer their own data-and-metadata downloads.


version 1.0.0 • in-review • data ≤ 2023-12-31