Reformatting conductivity timeseries data table in Reaeration (DP1.20190.001) and Salt-based discharge (DP1.20193.001)
October 31, 2023
The conductivity data in Reaeration and Salt-based discharge are being moved into individual files per sampling event. There will be one file for each hoboSampleID in the rea_backgroundFieldCondData table.
Performance problems are developing in the observational data processing pipeline due to the inclusion of high-volume tables containing high-frequency sensor data – in particular, the rea_conductivityFieldData and sbd_conductivityFieldData tables in data products Reaeration (DP1.20190.001) and Salt-based discharge (DP1.20193.001), respectively. To resolve the problem, these tables are being moved to the large file storage system for observational data, already in use for microbial community composition and leaf-level spectral data.
Downloaded data will continue to include rea_conductivityFieldData and sbd_conductivityFieldData tables, but these data tables will contain a single record for each hoboSampleID, of which there are usually two per sampling event. The 0.1 Hz conductivity data are in the expanded data package, stored in one file per hoboSampleID. The neonUtilities R package (>=2.4.0), using either stackByTable() or loadByProduct(), stacks the conductivity data files and names the resulting table rea_conductivityRawData or sbd_conductivityRawData. To produce the equivalent of the previous xxx_conductivityFieldData table, join the xxx_conductivityFieldData to the xxx_conductivityRawData, joining on the hoboSampleID.
Users of these data should update neonUtilities to version 2.4.0 or greater to ensure correct data stacking. The reaRate package has not been updated; users will need to join the rea_conductivityFieldData and rea_conductivityRawData tables before running the package functions. An example of how to make this join is available in the NEON-reaeration GitHub repo in the exampleScript.R file in the testingAndDev folder. Researchers with additional questions should use the Contact Us form and select DP1.20190.001 in the Data Product dropdown list.
The updated table structure will be implemented on November 2, 2023 in all Provisional data. The update will appear in RELEASE-2024 (to be published January 2024) and Provisional data moving forward. Because Releases do not change, the previous table structure will persist in RELEASE-2021, RELEASE-2022, and RELEASE-2023.
