DC_OCEAN: an open-source algorithm for identification of duplicates in ocean databases
Author(s)
Song, Xinyi
Castelao, Guilherme
Language
English
Obiettivo Specifico
OSA4 Ambiente marino, fascia costiera ed Oceanografia
Status
Published
JCR Journal
JCR Journal
Peer review journal
Yes
Journal
Issue/vol(year)
/11(2024)
ISSN
2296-7745
Publisher
Frontiers SA
Pages (printed)
1403175
Date Issued
October 2, 2024
Abstract
A high-quality hydrographic observational database is essential for ocean and climate studies and operational applications. Because there are numerous global and regional ocean databases, duplicate data continues to be an issue in data management, data processing and database merging, posing a challenge on effectively and accurately using oceanographic data to derive robust statistics and reliable data products. This study aims to provide algorithms to identify the duplicates and assign labels to them. We propose first a set of criteria to define the duplicate data; and second, an open-source and semi-automatic system to detect duplicate data and erroneous metadata. This system includes several algorithms for automatic checks using statistical methods (such as Principal Component Analysis and entropy weighting) and an additional expert (manual) check. The robustness of the system is then evaluated with a subset of the World Ocean Database (WOD18) with over 600,000 <jats:italic>in-situ</jats:italic> temperature and salinity profiles. This system is an open-source Python package (named DC_OCEAN) allowing users to effectively use the software. Users can customize their settings. The application result from the WOD18 subset also forms a benchmark dataset, which is available to support future studies on duplicate checks, metadata error identification, and machine learning applications. This duplicate checking system will be incorporated into the International Quality-controlled Ocean Database (IQuOD) data quality control system to guarantee the uniqueness of ocean observation data in this product.
Type
article
File(s)![]()
Name
Song2024fmars-11-1403175 (1).pdf
Type
Main Article
Size
4.37 MB
Format
Adobe PDF
Checksum (MD5)
The MD5 message-digest algorithm can be used to verify the integrity of the file that you have uploaded. You can calculate its value locally via tools that are generally available in each operative system like md5sum
c9d0dcfaab4958ceb18662b29ee592bd
