Content
Summary
Post-processing large bioactivity data harmonizes disparate experimental records from public repositories like PubChem into interoperable, FAIR-compliant databases. This process resolves issues in terminology, format, and data provenance, enabling the integration of billions of data points for predictive modeling and drug discovery. It distinguishes between raw test result values and the specific justifications that support final experimental outcomes.
Intuition
Think of post-processing as a massive translation and organization effort. Public repositories like PubChem have made data findable and accessible, but the data often lacks interoperability due to diverse sources and formats. Post-processing acts as the bridge, taking these disconnected bioassay records and rebuilding them into a structured, relational database. This allows researchers to perform reliable queries across millions of records, turning raw, messy data into a resource that can drive computational technologies like QSAR and machine learning. The goal is not just storage, but creating a system where data is Findable, Accessible, Interoperable, and Reusable (FAIR).
Derivation
- Identify the problem: Data in repositories like PubChem is findable and accessible but lacks interoperability due to diversity in data types and sources.
- Define the solution: Post-processing integrates separate experimental records into an interoperable and relational database.
- Establish the goal: The primary objective is to make data FAIR (Findable, Accessible, Interoperable, Reusable) to support reuse cases in pharmacology and toxicology.
- Describe the method: The process harmonizes terminologies, concepts, and formats to resolve complexity in bioassay designs.
- Highlight efficiency: The approach is designed to be cost-effective and easy to develop, allowing processing of billions of data points with limited programming skills.
- Specify technical details: Use of ontologies like BAO and MIABE aids in designing the database to represent bioassay concepts effectively.
- Address provenance: The process resolves 'Justifications' by flagging specific data points (e.g., IC50) that support an experimental Outcome, distinguishing them from raw test result values (TIDs).
Worked example
A researcher wants to use PubChem data for a QSAR model but finds the data inconsistent and difficult to query across different bioassay types. How does post-processing address this?
- The researcher identifies that while PubChem data is accessible, it lacks the interoperability needed for structured queries.
- Post-processing tools harmonize the disparate terminologies and formats from various sources.
- The system distinguishes between raw Test Result Values (TIDs) and the 'Justifications' (like specific IC50 values) that directly support an experimental Outcome.
- Ontologies such as BAO and MIABE are applied to standardize the representation of bioassay concepts.
- The data is rebuilt into a relational database, enabling the researcher to perform reliable, structured queries across millions of records.
Answer Post-processing transforms the raw, heterogeneous PubChem data into a structured, FAIR-compliant database. By harmonizing formats and clearly defining Justifications versus raw values, it enables the researcher to efficiently query and integrate billions of data points for their QSAR model.
Common mistakes
- Assuming that making data findable and accessible (as done by PubChem) automatically ensures interoperability and reusability.
- Failing to distinguish between raw test result values (TIDs) and the Justifications that support a final experimental Outcome.
- Overlooking the need for curation and meta-information enrichment required to achieve high interoperability.