close
Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2011 Mar 9:12:71.
doi: 10.1186/1471-2105-12-71.

LabKey Server: an open source platform for scientific data integration, analysis and collaboration

Elizabeth K Nelson  1 , Britt Piehler, Josh Eckels, Adam Rauch, Matthew Bellew, Peter Hussey, Sarah Ramsay, Cory Nathe, Karl Lum, Kevin Krouse, David Stearns, Brian Connolly, Tom Skillman, Mark Igra
Affiliations

LabKey Server: an open source platform for scientific data integration, analysis and collaboration

Elizabeth K Nelson et al. BMC Bioinformatics. .

Abstract

Background: Broad-based collaborations are becoming increasingly common among disease researchers. For example, the Global HIV Enterprise has united cross-disciplinary consortia to speed progress towards HIV vaccines through coordinated research across the boundaries of institutions, continents and specialties. New, end-to-end software tools for data and specimen management are necessary to achieve the ambitious goals of such alliances. These tools must enable researchers to organize and integrate heterogeneous data early in the discovery process, standardize processes, gain new insights into pooled data and collaborate securely.

Results: To meet these needs, we enhanced the LabKey Server platform, formerly known as CPAS. This freely available, open source software is maintained by professional engineers who use commercially proven practices for software development and maintenance. Recent enhancements support: (i) Submitting specimens requests across collaborating organizations (ii) Graphically defining new experimental data types, metadata and wizards for data collection (iii) Transitioning experimental results from a multiplicity of spreadsheets to custom tables in a shared database (iv) Securely organizing, integrating, analyzing, visualizing and sharing diverse data types, from clinical records to specimens to complex assays (v) Interacting dynamically with external data sources (vi) Tracking study participants and cohorts over time (vii) Developing custom interfaces using client libraries (viii) Authoring custom visualizations in a built-in R scripting environment. Diverse research organizations have adopted and adapted LabKey Server, including consortia within the Global HIV Enterprise. Atlas is an installation of LabKey Server that has been tailored to serve these consortia. It is in production use and demonstrates the core capabilities of LabKey Server. Atlas now has over 2,800 active user accounts originating from approximately 36 countries and 350 organizations. It tracks roughly 27,000 assay runs, 860,000 specimen vials and 1,300,000 vial transfers.

Conclusions: Sharing data, analysis tools and infrastructure can speed the efforts of large research consortia by enhancing efficiency and enabling new insights. The Atlas installation of LabKey Server demonstrates the utility of the LabKey platform for collaborative research. Stable, supported builds of LabKey Server are freely available for download at http://www.labkey.org. Documentation and source code are available under the Apache License 2.0.

PubMed Disclaimer

Figures

Figure 1
Figure 1
The Atlas data repository. Atlas facilitates collaboration among scientists, clinics and labs distributed across the globe by acting as a centralized, integrated, secure data repository. It currently serves the following members of the Global HIV Vaccine Enterprise: the Center for HIV Vaccine Immunology (CHAVI), the Collaboration for AIDS Vaccine Discovery (CAVD), the HIV Vaccine Trials Network (HVTN), the Microbicide Trials Network (MTN), the Vaccine Immunology Statistical Center (VISC) and the HIV Prevention Trials Network (HPTN).
Figure 2
Figure 2
LabKey Server's modular architecture. LabKey Server consists of core services and specialized modules. Modules are autonomous, so they can be added or upgraded independently to add new functionality.
Figure 3
Figure 3
LabKey Server query service. The query service can be called on by LabKey Server's APIs or web-based interface. In either case, the service receives a request for one of the following: • A table and a column list. A column list can include columns from the requested table or columns from related tables. • A SQL query based on pseudo-tables known to the query service. For an API call, the following sequence of events occurs: 1. When the request is received by the server's API layer, the layer checks folder security and translates the request into calls to the query service. 2. The query service then uses schema information describing physical tables and pseudo-tables to translate the input query into a SQL query of physical tables. The query is formulated in the dialect understood by the underlying relational database. Schema information is supplied by other LabKey modules. 3. The database returns a tabular result. 4. The tabular result is annotated with additional information about the columns (e.g. user-friendly label, description and formatting hints). 5. The appropriate LabKey client library converts this standard data/metadata into a form easily understood by the client language. For example, an R dataset would be returned as the result of a call by an R client API.
Figure 4
Figure 4
Typical setup steps for specimen tracking and requests. 1. An administrator sets up a folder to display the "Specimens" web part and configures permissions for those who will be interacting with specimens. The administrator then imports an initial archive of specimen records and information about repositories and sites. A populated "Specimens" web part is shown in A. 2. A specimen data manager configures the specimen request process, including configuring request steps (B), identifying actors in the request process (C), configuring requirements for request approval, and setting up notification procedures (D).
Figure 5
Figure 5
Typical steps for vial search and request. 1. A scientist logs on the system from a participating lab, then searches for vials or specimens of interest, as shown in A. 2. To further narrow down the possible vials of interest, the scientist builds a custom view of the subset of vials she has identified through search. This custom view integrates information from diverse sources using shared identifiers. In the example shown in B, shared participant and visit identifiers are used to join in data from a related dataset. In this case, the gender of the participant who provided the sample (as defined in a separate, demographic dataset) is drawn into the vial data view. 3. The scientist then creates a new specimen request and uses the "specimen shopping cart" to add desired vials to this request, as shown in C. 4. When finished, the scientist submits the finished request, as shown in D. 5. Designated reviewers are notified of the requests and approve them. 6. Specimen repository workers are notified of the approved requests, fill the requests and update the web-based interface. Vial(s) that have been used are no longer requestable. 7. After receiving a specimen vial, the scientist performs an assay on the specimen. Results from this assay may be marked with the vial identifier or participant/visit identifiers so that they can be associated with specimen information on the LabKey Server. The process for associating assay results with specimens is shown later in Figure 7.
Figure 6
Figure 6
Typical steps for designing an assay. 1. An administrator creates an assay-type folder to act as a staging area for assay data before the data has undergone quality control. 2. A scientist creates a new assay design to match the contents of experimental results stored in spreadsheets. In this example, the assay design is based on a generalized assay template, but other assay types could be used. The assay design is named "GenericAssay," as marked by A. 3. The scientist then adds a set of batch fields that must be filled out when each batch of runs is imported. User-defined batch fields are backed by the same set of customizable properties as system-defined fields. This image shows label, description, type, lookup and custom URL properties for several fields. Other properties include conditional formatting, default values and regular expression validators. Here, the "Machine" batch field (B) is configured (in popup C) as a lookup to a simple list ("Lab Machines"). At the time of data import, the user will be presented with a defined list of options for populating the "Machine" field that are drawn from the primary keys of the "Machines" list. 4. Next, the scientist defines the run fields (D) that must be collected for each run. In this example, only one field ("MachineSetting") is defined. 5. Finally, to save time, the scientist infers the assay's data fields (E) from a representative spreadsheet file. These fields could also have been designed manually, in the same manner as the batch and run fields.
Figure 7
Figure 7
Typical steps for populating an assay. 1. The user selects the assay design that matches experimental data and chooses to sequentially import run data. This example uses the assay design from Figure 6. 2. For each batch of data, the user is prompted for batch properties (A). Here, the "Machine" options are provided as a defined vocabulary to reduce errors and variability in data entry. 3. For each run, the user is prompted for run properties and run data (B), as described in the assay design. A template of expected columns can be exported to help with matching data formats. 4. The user can import several runs sequentially using the same batch properties. C shows a summary view of the three runs that have been imported to this assay design using the same batch properties. 5. D shows an example of results imported as a single run for this assay. Note that the "Machine" column is defined as a lookup to another table, so each of its entries is hyperlinked to details for the appropriate machine, as provided by the "Lab Machines" list. 6. After assay data has been reviewed for quality control, it can be moved into a study folder for sharing with collaborators and integration with other types of study data. During the import process, the participant/visit identifiers for each row of assay data are matched (E) to identifiers for specimens in the target study. This allows viewers of the assay data within the study to quickly navigate to data for associated specimens.
Figure 8
Figure 8
Detailed NAb assay results view. The "details" view for a NAb assay visualizes measures of neutralization success and provides several calculations for areas under the neutralization curve.
Figure 9
Figure 9
Example of joining datasets based on built-in relationships. Results from related data sources can be joined into a common grid view using built-in relationships between study entities. Here, the grid view customization tool is used to pull in columns from data sources that include the same participant/visit pairs. The results of an assay ("Lab Results") are joined with the columns from another assay ("HIV Test Results") and the results of a physical exam ("Physical Exam"). The joined custom view can be saved privately, made visible to collaborators or exported in various formats, including Excel.
Figure 10
Figure 10
Examples of analyzing and visualizing a joined custom view. Here, the joined custom view created in Figure 9 is used to construct a participant chart (A), crosstab view (B) and R chart (C and D) through the "Views" menu available above the joined grid. A. The participant-specific chart view marked by A displays the same type of information for each study participant, in sequence. Users can toggle between the displayed chart and its associated dataset using the "View" and "Data" tabs. This particular chart displays the progression of HIV viral loads and CD4+ counts over time for each study participant. B. A crosstab view like the one shown in B could be used to verify that different tests for determining HIV status produce consistent results. C. The R script displayed in the "Source" tab of the R script editor (C) uses data from the joined custom view to compare how CD4+ counts and HIV viral loads change over time. The scripting environment makes the source dataset available as the labkey.data data frame, as circled in red. D. The results of the script are displayed by selecting the "View" tab of the R script builder, as shown in D. Selecting the "Data" tab would display the source dataset (not shown). R views, just like other views, can be saved privately or made visible to collaborators with sufficient permissions to view the source data. Within LabKey Server's R environment, users can also invoke stored scripts and leverage advanced analysis or visualization packages, such as those provided by Bioconductor[80].
Figure 11
Figure 11
Example of joining data sources through a user-defined lookup relationship. The "Lab Machines" list, a simple table, was used in Figure 6 to provide a defined vocabulary for populating the "Machine" batch field of the "GenericAssay." The "Machine" batch field was defined as a lookup to the "Lab Machines" list. The existence of this lookup relationship allows us to join machine information into grid views of assay run data. Here, as circled in red, the relevant machine's "Contact Person" is joined into a grid view of assay results.
Figure 12
Figure 12
Example of creating a custom SQL view. This figure demonstrates how a custom SQL view can add a calculated column to a joined view and label the column using custom metadata. Part A of this figure shows LabKey Server's schema browser, which allows a developer to view, add or edit custom queries. Part B shows how the SQL source editor has been used to add a calculated column to a table as part of a custom query. It also shows how the table metadata editor has been used to edit the column's properties and add a custom title. The grid view produced by this custom query is shown in C.
Figure 13
Figure 13
Simplified study schema. This simplified schema outlines the relationships between entities in a study. Data tables in a study can be integrated when they contain shared quantities, such as participants, visits or specimens.
Figure 14
Figure 14
LabKey Server data flows. Data flows into a LabKey Server from diverse sources, undergoes quality control and becomes available to a range of data customers through a web-based portal.
Figure 15
Figure 15
Study portal page. A study portal provides a jumping-off point for investigating patterns in shared datasets. The interface can be customized to list the different types of data or views available in the study and to display custom data tables or visualizations. The "Study Navigator" shown in this figure is a data summary tool that rolls up information from many study datasets based on participant identifiers.
Figure 16
Figure 16
Participant-specific data summaries. A data overview is provided for each participant in a study. It displays all datasets in the study associated with a particular participant across all visit dates. Participant-specific charts can be displayed for each dataset, as shown for the "Physical Exam" dataset. A user can navigate through equivalent views for other individuals using "Next Participant" link and "Previous Participant" links.
Figure 17
Figure 17
A custom, interactive form for designing an R view. Part A of this figure shows an example of an interactive R chart designer. This particular interface (available only on Atlas) was created using the LabKey JavaScript client libraries. It allows users to create custom views of antibody binding data. Users employ a graphical form to select participants of interest, isotypes, antibodies and groupings for plots. The form sends user selections to a parameterized R script that runs on a joined view of relevant datasets. Part B of this figure shows the R view produced by this form. It shows binding antibody graphs by isotype (interferon alpha (IgA) and interferon gamma (IgB)) for two study participants. The graphs plot glycoprotein 41 reading values (GP41) and viral loads for each participant and each isotype against the number of days that have passed between enrolment of the participant in the study and the collection of the tested specimen. Participant identifiers in this figure have been modified to hide study participant identities.
Figure 18
Figure 18
Custom, per-participant views of data. Participant-specific summary views show all data available for a particular participant, typically along with visualizations. These views can be customized using the graphical interface or through custom scripts. This figure shows a custom view built on Atlas using LabKey Server's R interface and JavaScript client library. It is available only on Atlas. Clinical data for the selected participant are displayed alongside a graph that plots viral loads and CD4+ counts against the number of days from enrolment in the study to specimen collection. Users can toggle between different types of data (e.g., "Requested Specimens," "Binding Antibodies" or "Cytokines") using tabs. Just as in the per-participant views shown in Figure 10A and Figure 16, users can employ the "Previous Participant" and "Next Participant" links to progress to the same data summary for other participants in the study. The participant identifier in this image is not real and the physical information for the participant has been obscured.
Figure 19
Figure 19
Custom study summaries defined in an Atlas module. LabKey Server modules can be used to encapsulate scientifically interesting applications. Atlas's vaccine study module defines several interfaces that generate interactive summaries of vaccine studies, including the interface displayed in this figure. This module is tailored to Atlas-specific data, so it is not part of the open source project. Generated summaries like the one shown in this figure make it easier for scientists to swiftly, publicly share study information and protocols after publishing associated research papers. The Atlas web portal respects user permissions, so study summaries display different amounts of information to different tiers of users. Atlas login is not required to view public study data, so you can explore the interfaces and data summaries defined in the vaccine study module without acquiring Atlas credentials: https://atlas.scharp.org/cpas/viscstudies/VISC/Completed%20CAVD%20Studies/studydesigns.view.
See this image and copyright information in PMC

References

    1. Apache License, Version 2.0. http://www.apache.org/licenses/LICENSE-2.0.html http://www.apache.org/licenses/LICENSE-2.0.html
    1. Coordinating Committee of the Global HIV/AIDS Vaccine Enterprise. The Global HIV/AIDS Vaccine Enterprise: Scientific Strategic Plan. Plos Med. 2005;2:e25. doi: 10.1371/journal.pmed.0020025. - DOI - PMC - PubMed
    1. Butler D. AIDS vaccine research becomes 'big science'. Nature. 2006;442:610–611. doi: 10.1038/442610a. - DOI - PubMed
    1. Klausner RD, Fauci AS, Corey L, Nabel GJ, Gayle H, Berkley S, Haynes BF, Baltimore D, Collins C, Douglas RG, Esparza J, Francis DP, Ganguly NK, Gerberding JL, Johnston MI, Kazatchkine MD, McMichael AJ, Makgoba MW, Pantaleo G, Piot P, Shao Y, Tramont E, Varmus H, Wasserheit JN. The Need for a Global HIV Vaccine Enterprise. Science. 2003;300:2036–2039. doi: 10.1126/science.1086916. - DOI - PubMed
    1. Esparza J, Yamada T. The discovery value of "Big Science". Journal of Experimental Medicine. 2007;204:701–704. doi: 10.1084/jem.20070073. - DOI - PMC - PubMed

Publication types

LinkOut - more resources