2. Download correlation data (Correlator Downloader)

Figure 2-1. The Correlation Downloader application is a single-window Java interface allowing you to specify all aspects of data types, filters and modes of download as you proceed from top to bottom and left to right.

2.1. Load a control file

You can optionally Load a data control file if you generated such a file in an earlier session. It contains all settings last saved by the user/owner of the file. Note:  The file is not managed in LIMS, it is up to you to keep it on a local drive. To load a control file:

• Use the Browse button to navigate your directory.
• Click the Load button to set the control file configurations.
  • The data type and sample parameters tables are populated with the values from the control file.
  • If previously saved parameters are fine, nothing else needs to be done. Proceed to 2.4. If changes to data type parameters need to be made, use the Select data type panel.

2.2. Select data type and filters

Two windows/tabs are available for selecting data types, use the appropriate one to specify analysis/component to be downloaded:

•  Quick select lists the data types typically used for stratigraphic correlation.
• Generic select allows user to select any data parameter available in the LIMS database – this should rarely be needed.

Regardless of your parameter selection, each download will always update a Sections file in your specified download directory. This is a list of the sections corresponding to the sample range selected in the Select sample area (see below) with their length as well as top and bottom CSF-A depths. The Correlator (or equivalent) application can use this table to compute accurate splice interval boundaries.

A special feature exists in the Quick select list: Drilling Disturbance Observations. This selection generates a table file with rows of sample identities (expedition, site, hole, core, type, section top offset, bottom offset) and CSF-A depths, every 0.5 cm for the interval corresponding to the samples selected in the Select sample area (see below). In addition, the table has two columns for Disturbance intensity and Disturbance flag, respectively, with the values subsampled every 0.5 cm from the two corresponding DESClogik parameters logged by core describers. These table values can then be plotted in Correlator and used to assess visually the effect of drilling disturbance on the core logging data.

To select data types:

• Select the data type (component) you want to add to the filter.
  • Note: If the data type already exists in the settings display panel to the right, it must be removed using the Remove button before it can be added with new filter parameters. This is to avoid that multiple data files with different filters exist and are downloaded.
• While the data type (component) is selected, set the filter parameters:
  • Consider the Split by instrument option.
• In a few cases, a data type (e.g., GRA density, magnetic susceptibility) may have been collected on multiple instruments (e.g., two core loggers). By default, one file is created for all data, assuming the data are comparable for all practical purposes. If you want to split the data into separate files per instrument, click this checkbox.
  • Set the following optional filters as needed:
• Set minimum value in units reported for this parameter;
• Set maximum value in units reported for this parameter;
• Set section top trim as an interval (cm).
• Set section bottom trim as an interval (cm).
• Set “# of measurements” to be trimmed from top and bottom ends of section.
• Check “Exclude curated section end points” to eliminate measurements taken near the end caps, with irregular edge and useless results.
• Check “Exclude measurements past section curated length” to eliminate ‘data overlaps’ created when sections continue to ‘grow’ while being processed through the lab.
• Click Add button to add the parameter with its settings to the Settings panel.
  • Note: If you want to change the filter settings for a data parameter, first select the row with the previous settings and use the Remove button, then configure and add the new settings.

2.3. Set the hole and core

If changes to expedition, site, hole, and/or core are needed, use the Select sample panel.

• Enter site, hole (and core) selection in the sample selector to the left.
• Click the Add button to add the selection to the Settings panel.
  • Note 1: You can specify only one hole or one core at a time and add them to the Settings panel. Each row in the settings table will generate a file for each data parameter. You can add as many rows as needed.
  • Note 2: If you want to change the site, hole, core settings, first select the row with the previous settings and use the Remove button to remove them from the configuration table, then configure and add the new filter settings.
• IMPORTANT: If you want to append the data for a core to an existing hole file (to avoid redundant downloading of same data), check the Append core data to hole check box.
  • For this to work properly, the hole file to which you append core data must exist in your data folder without a core designation in the file name.
  • This probably isn’t an issue in the shipboard workflow as you first download whatever exists for the hole, and then later add individual core data as they become available.
  • However, if you first download core 1 explicitly and if you don’t remove the “-1” from the file names, your next download for core 2 will create a new set of files.
• ALSO IMPORTANT: Although the interface allows you to select multiple cores, you cannot download data for a set of cores (it’s a benign bug). You can either download all data for a hole, or all data for one core.

2.4. Apply an external cull file

In the Apply external interval culling file area, the user has the option to select and apply a list of intervals for which the data shall be culled. The intervals may originate from any source of information in any of the labs, and are entered by assigned users via Excel or a text editor.

About the file:

• Users keep the file on a local drive.
• The file layout has to follow the following column format: Site, Hole, Core, Coretype, Section, Top (cm), Bottom (cm), Comment(s).
• The format must be CSV.

To apply the culling file:

• Click the Browse button, navigate your directory, and select the file. The file and its path will be saved in the data control file so updated versions can easily be used in iterative data downloads.
• Click the Apply check box if you want the interval data to be culled.
• Click the Remove button to remove the path the file if you want to browse to a different file.

2.5. Save a control file

The user has a Save control file button to save the latest changes.

• The message “unsaved changes” appears in red next to the button if filter parameters were changed but not saved to the control file.
• You don’t need to save the parameters to the control file. The download will use whatever parameters are in the Settings display panels.
• You can keep multiple data control files on your local computer to choose from in future downloads.

2.6. Download the data

Use Download correlation data button to download the data file(s).

• A directory browser window will open where you can set the location for the files to be stored.
• A single CSV file will be created for each hole and data type.
• Best practice is to create a new folder for each site because (1) many files will be in use during an expedition project; and (2) user can more easily go back to working with a previous site.

3. Upload and manage affine table and splice interval table files

3.1. Confirm file format and content

Before attempting upload of an affine or splice file using the SCORS Uploader and Manager application, ensure that it complies with the specified format (see Appendix 1 and 2, or recent examples from the LIMS database). The Correlator application should be generating the correct formats. The file must be CSV format and have the prescribed number and types of columns. The uploader will perform a limited number of tests and will reject non-compliant files.

3.2. Load affine

To load an affine table:

• Select the Load affine tab from among the available tabs (Fig. 3-1):
  • Load affine (will create a new CCSF depth scale)
  • Load splice (requires associated affine to be loaded already)
  • Manage (e.g., remove redundant affine/depth scales and splices)

Figure 3-1. SCORS Uploader user interface, Load affine tab.

• On the Select affine table file line, click the Choose File button and navigate to the file in your directory.
• Click Open/Load in your navigator
• On the Select expedition that created the affine line, select the expedition (project) from the choice list. It will appear in the affine name.
• Click Validate Affine button.
  • If the validation fails, you will get an error message that should be very explicit in terms of type and location of error. The most common errors encountered in initial uploads were:
• Scrambled character for negative sign, presumably generated by Excel that interpreted the value as a formula.
• Commas in the comment columns – these are not allowed in a CSV file.
• Commas only in the last row, which are not visible when viewing and editing the file in Excel, but easily so in a text editor.
  • If the validation passes, a blinking message will appear: Affine validation in progress, please wait.
  • During the validation, the field Name generated from content will be populated; it includes the sites(s) and hole(s) present in the file you loaded.
  • The Current date (YYYYMMDD) field will also be populated.
  • If validation fails, the message Validation failed, upload aborted, check the affine file format and content will appear. Click OK and check your file.
  •  If validation is successful, a new blinking message appears: Affine table file upload to asset manager in progress – please wait. Once this message disappears, you can proceed.
• On the line Enter optional name extension (maximum 10 characters) you can add your initials, the time of day, or any other potentially useful information you want to appear in the CCSF scale name. The number of characters is limited to 10 to avoid overly lengthy file names. When leaving the field, the display labeled Final affine and depth scale name will display the complete standard name, including the expedition number you selected.
• On the line Add new alternate scale or select existing affine/scale to replace, you can make an important decision:
  • If you want to add a new affine and associated CCSF depth scale to the data base without replacing an existing one, keep the default Add a New Record in the choice list.
  •  If you want to replace an existing affine and associated CCSF depth scale, select the appropriate name from the choice list. The affine table and associated depth scale will be removed from the database. This action will reduce the accumulation of outdated information, user confusion, and the need for later cleanup in the database. However, consider that in some cases, it may be desirable to keep an older version (particularly if update is provided postcruise) because investigators may have been working with it.
• Add any information useful to other correlation specialists or users of the depth scales in the Affine description field. The limit is 250 characters.
• To complete the process, click the Upload Affine button.
  •  A blinking message will appear: Affine transfer and CCSF depth scale creation in progress
  • The program will display a message when upload is successfully completed.

To confirm successful upload, go to the LIMS reports described below.

The uploaded affine files are stored and accessible in exactly the way they were uploaded. In addition, the affine tables are converted into Oracle database tables such that the cumulative offsets can be used to compute the alternate CCSF depths for samples and results. For any report available within the LIMS Reports framework, users can select the appropriate CCSF scale from the alternate depth choice list. At that point, the CCSF depths are reported for each observation or sample in the report, provided the associated core was included in the affine table to begin with.

3.3. Upload splice tables

To load a splice interval table:

• Select the Load splice tab from the three tabs at your disposal (Fig. 3-2).
• Follow the same procedure as described for loading an affine. The following two steps are unique to loading a splice.
• On the line Select affine for this splice, select the affine that applies to the splice you are uploading. Failure to select the correct affine will certainly produce unexpected results.
• On the line Select optional splice tie point table file you can optionally browse to your splice tie point table (STPT), click the Load STPT button, which will load the file along with the splice interval table file. The STPT will not be used in splice calculations.

The uploaded splice files are stored and accessible in exactly the way they were uploaded. In addition, the splice interval tables are converted into Oracle database tables such that can be used to provide data by splice.

A splice was associated with an affine table at the time of upload and the corresponding CCSF scale is automatically set when user selects a splice from a choice list in the LIMS Report interface.

Figure 3-2. SCORS Uploader user interface, Load splice tab.

3.4. Manage affine tables (CCSF scales) and splices

The Manager tab allows authorized users to delete redundant affine (and associated depth scales) and splices from the database. Unauthorized users cannot use this tab.

If a user cancels an affine table by accident, it cannot be re-instated through the tools provided. However, a developer or database administrator can still recover the original upload files if needed (for a period of at least one year after).

To cancel an affine and/or splice:

• Select the Manager tab (Fig. 3-3)
• Select the affine or splice you want to cancel and click the Cancel affine or the Cancel splice button.
  • Note: If a splice has been associated with the affine at some point, the splice has to be canceled first. This is to ensure that “orphaned splices” do not exist in the database.
• If you are canceling a splice, consider if you want to cancel the associated affine/CCSF scale as well (in case it was updated). If so, select the affine you want to cancel and click the Cancel affine button.

Figure 3-3. SCORS Uploader user interface, Manager tab.

4. Optional: Check/fix the splice interval file

4.1. Purpose

The splice interval table (SIT) files generated by the Stratigraphic Correlation Specialist (SCS) using third party correlation applications of their choice may contain information that is inconsistent with the sample registry data in LIMS. Uploading inconsistent information to the LIMS would create unexpected results further downstream, particularly when LIMS retrieves data by splice, or when investigators use the splice table to sample cores. Correlator v. 1 had this problem because it computed section and splice interval boundaries from the data. Correlator v.2 and later should not have such problems anymore and this application is not needed when using that application. However, since the JRSO cannot control what application the SCS use, and how those applications compute depths and section-offset identities, we keep the Splice File Fixer, available to help create splice intervals that are consistent in terms of depth and section offsets.

The Splice File Fixer creates two CSV files for each splice input file:

• A difference report file that has the same name as the input file plus the name extension *DIFF. This file has two rows for each input row (splice interval):
  • INPUT line with the same numbers as read from the input file; it is followed by the
  • OUTPUT line with corrected numbers based on the logic described below.
• A corrected splice file that has the same file name as the input file plus a name extension *CORR. It has exactly the same columns as the input file and is the file to be uploaded to LIMS if the *DIFF file doesn’t contain significant issues.

4.2. Run Splice File Fixer

Before running the Splice File Fixer, make sure the affine table was loaded to the LIMS because the CCSF scale must have been created in the LIMS before a splice can be tested and corrected. Ensure that the SIT complies with the specified format (Appendix 2). The file must be CSV format and have the prescribed number and types of columns.

Figure 4-1. SpliceFileFixer user interface.

To run the SpliceFileFixer (Fig. 4-1):

• On the Select splice interval table input file line, click the Browse button and navigate to the input SIT file in your directory.
• Click Open/Load in your navigator
• On the Associate existing affine table (CCSF depth scale) line, select the scale corresponding to your SIT. If you don’t see the scale corresponding to a recently uploaded affine table, click the Refresh Affine button.
• If you want to change the threshold value for reporting differences between input and output offset and depth values, set it in the Differences will be reported if they are greater than line.
• Click Validate Input File, and Generate Corrected File and Difference Report button.
  • You will see a log of processing steps, with the final reading: “The splice interval table processing has ended or has completed.”

You can now analyze the resulting *DIFF file and upload the *CORR file if satisfied.

4.3. The logic

The Splice File Fixer logic assumes that users trust their selection of core identity (hole-core-section) and CCSF depths for the splice intervals, but may not be certain about the computations of corresponding CSF-A depths and section offsets.  The following is the approximate sequence of tests and computations performed by the Splice File Fixer application.

• Create a copy of the input file. Floating point values for offset and depth values are rounded to 1 mm precision fixed values to avoid issues with very small floating-point number differences in depth comparison tests.
• For each splice interval:
  • Get all the sections for the given core. Each section record in LIMS has a stored CSF-A as well as CCSF depth (based on loaded affine table) for both top and bottom of section and the array of these four values per section will be used here.
• Proceed with the following steps separately for top and bottom of the splice interval, respectively.
  • Find the section that contains the given CCSF depth.
• If the CCSF for the given core is smaller than the smallest section top depth, return message “ABOVECORE” in the *DIFF output file. This will fail splice file upload and user needs to address the issue somehow.
• If the CCSF for the given core is greater than the largest section bottom depth, return message “BELOWCORE” in the *DIFF output file. This will fail splice file upload and user needs to address the issue somehow.
• If two section IDs are returned because the CCSF corresponds exactly to the bottom of a section and the top of the subsequent section, select the section with the smaller section number.
  • For the section determined to be the correct one, compute

offset = CCSF(given) – CCSF(section_top).

• • Then compute CSF-A = CSF-A(section_top) + offset
  • Print the half-line to the output file. Repeat with the bottom of interval if this was the top.
  • Proceed with the next splice interval.
• Read the discrepancy threshold value provided by the user (default: 1 cm; Fig. 4-1).
  • If discrepancies in section offset and CSF-A depths are larger than the threshold value they will be considered significant and will prompt a difference report, consisting of the OUTPUT line printed below the INPUT line.
• The file has the same name as the input SIT file plus the extension *DIFF.
• The file allows users to review potential issues quickly and take action if needed. A corresponding screen view is also created.
  • If differences in section offset and CSF-A depths are equal to or smaller than the threshold value they will be considered insignificant and are not reported.
• Print the output file *.DIFF.
• Print the DIFF report on the screen, in the application user interface, as well.

5. Reports

5.1. Correlation (affine and splice) table reports

Stratigraphic correlation data is reported in four types of reports (Fig. 5-1, lower left):

• List of affine tables
• Affine tables
• List of splice interval tables
• Splice interval tables

Figure 5-1. Screen capture of correlation reports options within the LIMS Reports framework.

List of affine tables report

This report lists the CCSF alternate depth scales available in LIMS, and the original affine table files that the scales are derived from, one per row (Fig. 5-2). Metadata for each scale are provided as well (see Table 6 for column descriptions). The list can be filtered by entering an expedition number.

To get a list of affine tables report (Fig. 5-2):

• In the Select report panel, expand the Stratigraphic Correlation list and select List of affine tables report.
• In the Hierarchy Search area, filter by Expedition(s), if so desired.
• View data on screen using the View data button.
• Option: download data in CSV format using the Download tabular data button.

Figure 5-2: List of affine tables report.

Table 5-1. Description of List of affine tables report columns.

Affine tables report

This report provides the full affine table information, one core per row (Fig. 5-3; see Table 5-2 for descriptions of columns). Sample identity information (i.e., hole and core) and the cumulative offset for each core resulting from depth shifting, are derived directly from user-uploaded files. However, the CSF-A and CCSF depths as well as differential offset and growth rate are computed in LIMS, based on latest sample registry data for hole and core. The original, unaltered affine table user file can also be downloaded.

To get an affine table report (Fig. 5-3):

• In the Select report panel, expand Stratigraphic Correlation list and select Affine Tables report.
• In the Additional Filters area, use the Alternate depth scale choice list to select the scale corresponding to the appropriate affine.
  • You can reduce the choice list by selecting an expedition from the Expedition(s) filter in the Hierarchy Search area.
• View data on screen using the View data button.
• Option: download data in CSV format using the Download tabular data button.

Figure 5-3: Affine table report.

Table 5-2. Description of Affine tables report columns.

List of splice interval tables report

This report lists the splice interval tables available in LIMS, one per row, including metadata and links generated by the LIMS or added by the user at the time of file upload (Fig. 5-4; see Table 5-3 for description of columns). The list can be filtered by entering an expedition number.

To get a list of splice interval tables report (Fig. 5-4):

• In the Select report panel, expand Stratigraphic Correlation list and select List of Splice Interval Tables report.
• In the Hierarchy Search area, filter by Expedition(s), if so desired.
• View data on screen using the View data button.
• Option: download data in CSV format using the Download tabular data button.

Figure 5-4: List of splice interval tables report.

Table 5-3. Description of List of splice interval tables report columns.

Splice interval tables report

This report provides the full splice interval table information, one interval per row (Fig. 5-5; see Table 5-4 for descriptions of columns). Sample identity information (i.e., hole, core, section, offset) is derived directly from user-uploaded files. However, the CSF-A and CCSF depths are computed in LIMS, based on latest sample registry data for hole and core. The original, unaltered splice interval table user file can also be downloaded.

To get an affine table report (Fig. 5-5):

• In the Select report panel, expand Stratigraphic Correlation list and select Splice table report.
• In the Splice Search area, use the choice list to select the appropriate splice.
  • You can reduce the choice list by selecting an expedition from the Expedition(s) filter in the Hierarchy Search area.
• View data on screen using the View data button.
• Option: download data in CSV format using the Download tabular data button.

Figure 5-5: Splice table report.

Table 5-4. Description of Splice interval tables report columns.

5.2. Alternate depths

This report feature provides CCSF alternate depth for any sample and data point in any data report that uses depth (CSF-A) as the primary independent variable. User must select the appropriate alternate scale for a report.

To retrieve CCSF depths with data (Fig. 5-6):

• In the Select report area, select desired data type by expanding the category of interest.
• In the Hierarchy Search area, filter Expedition(s), Site(s), Hole(s), Core(s), etc., as needed.
• In the Additional Filters area, select the appropriate Alternate depth scale from the choice list.
• View data on screen using the View data button. The alternate depths will be shown in the column nest to the CSF-A depths.
• Option: download data in CSV format using the Download tabular data button.

Figure 5-6. Data report example with alternate depths shown in column #10.

5.3. Data by splice

One of the major goals of stratigraphic correlation is to generate “spliced data”, or to retrieve “data by splice”. To get data by splice (Fig. 5-7):

• In the Select report area, select desired data type by expanding the category of interest.
• In the Splice Search area, select the appropriate splice from choice list.
• You can filter the splice list by selecting an expedition in the Hierarchy Search area.
• View data on screen using the View data button. The alternate depths will be shown in the column nest to the CSF-A depths.
• Option: download data in CSV format using the Download tabular data button.

Figure 5-7. Data set extracted for user-selected splice.

Appendix 1: Specifications for affine tables

The affine table exported from Correlator has the 15 columns described in the table below. In order to take advantage of the features offered from within the LIMS database, affine tables exported from Correlator or any other correlation programs must meet import requirements of the SCORS Uploader and Manager application. As a minimum requirement, the first 7 columns must be present.

Appendix 2: Specifications for splice interval tables

The splice table exported from Correlator has the 15 columns described in the table below. In order to take advantage of the features offered from within the LIMS database, splice tables exported from Correlator or any other correlation programs must meet import requirements of the SCORS Uploader and Manager application.

 Archive Versions

SCORS LIMS User Guide - 27Sept2022

SCORS LIMS User Guide - 24022020