Skip to main content

Data Management

Using the data management tools, you can access and manage the output files from sequencing runs and flow executions. Data management interfaces are accessible through different tabs on the Run Details pages. The Run Files tab shows the output files from a sequencing run. Other tabs show files from the executions of flows, such as the FASTQ Files tab for Bases2Fastq executions.

Within the data management interfaces, you can perform the following actions:

  • Download output files from a sequencing run or flow execution.
  • Access flow execution details and output files.
  • Reexecute a flow using a new run manifest or execution settings.
  • Stop a flow execution that is in progress.

Compatibility

Data management is available for runs and flow executions that use one of the following providers as a storage connection:

File Availability

The availability of a file depends on the file status, the cloud bucket settings, and the type of associated run activity.

  • Sequencing files populate throughout a run and are in sync with the run status on the instrument.
  • FASTQ files become available when a Bases2Fastq flow execution reaches a final status.
  • Available files depend on the object storage class for files in the cloud bucket. For details on file availability with ElemBio Catalyst, see Data Storage Retention.
  • Archived files are visible but unavailable for download.

Downloading Files

You can download a single file, multiple files, or all files from a sequencing run or flow execution. In the Actions column, you can select the Download icon to download a single file. Selecting the Copy URI icon allows you to copy a Uniform Resource Identifier (URI) to download a file in the AWS CLI.

Downloading multiple files uses a curl config script file, which contains a metadata header with execution instructions and presigned URLs to the selected files. Presigned URLs in the script expire 7 days after creation. The expiry time appears in the header of the file for reference. If the download does not complete within 7 days, you must generate a new script.

For information on output files, see the Element AVITI System User Guide (MA-00008) and the Bases2Fastq Documentation.

Download Multiple Selected Files

  1. On the Run Details page, select the tab with the files you want to download.
  2. Select the checkboxes for the files you want to download.
  3. Select Download.
  4. Review the number of files and total download size to make sure you selected the correct files.
  5. Select Download curl config file and review the information in a text editor.
  6. Open a terminal window for the CLI on your OS.
  7. Select the tab for your OS: MacOS/Linux or Windows.
  8. Copy the curl command and run it in the CLI.

Download All Files

When you download all files for an output location, ElemBio Cloud offers an option to use an AWS CLI command. The command enables more efficient downloads and uses 12-hour temporary credentials that the user interface provides.

If you encounter an error when using the AWS CLI command, make sure your environment variables are set correctly. The env command lets you examine the configuration of the environment variables. For more information on environment variables, see Environment variables to configure the AWS CLI in the AWS Documentation.

  1. On the Run Details page, select the tab with the files you want to download.
  2. Select Download All.
  3. Wait for the interface to prepare the download, and then select Continue to Download.
  4. Select one of the following download options. Follow all prompts.
    • To download using a temporary curl script with a presigned URL, select Download with a script.
    • To download using the AWS CLI tool, select Download with a CLI.

Flow Executions

In the tabs for flow executions, such as the FASTQ Files tab, you can reexecute a flow or stop an in-progress execution. Using the Execution ID drop-down menu, you can access files and information from previous executions.

Reexecute a FASTQ Flow

Reexecuting a FASTQ flow allows you to resolve errors, address quality issues, apply new optional arguments, or use a corrected run manifest.

  1. On the Run Details page, select FASTQ Files.
  2. Select Reexecute.
  3. When prompted, select the run manifest for the reexecution.
    1. Select Browse, and then browse to the run manifest file.
    2. Select the run manifest file, and then select OK.
  4. Enter any optional arguments in the Parameters field.

    For example, to add the QC only mode optional argument, enter --qc-only.

  5. Select Execute Flow.