Data Browsing

ElemBio Cloud offers multiple options for accessing and browsing your data. Data Explorer displays the contents of cloud storage buckets with compatible storage connections, providing full visibility to your files and folder structures. Tabs on the Run Details pages provide data browsers that are limited to individual runs and flow executions. Using these features, you can complete the following tasks:

• Browse and navigate run output files or the output files from flow executions.
• Download files or directories directly to your local environment.
• View key details and metadata, such as file size, archive status, and timestamp of the last update.
• Search and filter files to navigate with efficiency.

Compatibility

Data browsing is compatible with storage connections for the following providers:

• ElemBio Catalyst
• AWS with IAM policies that enable S3:ListBucket and S3:GetObject permissions

File Availability

The availability of a file depends on the file retention status, the cloud bucket settings, and the type of associated run activity.

• Sequencing files populate throughout a run and are in sync with the run status on the instrument.
• FASTQ files and other analysis output files become available when a flow execution reaches a final status.
• Available files depend on the object storage class for files in the cloud bucket. For details on file availability with ElemBio Catalyst, see Data Storage Retention.
• Archived files are visible but unavailable for download. To make archived files available again in ElemBio Catalyst, contact Element Technical Support.

Downloading Files

Using the data browsing features, you can download a single file, multiple files, or all the files in a folder.

• To download a single file, select the Download icon in the Actions column.
• To download a multiple selected files, use the interface to select files and generate a dynamic download script.
• To download all files at a prefix, use the AWS CLI or generate a download script.

For information on output files, see Run Output Files and the Bases2Fastq Documentation.

Note:

The Copy URI icon in the Actions column lets you copy the Uniform Resource Identifier (URI) for a single file. You can use the URIs to individually download files using the AWS CLI.

Download with Script

ElemBio Cloud lets you download multiple selected files or all files at a prefix using a download script. The script uses the curl protocol and relies on presigned URLs that are dynamically generated.

The script contains a metadata header with execution instructions and presigned URLs for the selected files. Executing the script creates a directory on your system and downloads files into it.

Before executing the script, make sure to meet the following requirements:

• Install or update curl v8.4 or later on your system. To check the version, use the command curl -V.
• If you are using Windows OS, use the curl.exe command, and execute the command in Windows Command Prompt or PowerShell.
• Execute before the presigned URLs expire. The URLs expire 7 days after creation, and the time of expiration appears in the script header. If you do not complete the download within 7 days, you must generate a new script to download the files.

Download Multiple Selected Files

1. Select the checkboxes for the files you want to download.
2. Select Download.
3. Review the number of files and total download size to make sure you selected the correct files.
4. Select Download curl config file and review the file in a text editor.
5. Open a terminal window for the CLI on your OS and navigate to the location of the curl config file.
6. In ElemBio Cloud, select the tab for your OS: MacOS/Linux or Windows.
7. Copy the curl command and run it in the CLI.

Download All Files with the Download Script

1. Select Download All.
2. Wait for the interface to prepare the download, and then select Continue to Download.
3. Select Download with a script.
4. Review the number of files and total download size to make sure all necessary files are selected.
5. Select Download curl config file and review the file in a text editor.
6. Open a terminal window for the CLI on your OS and navigate to the location of the curl config file.
7. In ElemBio Cloud, select the tab for your OS: MacOS/Linux or Windows.
8. Copy the curl command and run it in the CLI.

AWS with CLI Command

ElemBio Cloud offers an option to use an AWS CLI command to download all files at a prefix. The command uses 36-hour temporary credentials that the user interface provides. Before you download data with the AWS CLI, you must configure three credential variables:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_SESSION_TOKEN

To set the credential variables, configure your environment using one of the following methods:

• Set credentials as local variables that the AWS CLI reads automatically.
• Set credentials using the AWS configuration and credential file with the aws configure command.

Download All Files with the AWS CLI

Select the tab for your preferred method of credential configuration.

AWS CLI with Local Variables

When you set the credential variables in the local terminal environment, the AWS CLI automatically detects them as known environment variables. Linux OS uses the export command, while Windows OS uses the set command to set the variables.

1. Select Download All.
2. Wait for the interface to prepare the download, and then select Continue to Download.
3. Select Download with a CLI.
4. Review the number of files and total download size to make sure all necessary files are selected.
5. Open a terminal window to access the AWS CLI.
6. Set the credentials provided by ElemBio Cloud. Use the commands for your OS.

Setting Example Credentials in Linux OS

export AWS_ACCESS_KEY_ID=ASIA52AWRNEXAMPLE
export AWS_SECRET_ACCESS_KEY=R8GTtGx7WwzQ9L1WQbPnHLEXAMPLE
export AWS_SESSION_TOKEN=R8GTtGx7WwzQ9L1EXAMPLE_SESSION_TOKEN_FROM_EBC
export AWS_DEFAULT_REGION=us-west-2

Setting Example Credentials in Windows OS

set AWS_ACCESS_KEY_ID=ASIA52AWRNEXAMPLE
set AWS_SECRET_ACCESS_KEY=R8GTtGx7WwzQ9L1WQbPnHLEXAMPLE
set AWS_SESSION_TOKEN=R8GTtGx7WwzQ9L1EXAMPLE_SESSION_TOKEN_FROM_EBC
set AWS_DEFAULT_REGION=us-west-2

7. Use the env command filtered with grep to verify that you set the credentials as expected.

   If the credentials are set as expected, the CLI lists all the variables.

Verifying Example Credentials

env | grep AWS
AWS_ACCESS_KEY_ID=ASIA52AWRNEXAMPLE
AWS_SECRET_ACCESS_KEY=R8GTtGx7WwzQ9L1WQbPnHLEXAMPLE
AWS_SESSION_TOKEN=R8GTtGx7WwzQ9L1EXAMPLE_SESSION_TOKEN_FROM_EBC
AWS_DEFAULT_REGION=us-west-2

8. Copy the aws command in ElemBio Cloud and run it in the CLI to download the dataset.

   The copied command downloads the files to the current directory in the CLI. To change the download location, replace the . in the command with the path to the preferred folder.

Example AWS CLI Command

aws s3 cp --recursive s3://elembio-quality-reads-inc-usw2-7b02-d-runs/runs/{Run Name}/ .

If you encounter an error when using the AWS CLI command after setting local variables, use the env command to ensure the environment variables are set as expected. For more information on environment variables, see Environment variables to configure the AWS CLI.

AWS CLI with AWS Configure

The AWS CLI lets you set the credential variables using the aws configure command. By default, aws configure only sets two variables. You must manually add AWS_SESSION_TOKEN after setting the other two variables.

1. Select Download All.
2. Wait for the interface to prepare the download, and then select Continue to Download.
3. Select Download with a CLI.
4. Review the number of files and total download size to make sure all necessary files are selected.
5. Open a terminal window to access the AWS CLI.
6. Use the aws configure command to set the access key and secret access key provided by ElemBio Cloud.

   The AWS CLI updates the credentials and config file when you run the aws configure command.

Setting Example Credentials for the AWS CLI

aws configure
AWS Access Key ID [None]: ASIA52AWRNEXAMPLE
AWS Secret Access Key [None]:  R8GTtGx7WwzQ9L1WQbPnHLEXAMPLE
Default region name [None]: us-west-2-example
Default output format [None]: json

7. Use the cat command to check the credentials file to verify that you set the access keys as expected.

   The credentials file is located at ~/.aws/credentials on Linux or macOS and C:\Users\{USER_NAME}\.aws\credentials on Windows.

Verifying Example Credentials in Linux OS or macOS

cat ~/.aws/credentials
[default]
aws_access_key_id = ASIA52AWRNEXAMPLE
aws_secret_access_key = R8GTtGx7WwzQ9L1WQbPnHLEXAMPLE

Verifying Example Credentials in Windows OS

cat C:\Users\{USER_NAME}\.aws\credentials
[default]
aws_access_key_id = ASIA52AWRNEXAMPLE
aws_secret_access_key = R8GTtGx7WwzQ9L1WQbPnHLEXAMPLE

8. Open the local AWS credential file and manually set the AWS_SESSION_TOKEN credential.
   • Make sure to use the lowercase aws_session_token format.
   • For LinuxOS or macOS, you can use the sudo nano command to access and manually edit the file.

Accessing and Editing a Credentials File in Linux OS or macOS

sudo nano ~/.aws/credentials

9. Run the cat command again to verify that you set the AWS_SESSION_TOKEN as expected in the AWS credential file.

Example AWS_SESSION_TOKEN in AWS Credential File

 [default]
 aws_access_key_id = ASIA52AWRNEXAMPLE
 aws_secret_access_key = R8GTtGx7WwzQ9L1WQbPnHLEXAMPLE
 aws_session_token = R8GTtGx7WwzQ9L1EXAMPLE_SESSION_TOKEN_FROM_EBC

10. Copy the aws command in ElemBio Cloud and run it in the CLI to download the dataset.

Example AWS CLI Command

aws s3 cp --recursive s3://elembio-quality-reads-inc-usw2-7b02-d-runs/runs/{Run Name}/ .