Data Browsing

ElemBio Cloud offers multiple options for accessing and browsing your data. The Data Explorer displays the contents of cloud storage buckets that have been enabled though compatible storage connections, providing full visibility to your files and folder structures. Tabs on the Run Details and Executions pages provide data browsers that are limited to individual runs and analysis output.

• Browse and navigate sequencing run output files, cytoprofiling run output files or the output files from flow executions.
• Download file(s) directly to your local environment.
• View file details, such as file size, archive status, and timestamp of the last update.
• Search and filter files to navigate with efficiency.

Compatibility

Data browsing is compatible with ElemBio Catalyst and AWS storage connections. If browsing an AWS bucket and files are not visible, ensure your IAM policy enables S3:ListBucket and S3:GetObject permissions.

File Availability

The availability of a file depends on the file retention status, the cloud bucket settings, and the type of associated run activity.

• Sequencing and cytoprofiling files populate throughout a run and are in sync with the run status on the instrument.
• FASTQ files and other analysis output files become available when a flow execution reaches a final status.
• Available files depend on the object storage class for files in the cloud bucket. For details on file availability with ElemBio Catalyst, see Data Storage Retention.
• Archived files are visible but unavailable for download. To make archived files available again in ElemBio Catalyst, contact Element Technical Support.

Downloading Files

Using the data browsing features, you can download a single file, multiple files, or all the files in a folder.

• To download a single file, select the Download icon in the Actions column.
• To download a multiple selected files, use the interface to select files and generate a dynamic download script.
• To download all files in a folder, use the AWS CLI or generate a download script.

For information on output files, see Sequencing Run Output Files, Cytoprfoling Run Output Files, Bases2Fastq Documentation, and the Cells2Stats Documentation.

Note:

The Copy URI icon in the Actions column lets you copy the Uniform Resource Identifier (URI) for a single file. You can use the URIs to individually download files using the AWS CLI.

Download with Script

ElemBio Cloud lets you download multiple selected files or all files at a prefix using a download script. The script uses the curl protocol and relies on presigned URLs that are dynamically generated.

The script contains a metadata header with execution instructions and presigned URLs for the selected files. Executing the script creates a directory on your system and downloads files into it.

Before executing the script, make sure to meet the following requirements:

• Install or update curl v8.4 or later on your system. To check the version, use the command curl -V.
• If you are using Windows OS, use the curl.exe command, and execute the command in Windows Command Prompt or PowerShell.
• Execute before the presigned URLs expire. The URLs expire 7 days after creation, and the time of expiration appears in the script header. If you do not complete the download within 7 days, you must generate a new script to download the files.

Download Multiple Selected Files

1. Select the checkboxes for the files and folders you want to download.
2. Select Download.
3. Review the number of files and total download size to make sure you selected the correct files.
4. Select Download curl config file.
5. Open a terminal window for the CLI on your OS and navigate to the location of the curl config file.
6. In ElemBio Cloud, select the tab for your OS: MacOS/Linux or Windows.
7. Copy the curl command and run it in the CLI.

Download All Files with the Download Script

1. Select Download All.
2. Wait for the interface to prepare the download, and then select Continue to Download.
3. Select Download with a script.
4. Review the number of files and total download size to make sure all necessary files are selected.
5. Select Download curl config file.
6. Open a terminal window for the CLI on your OS and navigate to the location of the curl config file.
7. In ElemBio Cloud, select the tab for your OS: MacOS/Linux or Windows.
8. Copy the curl command and run it in the CLI.

Download with AWS CLI

ElemBio Cloud offers an option to use AWS CLI command to download all files at a prefix. The command uses 36-hour temporary credentials that the user interface provides. To download data with AWS CLI, you must set three credential variables:AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN. Configure your download environment using one of the following methods.

AWS CLI with Local ENV Variables

When you set the credential variables in the local terminal environment, AWS CLI automatically detects them as known environment variables. Linux OS uses the export command, while Windows OS uses the set command to set the variables.

1. Select Download All.
2. Wait for the interface to prepare the download, and then select Continue to Download.
3. Select Download with a CLI.
4. Review the number of files and total download size to make sure all necessary files are selected.
5. Open a terminal window to access the AWS CLI.
6. Set the credentials provided by ElemBio Cloud. Use the commands for your OS.

Setting Example Credentials in Linux OS

export AWS_ACCESS_KEY_ID=ASIA52AWRNEXAMPLE
export AWS_SECRET_ACCESS_KEY=R8GTtGx7WwzQ9L1WQbPnHLEXAMPLE
export AWS_SESSION_TOKEN=R8GTtGx7WwzQ9L1EXAMPLE_SESSION_TOKEN_FROM_EBC
export AWS_DEFAULT_REGION=us-west-2

Setting Example Credentials in Windows OS

set AWS_ACCESS_KEY_ID=ASIA52AWRNEXAMPLE
set AWS_SECRET_ACCESS_KEY=R8GTtGx7WwzQ9L1WQbPnHLEXAMPLE
set AWS_SESSION_TOKEN=R8GTtGx7WwzQ9L1EXAMPLE_SESSION_TOKEN_FROM_EBC
set AWS_DEFAULT_REGION=us-west-2

7. Use the env command filtered with grep to verify that you set the credentials as expected.

   If the credentials are set as expected, the environment variables display.

Verifying Example Credentials

env | grep AWS
AWS_ACCESS_KEY_ID=ASIA52AWRNEXAMPLE
AWS_SECRET_ACCESS_KEY=R8GTtGx7WwzQ9L1WQbPnHLEXAMPLE
AWS_SESSION_TOKEN=R8GTtGx7WwzQ9L1EXAMPLE_SESSION_TOKEN_FROM_EBC
AWS_DEFAULT_REGION=us-west-2

8. Copy the aws command in ElemBio Cloud and run it in the CLI to download the dataset.

   The copied command downloads the files to the current directory in the CLI. To change the download location, replace the . in the command with the path to the preferred folder.

Example AWS CLI Command

aws s3 cp --recursive s3://elembio-quality-reads-inc-usw2-7b02-d-runs/runs/{Run Name}/ .

If you encounter an error when using the AWS CLI command after setting local variables, use the env command to ensure the environment variables are set as expected. For more information on environment variables, see Environment variables to configure the AWS CLI.

AWS CLI with AWS Profile (aws configure)

Set AWS credentials as a local aws profile using the aws configure command. By default, aws configure only sets two of the required variables so you must manually add AWS_SESSION_TOKEN after setting the other two variables.

1. Select Download All.
2. Wait for the interface to prepare the download, and then select Continue to Download.
3. Select Download with a CLI.
4. Review the number of files and total download size to make sure all necessary files are selected.
5. Use the aws configure command to set the access key and secret access key provided by ElemBio Cloud.

   The AWS CLI updates the credentials and config file when you run the aws configure command.

Setting Example Credentials for the AWS CLI

aws configure
AWS Access Key ID [None]: ASIA52AWRNEXAMPLE
AWS Secret Access Key [None]:  R8GTtGx7WwzQ9L1WQbPnHLEXAMPLE
Default region name [None]: us-west-2-example
Default output format [None]: json

6. Use the cat command to check the credentials file to verify that you set the access keys as expected.

   The credentials file is located at ~/.aws/credentials on Linux or macOS and C:\Users\{USER_NAME}\.aws\credentials on Windows.

Verifying Example Credentials in Linux OS or macOS

cat ~/.aws/credentials
[default]
aws_access_key_id = ASIA52AWRNEXAMPLE
aws_secret_access_key = R8GTtGx7WwzQ9L1WQbPnHLEXAMPLE

Verifying Example Credentials in Windows OS

cat C:\Users\{USER_NAME}\.aws\credentials
[default]
aws_access_key_id = ASIA52AWRNEXAMPLE
aws_secret_access_key = R8GTtGx7WwzQ9L1WQbPnHLEXAMPLE

7. Open the local AWS credential file and manually set the AWS_SESSION_TOKEN credential.
   • Make sure to use the lowercase aws_session_token format.
   • For LinuxOS or macOS, you can use the sudo nano command to access and manually edit the file.

Accessing and Editing a Credentials File in Linux OS or macOS

sudo nano ~/.aws/credentials

8. Run the cat command again to verify that you set the AWS_SESSION_TOKEN as expected in the AWS credential file.

Example AWS_SESSION_TOKEN in AWS Credential File

 [default]
 aws_access_key_id = ASIA52AWRNEXAMPLE
 aws_secret_access_key = R8GTtGx7WwzQ9L1WQbPnHLEXAMPLE
 aws_session_token = R8GTtGx7WwzQ9L1EXAMPLE_SESSION_TOKEN_FROM_EBC

9. Copy the aws command in ElemBio Cloud and run it in the CLI to download the dataset.

Example AWS CLI Command

aws s3 cp --recursive s3://elembio-quality-reads-inc-usw2-7b02-d-runs/runs/{Run Name}/ .