Skip to main content

Access Data

The Source Data Proxy provides S3-compatible access to all data hosted on Source Cooperative. You can access data through the proxy without authentication, making it easy to programmatically download datasets using standard AWS CLI commands.

Using the AWS CLI​

All Source datasets are accessible using the AWS CLI with these parameters:

  • Endpoint URL: https://data.source.coop
  • No authentication required: Add --no-sign-request to all commands

Example: Fields of The World Dataset​

The Fields of The World (FTW) dataset is fully open and available at https://source.coop/kerner-lab/fields-of-the-world

You can find and download all of the files there individually, or you can use the AWS CLI to access the data using the Source Data Proxy by adding --endpoint-url https://data.source.coop --no-sign-request to your standard AWS commands.

Listing Dataset Contents​

To list the contents at the root level of a dataset:

aws s3 ls s3://kerner-lab/fields-of-the-world --endpoint-url https://data.source.coop --no-sign-request

To recursively list all contents in a specific directory within the dataset (e.g., the vietnam directory):

aws s3 ls s3://kerner-lab/fields-of-the-world/vietnam --endpoint-url https://data.source.coop --no-sign-request --recursive

Add --summarize to any ls command to see a count and total byte volume of the listed objects. For example:

aws s3 ls s3://kerner-lab/fields-of-the-world/vietnam --endpoint-url https://data.source.coop --no-sign-request --recursive --summarize

Downloading Datasets​

Use the cp command to copy data to local storage:

aws s3 cp s3://kerner-lab/fields-of-the-world/README.md . --endpoint-url https://data.source.coop --no-sign-request

Authenticated Access with the Source CLI​

Public datasets work with --no-sign-request. To access data that requires authentication, use the Source CLI, which logs you in and supplies temporary AWS credentials through an AWS profile:

source-coop login
aws s3 ls s3://your-org/product-id --profile source-coop

See Upload with the Source CLI for the one-time install and profile setup.

For software that runs on its own, with nobody there to log in, use a service account instead.

Getting Started with AWS CLI​

If you don't have the AWS CLI installed, follow the AWS CLI Getting Started Guide.

Current Status​

The Source Data Proxy is currently in beta. We're working to make performance indistinguishable from accessing objects directly via AWS S3.