Upload Your Data

This guide explains how to deliver your data to Source Cooperative in a secure and simple way. It is written for data providers and does not require an Amazon Web Services (AWS) account or deep cloud storage knowledge.

If you do not see the option to upload (for example, Edit Mode or View Credentials on your product page under the lock icon), contact hello@source.coop to request upload access.

Choose how to upload

	Best for
In the browser	Small or one-time uploads, without installing anything	Easiest
With the Source CLI	Large uploads and scripts that you run yourself	Recommended for larger uploads
With a service account	Pipelines, scheduled jobs and GitHub Actions: anything that runs with nobody at the keyboard	Advanced

Already upload from your own AWS account with an IAM role? That method is deprecated in favor of service accounts, but its instructions are still at Direct uploads.

How uploads work: the Source data proxy

Uploads go through the Source data proxy at https://data.source.coop. The proxy speaks the S3 API, the object storage protocol that Amazon S3 introduced and that most storage tools now support. It is not itself an AWS service.

Behind the proxy, a product's data may be stored with any of several object storage providers: Amazon S3, Google Cloud Storage, Azure Blob Storage, Cloudflare R2, and others. You don't need to know which one. You always upload to the same proxy address, and the proxy writes to the right place.

That is why these guides use AWS tools:

• The AWS CLI and AWS SDKs (boto3, etc.) are used only as S3 clients. You point them at https://data.source.coop instead of AWS. Any S3-compatible client that lets you set a custom endpoint URL will work.
• Your upload credentials come from Source Cooperative, not AWS. They use the AWS credential format (access key ID, secret access key, session token) because S3 clients expect it, but they are only valid at the Source data proxy.
• Addresses are proxy addresses. s3://account-id/product-id/ means the product-id product in the account-id account on Source Cooperative. It is not an Amazon S3 bucket. Where your data is physically stored doesn't change it.

Where to upload your data

You may upload only to your product's prefix on the data proxy, for example:

s3://account-id/product-id/

This matches your product page URL, https://source.coop/account-id/product-id.

You may not upload outside this path.

Need help?

If you’re unsure which option is right for you, or need help setting up automated access, contact the Source Cooperative team at hello@source.coop, and we will help you choose the best approach for your use case.

Thank you for contributing data to Source Cooperative.