Sold by: Common Crawl
Open data
|
Deployed on AWS
A corpus of web crawl data composed of over 300 billion web pages.
Overview
A corpus of web crawl data composed of over 300 billion web pages.
Features and programs
Open Data Sponsorship Program
This dataset is part of the Open Data Sponsorship Program, an AWS program that covers the cost of storage for publicly available high-value cloud-optimized datasets.
Pricing
This is a publicly available data set. No subscription is required.
Legal
Content disclaimer
Vendors are responsible for their product descriptions and other product content. AWS does not warrant that vendors' product descriptions or other product content are accurate, complete, reliable, current, or error-free.
Delivery details
AWS Data Exchange (ADX)
AWS Data Exchange is a service that helps AWS easily share and manage data entitlements from other organizations at scale.
Open data resources
Available with or without an AWS account.
- How to use
- To access these resources, reference the Amazon Resource Name (ARN) using the AWS Command Line Interface (CLI). Learn more
- Description
- Crawl data (WARC and ARC format)
- Resource type
- S3 bucket
- Amazon Resource Name (ARN)
- arn:aws:s3:::commoncrawl
- AWS region
- us-east-1
- AWS CLI access (No AWS account required)
- aws s3 ls --no-sign-request s3://commoncrawl/
Resources
Vendor resources
Support
Managed By
How to cite
Common Crawl was accessed on DATE from https://registry.opendata.aws/commoncrawl .
License
This data is available for anyone to use under the Common Crawl Terms of Use
Similar products
Use Common Crawl (AWS Data Exchange for AWS Lake Formation Test Product) to understand how to interact with data made available via AWS Lake Formation.
Common Room helps teams capture and act on every buying signal in one place. It empowers go-to-market teams with AI-driven insights for effective outreach.
End-to-end implementation and migration of Qualtrics programs, including configuration, integrations, dashboards, localization, testing, and go-live support.
Ongoing Qualtrics delivery capacity and program and platform management on a retainer basis. Choose a Build Retainer for net-new development or Managed Services for ongoing management of live programs.