Running fast data processing scripts locally on a laptop, Querying and introspecting large CSV/Parquet files (including in S3), Figuring out the shape of large structured data via CLI probing queries
It's an embeddable SQL database that excels at fast, local analysis of large data files without needing a separate database server.
Reviews from businesses like yours — matched by size and type. Available once you create a profile.
Complete profileRelated Tools
Community Reviews
Reviews from Reddit, Product Hunt, Hacker News & AlignAI members
DuckDB + Parquet is a way nicer angle than what I had in mind.
Hacker News • HackerNews • Use case: Storing and querying performance trace data as a nicer alternative to parsing xctrace output
performant database that lets you directly query JSON, parquet, csv files with SQL queries and convert one to the other.
Hacker News • HackerNews • Use case: performant database for querying JSON, parquet, csv files with SQL and converting between them
DuckDB's out-of-core execution also means it can handle datasets larger than available RAM natively, which is one of the reasons we picked it over SQLite. For really large exports... DuckDB eats it for breakfast.
Hacker News • HackerNews • Use case: Local data storage and querying for a CRM, handling large exports (e.g., 100k+ contacts)
persist to DuckDB for querying across test history
Hacker News • HackerNews • Use case: Persisting data for querying across test history
Our tech stack is comprised of kdb, JavaScript, React, Python and AWS.
Hacker News • HackerNews • Use case: Part of their core tech stack for financial data and analytics.
Our tech stack is comprised of q kdb+, JavaScript, React, Python and various AWS products.
Hacker News • HackerNews • Use case: Part of their core tech stack for financial data processing.
Our tech stack is comprised of q kdb+, JavaScript, React, Python and various AWS products.
Hacker News • HackerNews • Use case: Part of their core tech stack for financial data processing.
Our tech stack is comprised of q kdb+, JavaScript, React, Python and various AWS products.
Hacker News • HackerNews • Use case: Part of their core tech stack for financial data processing.
It figures out the shape of the data on its own within seconds by writing probing queries in DuckDB and jq.
Hacker News • HackerNews • Use case: Working with large structured data (CSV, Parquet) via its CLI to write probing queries and figure out the shape of data.
DuckDB works way better than any dedicated CSV tool because it packs a full database engine that can return aggregates, explore the limits of your data... For Parquet, I just point DuckDB to the 100s of GBs of Parquet files in S3... and it's blazing fast at introspecting that data.
Hacker News • HackerNews • Use case: As an engine to query and introspect large CSV and Parquet files, including data in S3, and for debugging ML model artifacts.
Shitty python script with DuckDB running locally on laptop, can get you long long way.
Hacker News • HackerNews • Use case: Running a data processing script locally on a laptop
Have You Used This Tool?
Sign in to share your experience with this tool.
Sign in