insert_to_clickhouse.py File Reference

Inserts filtered benchmarking logs into a ClickHouse database. More...

Go to the source code of this file.

Namespaces
namespace	pipeline
namespace	pipeline.insert_to_clickhouse

Functions
None	pipeline.insert_to_clickhouse.insert_batch (str batch_id)
	Filters and inserts a batch of records into ClickHouse.
	pipeline.insert_to_clickhouse.main ()
	CLI entrypoint for inserting a batch into ClickHouse.

Detailed Description

Inserts filtered benchmarking logs into a ClickHouse database.

Description

This script reads a Parquet dataset, filters it by BatchID, and inserts the matching records into the benchmark.performance table in ClickHouse. It uses the clickhouse-driver to perform inserts and ensures that the table schema matches the format defined in pipeline.schema.

Usage

$ python3 insert_to_clickhouse.py –batchid <BATCH_ID>

Example

$ python3 insert_to_clickhouse.py –batchid "batch_202405"

Notes

• ClickHouse connection parameters are loaded from .envvia scripts/config.py
• The Parquet file path is set in DB_PATH
• You may call insert_batch(batch_id) directly from other scripts or notebooks

Definition in file insert_to_clickhouse.py.