scratch
§ Experimental Evaluation Setup I'm Happy With
created 2025-01-29 · last edited 2025-06-10
- Use python as a lightweight scripting language, with
asyncio for spawning many processes. - Use
logging to log info, and use python's inbuilt logger to write to a log file as well as stdout/stderr. - Write all data into sqlite database files, and use sqlite database file to be crash-tolerant, to keep track of how many tasks have finished.
- View sqlite database quickly using
visidata in the command line, and load into further processing scripts via duckdb. - To ensuire that scripting does not go stale, have two modes of the script: One that can perform a 'test run' of 10-20 rows that produces the data, and hook the 'test run' mode into CI.
- Write a
makefile and a README.txt that explains how to run the script.
❦