- https://tech.lgbt/@risottobias
-
privacy, security, design
- Joined on
2024-12-29
if prefix includes the processing node (/cache_policy/retention_policy/collector_id/host_id/epoch_ms/chunk_ms.sqlite.gz.age) then two simultaneous collectors could upload. when new epoch is made,…
log collectors by default don't have the private keys they're encrypting to, so they can't by default read until a cache request condition comes in. that could be separated out by requiring a hot…
90 day upload API key anyway?, automatically rotate age keys (auto-rotate syslog keys too? - syslog addresses are just upload ones, but could have them expire after 2 years?), maybe 90 min, 2 year…
some log processors will only be uploaders, so could be writing to a public key they don't have a read on.
configurable: age key rotations and partitioning key phases: per-log-processor age key (in multiples of 7, so that's 13 age keys in play in 90 days) log processor does not keep around older age…
cache policy is a bounded-and
- one of {128mb, 2,000 events, 15 minutes}
- and one of {512mb, 10,000 events, 1 hour}
^ allow them to tune that.
- web dash host (manages API keys,…
forgot the max amount of data that can be stored in a cipher before you need a new key, but ideally the recipient encryption should be readable on multiple boxes (e.g. ingest-A's key, ingest-B's…
index format sqlite tables (for logs vs for user preferences/searches):
need to relate: ingest api key, ingest time, event time, host, log source, tags -> which chunk it's in -> raw log
folde…
sqlite index: on ingestion box, tag log batches, compress in chunks (e.g. 15 minute increments, 2,000 EPS, or 128mb, whichever is larger) - old project idea was parquet, drill, minio (no zookeeper…
it's not glamorous and scalable, it doesn't trust the S3 storage (age encrypted, minisign signed), sqlite indexing, $10/TB stored, search is throttled, ingest is throttled (.5 TB/month) but can be…