I am creating a couple of bigger database tables with at least hundreds of millions of observations, but growing. Some tables are by minute, some by milliseconds. timestamps are not necessarily unique.
Should I create separate year, month, or date and time columns? Is one unique datetime column enough? At what size would you partition the tables?
Raw data is in csv.
Currently I aim for postgres and duckdb. Does timescaledb make a significant difference?
Thanks! That helps a lot