A quick script to ZSTD all your shredded tables

Aug 25, 2017. | By: Simon Rumble

Mike’s recent post about compressing Snowplow tables works great for atomic.events, with clients seeing compression down to 30% of the original size or so. But what about all your shredded tables?

For now you have to manually convert the output from igluctl while we wait for our pull request to make it into a release, from then on this will be automatic. There’s also a pull request in to change the default compression for atomic.events too.

Run the following code from the command line from the root of your schema definitions and it’ll automatically convert everything relevant to ZSTD. This was written by one of our staff.

sed -s -r -i -e '/root_(id|tstamp)/s/ENCODE (BYTEDICT|DELTA|DELTA32K|LZO|MOSTLY8|MOSTLY16|MOSTLY32|RUNLENGTH|TEXT255|TEXT32K|ZSTD)/ENCODE RAW/' -e '/root_(id|tstamp)/!s/ENCODE (BYTEDICT|DELTA|DELTA32K|LZO|MOSTLY8|MOSTLY16|MOSTLY32|RAW|RUNLENGTH|TEXT255|TEXT32K)/ENCODE ZSTD/' -- sql/com.yourcompany/*.sql

About

We exist to make organisations better understand their businesses by enabling all decision makers in a company to work with the same version of the truth.

Social Links

Our Bunker

Level 5, 104 Commonwealth Street
Surry Hills, 2010, Surry Hills,
NSW, Australia
contact@snowflake-analytics.com
AU: 1300 971 915
US: 1 415 963 4782
Privacy