Compressed files

lukas_demetz

Hi there,

to overcome the bad connection in Namibia, we began compressing the raw data before uploading it.

This means, however, that you will need 7Zip for the foreseable future to work with the data.

Please download it here: https://www.7-zip.org/download.html

The archives are split in 2MB big chunks, to decompress you will have to download them all and extract them. I will work on a way to automate this, but it will take a little while to do it safely.

lukas_demetz

For the tech savvy, here is what happens in the background for uploading the files:

Every single frame is compressed using 7-Zip, and split into 2 MB big chunks
The uploader is transferring the data to an S3 (AWS) space in Cape Town. Why? The routing currently slows down the transfer speed considerably if going straight to London or Frankfurt...
Another VPS instance is copying the data from Cape Town to Frankfurt

Sounds complicated? 😅 Well, it is. But still it delivers a 10x speed increase over going straight to Frankfurt. And another ₁.8x increase due to compression. At the end of the day we are now able to get the iDK data uploaded within the day. For the Veloce, things are a bit slower as the frames are about 4x as big 😆

fernandoyang

Hi Lukas，

I've tried decompressing the data but I'd no luck. Using Mac with KEKA and Windows 10 with latest version of 7zip, I encountered error of 'Unexpected end of archive.'

Fernando

lukas_demetz

Hi Fernando,
the most classic reason for this would be on eor more missing 'splits' of the archive. I am currently investigating the issue, again it's a result of telecom namibia being unable to fix the issue since now 5 weeks 😢

lukas_demetz

Update: i forgot to remove a size filter, resulting in archives of less than 1MB to not being uploaded. Guess what, those were the ones missing to complete several archives. They are now being uploaded and then the archives should decompress just fine.

fernandoyang

lukas_demetz Problem solved! Thanks!

lukas_demetz

fernandoyang Glad to hear that 😀
Also, the uploads from the Veloce are now coming, too. Have been able to upload 4.5GB in about 11 hours, so not too bad

lukas_demetz

Small tip:
To quickly extract all files in one folder use this:
7z e *.001

(Of course you will need 7Zip to be in your path/installed on your machine)