Duplicate files on FTP

adriancarr

Hi,

For some of my images I'm seeing duplicate fts files of the FTP, with the same timestamp and md5sum (but different file names). I've seen it for 2 RIDs on the 'Hakos Veloce 200 RH'. The download via the GUI also doesn't seem to work properly and only downloads a subset of the files.

I can filter out the duplicates, so it isn't a major issue for me, but it may be worth looking into the bug.

Adrian

MD5 (RAW-3667-0-NGC 1976-Ha7-1-5-120-B1-Ha7-20210720-042829-v2021-07-20-043442-000.fts) = 215c814227b2e6a409310034fd7247b4
MD5 (RAW-3667-0-NGC 1976-Ha7-1-5-120-B1-Ha7-20210720-042829-v2021-07-20-043622-000.fts) = 215c814227b2e6a409310034fd7247b4

lftp *USERNAME*@files.skygems-observatories.com:/Veloce Namibia/3667-0-NGC 1976/RAW> ls
-rw-r--r-- 1 root root    122351040 Jul 20 03:31 RAW-3667-0-NGC 1976-Green-1-5-120-B1-Green-20210720-032918.fts
-rw-r--r-- 1 root root    122351040 Jul 20 03:34 RAW-3667-0-NGC 1976-Green-1-5-120-B1-Green-20210720-033232.fts
-rw-r--r-- 1 root root    122351040 Jul 20 03:37 RAW-3667-0-NGC 1976-Green-1-5-120-B1-Green-20210720-033541.fts
-rw-r--r-- 1 root root    122351040 Jul 20 03:41 RAW-3667-0-NGC 1976-Green-1-5-120-B1-Green-20210720-033849.fts
-rw-r--r-- 1 root root    122351040 Jul 20 03:44 RAW-3667-0-NGC 1976-Green-1-5-120-B1-Green-20210720-034159.fts
-rw-r--r-- 1 root root    122351040 Jul 20 04:24 RAW-3667-0-NGC 1976-Ha7-1-5-120-B1-Ha7-20210720-042207.fts
-rw-r--r-- 1 root root    122351040 Jul 20 04:27 RAW-3667-0-NGC 1976-Ha7-1-5-120-B1-Ha7-20210720-042523.fts
-rw-r--r-- 1 root root    122351040 Jul 20 04:30 RAW-3667-0-NGC 1976-Ha7-1-5-120-B1-Ha7-20210720-042829-v2021-07-20-043442-000.fts
-rw-r--r-- 1 root root    122351040 Jul 20 04:30 RAW-3667-0-NGC 1976-Ha7-1-5-120-B1-Ha7-20210720-042829-v2021-07-20-043622-000.fts
-rw-r--r-- 1 root root    122351040 Jul 20 04:33 RAW-3667-0-NGC 1976-Ha7-1-5-120-B1-Ha7-20210720-043139-v2021-07-20-043443-000.fts
-rw-r--r-- 1 root root    122351040 Jul 20 04:33 RAW-3667-0-NGC 1976-Ha7-1-5-120-B1-Ha7-20210720-043139-v2021-07-20-043622-000.fts
-rw-r--r-- 1 root root    122351040 Jul 20 04:37 RAW-3667-0-NGC 1976-Ha7-1-5-120-B1-Ha7-20210720-043445-v2021-07-20-044012-000.fts
-rw-r--r-- 1 root root    122351040 Jul 20 04:37 RAW-3667-0-NGC 1976-Ha7-1-5-120-B1-Ha7-20210720-043445-v2021-07-20-044103-000.fts
-rw-r--r-- 1 root root    122351040 Jul 20 04:08 RAW-3667-0-NGC 1976-Luminance-1-5-120-B1-Luminance-20210720-040555-v2021-07-20-040919-000.fts
-rw-r--r-- 1 root root    122351040 Jul 20 04:08 RAW-3667-0-NGC 1976-Luminance-1-5-120-B1-Luminance-20210720-040555-v2021-07-20-041101-000.fts
-rw-r--r-- 1 root root    122351040 Jul 20 04:11 RAW-3667-0-NGC 1976-Luminance-1-5-120-B1-Luminance-20210720-040905-v2021-07-20-041406-000.fts
-rw-r--r-- 1 root root    122351040 Jul 20 04:11 RAW-3667-0-NGC 1976-Luminance-1-5-120-B1-Luminance-20210720-040905-v2021-07-20-041535-000.fts
-rw-r--r-- 1 root root    122351040 Jul 20 04:14 RAW-3667-0-NGC 1976-Luminance-1-5-120-B1-Luminance-20210720-041214.fts
-rw-r--r-- 1 root root    122351040 Jul 20 04:17 RAW-3667-0-NGC 1976-Luminance-1-5-120-B1-Luminance-20210720-041526-v2021-07-20-041924-000.fts
-rw-r--r-- 1 root root    122351040 Jul 20 04:17 RAW-3667-0-NGC 1976-Luminance-1-5-120-B1-Luminance-20210720-041526-v2021-07-20-042051-000.fts
-rw-r--r-- 1 root root    122351040 Jul 20 04:20 RAW-3667-0-NGC 1976-Luminance-1-5-120-B1-Luminance-20210720-041837-v2021-07-20-042411-000.fts
-rw-r--r-- 1 root root    122351040 Jul 20 04:20 RAW-3667-0-NGC 1976-Luminance-1-5-120-B1-Luminance-20210720-041837-v2021-07-20-042500-000.fts
-rw-r--r-- 1 root root    122351040 Jul 20 03:10 RAW-3667-0-NGC 1976-O3-1-5-120-B1-O3-20210720-030759-v2021-07-20-031107-000.fts
-rw-r--r-- 1 root root    122351040 Jul 20 03:10 RAW-3667-0-NGC 1976-O3-1-5-120-B1-O3-20210720-030759-v2021-07-20-031135-000.fts
-rw-r--r-- 1 root root    122351040 Jul 20 03:14 RAW-3667-0-NGC 1976-O3-1-5-120-B1-O3-20210720-031213.fts
-rw-r--r-- 1 root root    122351040 Jul 20 03:18 RAW-3667-0-NGC 1976-O3-1-5-120-B1-O3-20210720-031623.fts
-rw-r--r-- 1 root root    122351040 Jul 20 03:22 RAW-3667-0-NGC 1976-O3-1-5-120-B1-O3-20210720-032030-v2021-07-20-032346-000.fts
-rw-r--r-- 1 root root    122351040 Jul 20 03:22 RAW-3667-0-NGC 1976-O3-1-5-120-B1-O3-20210720-032030-v2021-07-20-032549-000.fts
-rw-r--r-- 1 root root    122351040 Jul 20 03:26 RAW-3667-0-NGC 1976-O3-1-5-120-B1-O3-20210720-032435-v2021-07-20-032952-000.fts
-rw-r--r-- 1 root root    122351040 Jul 20 03:26 RAW-3667-0-NGC 1976-O3-1-5-120-B1-O3-20210720-032435-v2021-07-20-033001-000.fts
-rw-r--r-- 1 root root    122351040 Jul 20 03:52 RAW-3667-0-NGC 1976-Red-1-5-120-B1-Red-20210720-034958-v2021-07-20-035310-000.fts
-rw-r--r-- 1 root root    122351040 Jul 20 03:52 RAW-3667-0-NGC 1976-Red-1-5-120-B1-Red-20210720-034958-v2021-07-20-035513-000.fts
-rw-r--r-- 1 root root    122351040 Jul 20 03:55 RAW-3667-0-NGC 1976-Red-1-5-120-B1-Red-20210720-035307-v2021-07-20-035815-000.fts
-rw-r--r-- 1 root root    122351040 Jul 20 03:55 RAW-3667-0-NGC 1976-Red-1-5-120-B1-Red-20210720-035307-v2021-07-20-035955-000.fts
-rw-r--r-- 1 root root    122351040 Jul 20 03:58 RAW-3667-0-NGC 1976-Red-1-5-120-B1-Red-20210720-035617.fts
-rw-r--r-- 1 root root    122351040 Jul 20 04:01 RAW-3667-0-NGC 1976-Red-1-5-120-B1-Red-20210720-035925-v2021-07-20-040408-000.fts
-rw-r--r-- 1 root root    122351040 Jul 20 04:01 RAW-3667-0-NGC 1976-Red-1-5-120-B1-Red-20210720-035925-v2021-07-20-040617-000.fts
-rw-r--r-- 1 root root    122351040 Jul 20 04:04 RAW-3667-0-NGC 1976-Red-1-5-120-B1-Red-20210720-040233.fts

lukas_demetz

Hi Adrian,

this is a known bug and is caused by bad internet connection. It happens mostly with the data coming from Namibia.
What you are seeing are the "versions" of one file which has been trasferred twice for redundancy. A "feature" in rclone (which we are using for file transfer) creates versions on-the-fly when an upload has to be restarted due to a loss of connection.

As for the download feature from the Dashboard, it is only meant to be a tool for those who don't have an FTP client. I will look at it, there is for sure some room for improvement 🙂

adriancarr

Hi Lukas,

Thanks for the clarification.

I'm running the 'rclone dedupe' command after copying the files to my computer. Would you be able to run on the file server once the transfer is complete?

rclone dedupe --dedupe-mode first --by-hash PATH

Adrian

lukas_demetz

@adriancarr Oh interesting, let me test this. Sounds like a good way of dealing with it 🙂