1.6TB Data - Sharp Decline in file copying speeds between USB drives

[m654321]

Why do file copying speeds decline significantly over time, when transferring large amounts of data, following an initial burst?

Let me explain my recent experience.
I have 1.6TB of video recordings (i.e. 3000+ files classified under different subject folders) on a 4TB USB Seagate drive (ntfs formatted).

I'm now backing these up onto a newly formatted 4TB USB Samsung drive (ext4 formatted). File copying speeds, as indicated by the File Operations notification, were initially 112MB/s (4 hours shown to complete). Three hours later, this had slowed to 35MB/s (9 hours to complete); 6 hours later to 31 MB/s, etc...  I should add that I'm using the GUI method of copying/pasting files and not any command-line method.

Is there any action one can take to maintain the initially high file copying speeds when copying large amount of data? Or, is the only way to maintain high speeds to laboriously copy smaller chunks of data at a time, rather than copying the whole lot (1.6TB) in one go as I have done.  Would a command-line method be a more speed-efficient way to go?

Many thanks for your advice on this one
Mike

PS. Apologies if this post has come in bits, rather than all at once - I had problems posting it due to the 403 error forbidden message - I discovered the reason appeared to be the copying and pasting my post from LibreOffice Writer.  The forbidden message didn't pop-up if I typed the post directly into the forum.

[gold_finger]

Don't know answer to your problem.  Did some searching -- saw several posts about same issue, most with no definitive answers.  Found one post that might be of help though.  Have a look at recommendation to add grub parameter for USBs here:  https://forums.linuxmint.com/viewtopic.p...6#p1291555

[ralphy]

I think you're seeing the effects of drive head seek latency when copying. Also, your drive will likely use its cache for smaller amount of data until it is finally written to the drive. This gives the impression that transfers are faster at the beginning as you have pointe out. As the cache fills up and the head seek latency increases depending on the number of files, their sizes and the final destination on the drive, you'll certainly see exactly what you described.


Alternatively, you can try rsync to monitor your transfer speed. For example:

Code:

rsync -PSauv sourcedirectory/* /dev/sdXX/destdirectory/

If you actually see a drop in transfer speed, as opposed to files that are just big in size, then your disk is I/O-bound hence more noticeable slow downs (often seen when writing heavily to a flash drive or an external USB device).

[m654321]

Code:

rsync -PSauv sourcedirectory/* /dev/sdXX/destdirectory/

Ralphy, many thanks for the explanation & code for monitory source & destination drive speeds.
Using the command line you've given me above, how would I substitute the following information...

Source directory: 'home/mike/Downloads' on /dev/sda
Destination directory: 'TVprogrammes1' on /dev/sdb2 

I notice 'rsync' is already installed by default in the LL3.4 package - can it also be used to transfer data.  If so, is it superior to the GUI copy/paste method? I suppose the problem with the cache bottle-necking is the same whether a GUI or command-line method is used to copy large amounts of data ...

Mike

[m654321]

I tried this ...

Code:

mike@D630:~$ rsync -PSauv Downloads/* /dev/sdb2/TV-30-05-17/
sending incremental file list
rsync: ERROR: cannot stat destination "/dev/sdb2/TV-30-05-17/": Not a directory (20)
rsync error: errors selecting input/output files, dirs (code 3) at main.c(652) [Receiver=3.1.1]
mike@D630:~$

[member=6629]ralphy[/member]
What I'm trying to do with the above is to get the stats for copying video files from home/mike/Downloads on my laptop (located on partition dev/sda6) to the directory named 'TV-30-05-17' on an external USB drive (located at dev/sdb2/). As you can see it didn't work ... thanks again ralphy for any advice.

[gold_finger]

/dev/sdb2 is partition designation for the USB drive partition, but is not the mount point for it in the file system; therefore won't work with command.


Plug-in USB drive and mount it again.  If your system auto-mounts and opens USB sticks/external drives when you plug them in, (which is default behavior in LL unless you changed it), you don't need to do anything more to mount it.  Then open a terminal and run this command to see what the "MOUNTPOINT" is for that 2nd partition on your USB drive:

Code:

lsblk

Mount point will likely be something like /media/mike/usb-drive-partition.  Use the mount point shown by lsblk when running the rsync command.


Example:

Code:

rsync -PSauv ~/Downloads/* /media/mike/usb-drive-partition/TV-30-05-17

NOTE:  If you have other things in addition to video files in your Downloads folder those will also get copied to the TV-30-05-17 directory with this command.

[m654321]

I’ve made an interesting discovery regarding the significant decline in copying/pasting speed, when transferring large amounts of data - the copying speed issue now appears to be solved...

As mentioned in the first post of this thread, I was copying 1.6TB of video files (3000+ files @ average size ~ 500MB) from a 4TB USB Seagate drive (ntfs format) to a 4TB USB Samsung drive (ext4 format). Speeds plummeted from an initial high of around 120-130 MB/s to less than 30 MB/s, resulting in a very significant increase in the time for data transfer to reach completion.

Investigation 1.
First, I wanted to check whether copying from files from a drive with an ntfs format to one with an ext4 format was the culprit.  I had a second 4TB USB Seagate drive (also ntfs formatted). I found no slowdown in speed when copying between USB nfts drives (e.g. speeds remained at 110 to 130 MB/s) throughout the 4 hours it took to copy the 1.6TB.  However, following this observation, I was left unsure whether this was a disk formatting issue or a difference in drive make/model influencing the outcome, i.e. behaviour of Seagate vs. Samsung – so, roll on to Investigation 2 below...

Investigation 2.
I then changed the format of the Samsung drive, from ext4 to nfts. If the copy/paste slowdown I observed initially is due to a difference in drive format between ‘donor’ and ‘receiver’ drive, then there should be no slowdown when copying from Seagate (ntfs) to Samsung (ntfs) – and that’s exactly what I subsequently observed.  Speeds remained high (110 to 130 MB/s) taking around 4 hours to transfer 1.6TB of data, as I observed with the previous Seagate (ntfs) to Seagate (ntfs) copying/pasting.

Conclusion.
The observations above indicate that the speed of copying/pasting a large amount of data, directly from one USB drive to another, is influenced significantly by the drive format.  To maintain high copying/pasting speeds throughout, both USB drives should have the same format, i.e. ntfs in the above case (or ext4 drives, observation of the 6 July 2017).  Conversely, a very significant reduction in data transfer speed will occur if the drives have different formats, i.e. in the above case with a ‘donor’ drive = ntfs, and  ‘recipient’ drive =  ext4.

In view of the above discovery, I’ll therefore now mark this thread as solved 

Regards
Mike

[m654321]

[size=1.35em]See last post - update added on 6 July 2017 (in blue font) under 'Conclusion'[/size]