Single Zip on Full Backup, and Dropbox Speed Issues

[DavidParticipant]

1. Is there a way to bundle all the zip files to a single zip when pushed to remote storage? Or to possibly create a date based folder with all updated zips in that folder.

I ask if this is possible as I use DropBox for my remote storage with multiple sites. I have told UD to create a folder for each site for basic managment. The issue is that I make 2 DB a day, and 2 a week of backups. It would be beautiful if I could name the backup something like DB-timestamp and FULL-timestamp … this is possible in a number of other plugins I have used. I realize I can give a manual backup an optional name, but that will not work for scheduled backups.

So, how to or feature:
Single Zip for backups for management
Naming of Scheduled Backups

2. Really slow chunk uploading to Dropbox.
I am on a dedicated Dual Xeon 5639 machine with SSD Drives, and 48GB of Ram and a 1GB port, yet to upload a 400Mb file to dropbox takes about 10-15min with UD. The same file with BackupBuddy takes 2-3min and BackWPUP about 3-5min. The only difference that I can tell is the chunking method used with UD.

Here is my system details:
Web server: nginx front end proxy, bacend Apache/2.4.18 (Unix) OpenSSL/1.0.1f (Linux s85661 64-bit #123-Ubuntu SMP Fri Feb 19 14:27:58 UTC 2016)
Peak memory usage: 14.75 MB
Current memory usage: 12.75 MB
Memory limit: 256M
PHP version: 5.5.34 – show PHP information (phpinfo)
MySQL version: 5.5.47
Curl version: 7.35.0 / SSL: OpenSSL/1.0.1f / libz: 1.2.8
ZipArchive::addFile: Yes
zip executable found: Yes: /usr/bin/zip

[udadminKeymaster]

Hi David,

1. No, there isn’t a way to create a single zip file. UD splits the backup uploads/plugins/themes to make it more efficient to restore just one of these… on many sites, the media (i.e. uploads) can be beyond 95% of the data… downloading/unzipping all of that, just to recover from a dodgy plugin/theme update, is very awkward.
However – you don’t normally have to know how many zips there are, or look at them. Can you explain more about why it’s an issue? There may be another solution.

2. Please can you post links to the backup log files from each of these? Then I can compare. I suspect that UD may be using a smaller chunk size for its uploading, resulting in more round trips, and we could probably improve the efficiency.

David

[udadminKeymaster]

Hi,

Actually – regarding 2, this is now implemented for our next release. Please see: https://updraftplus.com/faster-dropbox-uploads/

David

[DavidParticipant]

ok, so regarding file names..

1. customer preference
2. easier organization of archival based files (say you do a DB 2X a day and keep 120 copies), much easier to find the file you are looking for.

If I could just assign a custom name to the scheduled backups that would worlds better. Currently every file is named “backup+date+time+type” so when you are looking through a directory with 120 DB and hundreds of split zips this is annoying to try and sort and locate the one I want.

I understand the idea behind the multiple zips, and that is fine also. If there was just a way to create a full zip that contained the multiple zips that would be great. This would allow a single item for all zips to be located in my file system and then I can extract that and upload for restore purposes what I need.

Lastly, this many files can have a negative impact on dropbox performance. Most operating systems have a max folder/file watch that they can do. So the more smaller files you load into a system the closer to this limit you get. For example on linux this is 10K files by default. So if you have more then 10K files in your dropbox folder it causes sync issues. There are FAQs on how to increase this, but the average consumer will not figure out why their dropbox is syncing slowly, or not at all due to pushing large amounts of files. Also, consider OSX HFS issue where it can only read and write one file at a time. This means that it has to queue, pause, download, pause, buffer to ram, pause, queue next file, pause, back to ram, pause, write to HD, pause, back to queue one file at a time. Linux and Windows is able to handle multiple threaded files at one time.

So, I guess in the end my first point of client preference is the biggest one. I live where bandwidth is not an issue and so dowloading huge files is fine. I understand the idea behind the split files, just really wish that there was an option to create a single zip if “I” like.

#2 Dropbox performance… good to hear that it will be addressed in the next release. I have my file chunk set to 10000Mb (10Gb) so I am sure it was not that. I noticed that because again you create multiple zips it has to chunk one zip send it verify, then restart the chunking and sending for each zip. This seems much less effecient then a single zip of all the other zips, and puts a longer load on my server as the lob is held open longer.

fyi… i get an error on that link. Tested from windows, osx, linux with chrome and firefox and seeing the same error. Using google dns.

The updraftplus.com page isn’t working

updraftplus.com didn’t send any data.
ERR_EMPTY_RESPONSE

[Author]

Posts