It could not upload it all!

  • This topic has 14 replies, 2 voices, and was last updated 10 years ago by udadmin.
Viewing 15 posts - 1 through 15 (of 15 total)
  • Author
    Posts
  • #70585
    Raymond
    Participant

    Wow it was taking a long time and both cores on my server was going 100% for hours! It’s super slow to upload. I guess because it’s using apache.

    The backup has 6 big data .zip files. It uploaded not all the 1st one stop at about 1.5 GB. It’s should’ve been about 2 GB zip.

    On both my server I can use samba. As I typed this I drag and drop the 2 GB to the same place from updraft where it should go. It only took about 3 minutes to upload it like that!

    Can only guess it error uploading with UpdraftPlus because it took so long and the hard drive went to sleep. Not sure.

    Samba can upload way faster with out both cores of the CPU 100%

    Can see it only takes about 16 minutes to upload the rest of the big backup .zip files in that photo.

    Guess with UpdraftPlus it would of been more then 24 hours with both cores of the CPU going 100%

    Got a little LCD screen on my server and got this photo of htop when updraftplus was uploading.

    CPU not 100% in this photo but it was a lot.

    Not sure if there is a better faster way to upload with in WordPress. That don’t take a lot of CPU power and time.

    -Raymond Day

    #70622
    udadmin
    Keymaster

    Hi Ray,

    Please can you post the backup log file? (Find it on the “Existing Backups” tab – each backup has a log button). Use pastebin.com or something similar to post it, as it’ll be too long for this forum.

    I would imagine that UD hadn’t actually reached the uploading stage yet – because, uploading does not use any significant CPU resources. It’s the creating of the zips that uses CPU and disk I/O, as files are read, written and compressed (which is unavoidable with a zip-based backup – see: https://updraftplus.com/faqs/how-much-resources-does-updraftplus-need/ ).

    Best wishes,
    David

    #70625
    udadmin
    Keymaster

    P.S. I notice from the screenshot that you’ve put the ‘split’ size up to 2Gb. The link in my previous post explains why this will very significantly increase CPU and disk I/O usage.

    Best wishes,
    David

    #71438
    Raymond
    Participant

    Guess last night it was trying to back up. Looks like it did delete the older backup and when it started to upload the new big .zip file it just stop that’s the end line in the log file.

    Here is the end of the log file about 20 lines of the end.

    7980.751 (9) UpdraftPlus WordPress backup plugin (https://updraftplus.com): 1.9.45.19 WP: 4.1 PHP: 5.5.9-1ubuntu4.5 (Linux ubuntu-i3 3.13.0-39-generic #66-Ubuntu SMP Tue Oct 28 13:30:27 UTC 2014 x86_64) MySQL: Server: Apache/2.4.7 (Ubuntu) safe_mode: 0 max_execution_time: 900 memory_limit: 256M (used: 2.7M | 3M) multisite: N mcrypt: N LANG: C ZipArchive::addFile: Y
    7980.751 (9) Free space on disk containing Updraft’s temporary directory: 1282574.5 Mb
    7980.752 (9) Backup run: resumption=9, nonce=489757871857, begun at=1419177603 (7980s ago), job type=backup
    7980.752 (9) The current run is resumption number 9, and there was nothing useful done on the last run (last useful run: 0) – will not schedule a further attempt until we see something useful happening this time
    7980.753 (9) Creation of backups of directories: already finished
    7980.753 (9) Saving backup status to database (elements: 14)
    7980.754 (9) Database dump (WordPress DB): Creation was completed already
    7980.754 (9) Saving backup history
    7980.756 (9) backup_2014-12-21-1100_my_weblog_489757871857-plugins.zip: plugins: This file has already been successfully uploaded
    7980.756 (9) backup_2014-12-21-1100_my_weblog_489757871857-themes.zip: themes: This file has already been successfully uploaded
    7980.756 (9) backup_2014-12-21-1100_my_weblog_489757871857-uploads.zip: uploads: This file has not yet been successfully uploaded: will queue
    7980.757 (9) backup_2014-12-21-1100_my_weblog_489757871857-uploads2.zip: uploads: This file has not yet been successfully uploaded: will queue
    7980.757 (9) backup_2014-12-21-1100_my_weblog_489757871857-uploads3.zip: uploads: This file has not yet been successfully uploaded: will queue
    7980.757 (9) backup_2014-12-21-1100_my_weblog_489757871857-uploads4.zip: uploads: This file has not yet been successfully uploaded: will queue
    7980.757 (9) backup_2014-12-21-1100_my_weblog_489757871857-uploads5.zip: uploads: This file has not yet been successfully uploaded: will queue
    7980.758 (9) backup_2014-12-21-1100_my_weblog_489757871857-uploads6.zip: uploads: This file has not yet been successfully uploaded: will queue
    7980.758 (9) backup_2014-12-21-1100_my_weblog_489757871857-uploads7.zip: uploads: This file has not yet been successfully uploaded: will queue
    7980.758 (9) backup_2014-12-21-1100_my_weblog_489757871857-others.zip: others: This file has not yet been successfully uploaded: will queue
    7980.758 (9) backup_2014-12-21-1100_my_weblog_489757871857-db.gz: db: This file has not yet been successfully uploaded: will queue
    7980.759 (9) Requesting upload of the files that have not yet been successfully uploaded (9)
    7980.760 (9) Cloud backup selection: sftp
    7980.760 (9) Beginning dispatch of backup to remote (sftp)
    7982.054 (9) SFTP: Successfully logged in
    7982.054 (9) SCP/SFTP upload: attempt: backup_2014-12-21-1100_my_weblog_489757871857-uploads.zip

    Guess that my be the best part of the long need to see.

    The Existing Backups tab under the top Backup date says:

    Dec 21, 2014 11:00
    (Not finished)

    -Raymond Day

    #71548
    udadmin
    Keymaster

    Hi Ray,

    The SFTP file storage method doesn’t use chunked/resumable uploads (because these weren’t available in the PHP SFTP library). As a result, the PHP timeout on your webserver has to be high enough to upload 100% of the zip archive before the timeout hits.

    You can work around this by any of these:
    a) increasing the PHP timeout
    or
    b) reducing the ‘split’ size in the expert section at the bottom of the ‘Settings’ tab, to produce smaller zip files
    or
    c) switch to a remote storage method that supports chunked/resumable uploading: these include: FTP, FTPS, S3, Dropbox, Rackspace, Google Drive, Copy.Com.

    Best wishes,
    David

    #72756
    Raymond
    Participant

    Don’t get why it all ways stop uploading on the 1ST backup .zip file and it takes hours and a lot of CPU time to do it.

    I just heard my CPU fan going a lot on my server. So I did htop to see why. It was updraftplus doing it’s backup.

    Got this photo to show that.

    Uses 100% of CPU power.

    I got this script that can back up my WordPress very fast. In like 3 minutes make the backup of my database and and upload folder to my WordPress. The upload folder has a lot of photos and videos in it so it’s big.

    Here is the script and I bet others would like it.

    #!/bin/bash
     
     
     #Variables
     name_base=<code>date +&quot;%Y-%m-%d&quot;</code> #define name base for archive off of date
     backupdir="/media/USBdisk2-3TB/home/part2/mystuff/WordPress_Backup"
     contentdir="/usr/share/wordpress/wp-content/uploads/"
     
     #Start backup
     
     tar cvf WordPress_Data-$name_base.tar $contentdir
     
     mysqldump -u root --password=########## --databases wordpress > $backupdir/WordPress_Data-$name_base.sql
     
     #Create tar file
     
     #Add mysql dump to archive
     tar -rvf $backupdir/WordPress_Data-$name_base.tar $backupdir/WordPress_Data-$name_base.sql
     
     #Cleanup
     rm $backupdir/WordPress_Data-$name_base.sql

    At the top can just change that for other servers and it will work.

    I just need to make it so it saves to my other LAN sever that will be easy.

    Can log in to it with out having a password in the script because I did a long key on it to both just easy 2 commands like this:

    ssh-keygen -t rsa
    cat /root/.ssh/id_rsa.pub | ssh [email protected] "mkdir -p ~/.ssh && cat >> /root/.ssh/authorized_keys"

    Have to change that a little too like the IP for others.

    Then have to get it to only save like 2 or so backups and delete the old backups.

    Seems like this is a better faster way to do a backup of WordPress.

    I guess in a few days my have that script so it does the 2 other things I said. Seems a lot better then updraftplus.

    Can just have a cron job run this script every 24 hours.

    -Raymond Day

    #72881
    Raymond
    Participant

    I put the post here because it would not let me edit it but found out can edit the URL to the post number to edit it. But don’t look like can delete this post.

    -Raymond Day

    #72898
    Raymond
    Participant

    Here is what the log allway says at the end:

    17556.216 (9) Saving backup history
    17556.218 (9) backup_2014-12-25-1101_my_weblog_915fece0ab3b-plugins.zip: plugins: This file has already been successfully uploaded
    17556.219 (9) backup_2014-12-25-1101_my_weblog_915fece0ab3b-themes.zip: themes: This file has already been successfully uploaded
    17556.219 (9) backup_2014-12-25-1101_my_weblog_915fece0ab3b-uploads.zip: uploads: This file has not yet been successfully uploaded: will queue
    17556.219 (9) backup_2014-12-25-1101_my_weblog_915fece0ab3b-uploads2.zip: uploads: This file has not yet been successfully uploaded: will queue
    17556.219 (9) backup_2014-12-25-1101_my_weblog_915fece0ab3b-uploads3.zip: uploads: This file has not yet been successfully uploaded: will queue
    17556.220 (9) backup_2014-12-25-1101_my_weblog_915fece0ab3b-uploads4.zip: uploads: This file has not yet been successfully uploaded: will queue
    17556.220 (9) backup_2014-12-25-1101_my_weblog_915fece0ab3b-uploads5.zip: uploads: This file has not yet been successfully uploaded: will queue
    17556.220 (9) backup_2014-12-25-1101_my_weblog_915fece0ab3b-uploads6.zip: uploads: This file has not yet been successfully uploaded: will queue
    17556.220 (9) backup_2014-12-25-1101_my_weblog_915fece0ab3b-uploads7.zip: uploads: This file has not yet been successfully uploaded: will queue
    17556.220 (9) backup_2014-12-25-1101_my_weblog_915fece0ab3b-uploads8.zip: uploads: This file has not yet been successfully uploaded: will queue
    17556.221 (9) backup_2014-12-25-1101_my_weblog_915fece0ab3b-others.zip: others: This file has not yet been successfully uploaded: will queue
    17556.221 (9) backup_2014-12-25-1101_my_weblog_915fece0ab3b-others2.zip: others: This file has not yet been successfully uploaded: will queue
    17556.221 (9) backup_2014-12-25-1101_my_weblog_915fece0ab3b-db.gz: db: This file has not yet been successfully uploaded: will queue
    17556.222 (9) Requesting upload of the files that have not yet been successfully uploaded (11)
    17556.222 (9) Cloud backup selection: sftp
    17556.223 (9) Beginning dispatch of backup to remote (sftp)
    17573.830 (9) SFTP: Successfully logged in
    17573.831 (9) SCP/SFTP upload: attempt: backup_2014-12-25-1101_my_weblog_915fece0ab3b-uploads.zip

    It all way uploads the 1st 2 files but the 1st .zip big file it only uploads it about 1/3 of it all the time.

    -Raymond Day

    #73042
    Raymond
    Participant

    Just edit my /etc/php5/apache2/php.ini file.

    upload_max_filesize = 256M

    and:

    max_execution_time = 280

    The max time was set at 30.

    Then restatered it with:

    service apache2 restart

    Time will tell if this works.

    My upload is set to:

    SFTP / SCP

    So it must be using sftp but maybe scp and that’s why it don’t work. Not sure.

    -Raymond Day

    #73064
    Raymond
    Participant

    Still not working. It always gets stuck at:

    Uploading files to remote storage (18%, file 3 of 11)

    This shows too:

    Fri, December 26, 2014 06:11 – next resumption: 3 (after 629s) –

    I guess it gives up after 10 resumptions.

    I will see if I can get it to save 100’s of small files as one big backup after it stops this backup.

    It is using sftp too because in the log I seen this:

    Beginning dispatch of backup to remote (sftp) so it should do big chunks of .zip file backups.

    -Raymond Day

    #73088
    Raymond
    Participant

    I set the upload chunk form 2000 to 500 and now looks like it’s working!

    Clicked to start the backup and looks like it auto started a new one. Looks like it will finish this time.

    I guess I could put the size to 1500 because that’s seems when it errored on the big .zip files. But to be on the save side I guess 1300 by be good.

    Just don’t like it that it has to put a backup’s in a lot of files if it could just be one file.

    All so I put my big key in the log in. So it don’t show my root password on there.

    The key is stored at:

    /root/.ssh/id_rsa file.

    I removed the Begin and end RSA Private key lines.

    -Raymond Day

    #73533
    udadmin
    Keymaster

    Hi Ray,

    I’ve emailed you a new version of UpdraftPlus which adds support for chunking/resuming for SFTP. Please give it a try, and let me know how you get on.

    David

    #73749
    Raymond
    Participant

    I installed the e-mail one and unzip it over the the I had. Pressed Yes for all.

    root@ubuntu-i3:/usr/share/wordpress/wp-content/plugins# unzip updraftplus.trunk.zip
    Archive:  updraftplus.trunk.zip
    replace updraftplus/readme.txt? [y]es, [n]o, [A]ll, [N]one, [r]ename: A
    

    Then ran it but it still get’s errors.

    Here is what WordPress says:

    Sat, December 27, 2014 20:28 - next resumption: 4 (after 818s) - show log - delete schedule
    Warning: A very large file was encountered: uploads/2010/07/Just got in What it's like.ogg (size: 290.8 Mb)
    Warning: A very large file was encountered: uploads/2010/06/How I left Dairy Sun 6-6-2010.ogg (size: 259.5 Mb)
    Warning: A very large file was encountered: uploads/2014/10/MAH00131.mp4 (size: 309.8 Mb)
    Warning: A very large file was encountered: uploads/2014/03/After-2-hours-made-room-in-the-cooler.mp4 (size: 253 Mb)
    Warning: A very large file was encountered: uploads/2014/11/MAH00143.mp4 (size: 703.5 Mb)
    Warning: A very large file was encountered: uploads/2014/11/MAH00138.mp4 (size: 285.7 Mb)
    Warning: A very large file was encountered: uploads/2014/11/MAH00139.mp4 (size: 410 Mb)
    Warning: A very large file was encountered: uploads/2014/06/7-AD-skids-in-cooler.-I-got-1-by-this-time-IONX0006.mp4 (size: 272.9 Mb)
    Warning: A very large file was encountered: uploads/2014/08/Nice-Sunday-at-work-B4-I-clocked-in-IONX0032.mp4 (size: 257.8 Mb)
    Warning: A very large file was encountered: uploads/2008/10/Beth_Lisa_Gracey_Wall_1988.avi (size: 267.9 Mb)
    Waiting until scheduled time to retry because of errors

    Can see it’s it’s forth try. So I am sure it will try and try and then give up.

    -Raymond Day

    #73888
    Raymond
    Participant

    Looks like it auto worked this time!

    Though the night it was working looks like. It takes hours to upload it. I guess about 8 or so but looks like it worked.

    Here is the ending of the log for it.

    35901.376 (14) Recording as successfully uploaded: backup_2014-12-27-2028_my_weblog_36b445fb207e-uploads24.zip (aa0eb3f7de7c74ebf314ddf4530aec72)
    35901.452 (14) Deleting local file: backup_2014-12-27-2028_my_weblog_36b445fb207e-uploads24.zip: OK
    35901.452 (14) SCP/SFTP upload: attempt: backup_2014-12-27-2028_my_weblog_36b445fb207e-others.zip
    35902.025 (14) Sftp chunked upload: 11.7 % uploaded
    35902.594 (14) Sftp chunked upload: 23.5 % uploaded
    35903.858 (14) Sftp chunked upload: 35.2 % uploaded
    35904.441 (14) Sftp chunked upload: 47 % uploaded
    35905.015 (14) Sftp chunked upload: 58.7 % uploaded
    35905.581 (14) Sftp chunked upload: 70.5 % uploaded
    35906.152 (14) Sftp chunked upload: 82.2 % uploaded
    35906.719 (14) Sftp chunked upload: 93.9 % uploaded
    35907.017 (14) Recording as successfully uploaded: backup_2014-12-27-2028_my_weblog_36b445fb207e-others.zip (204abae3f9fc541233e77f9140369e41)
    35907.020 (14) Deleting local file: backup_2014-12-27-2028_my_weblog_36b445fb207e-others.zip: OK
    35907.021 (14) SCP/SFTP upload: attempt: backup_2014-12-27-2028_my_weblog_36b445fb207e-db.gz
    35907.596 (14) Sftp chunked upload: 12.6 % uploaded
    35908.169 (14) Sftp chunked upload: 25.1 % uploaded
    35908.741 (14) Sftp chunked upload: 37.7 % uploaded
    35909.314 (14) Sftp chunked upload: 50.2 % uploaded
    35909.903 (14) Sftp chunked upload: 62.8 % uploaded
    35910.477 (14) Sftp chunked upload: 75.3 % uploaded
    35911.053 (14) Sftp chunked upload: 87.9 % uploaded
    35911.608 (14) Recording as successfully uploaded: backup_2014-12-27-2028_my_weblog_36b445fb207e-db.gz (3b9668a4dda24b1552b63ffbaf319d10)
    35911.611 (14) Deleting local file: backup_2014-12-27-2028_my_weblog_36b445fb207e-db.gz: OK
    35911.613 (14) Retain: beginning examination of existing backup sets; user setting: retain_files=2, retain_db=2
    35911.613 (14) Number of backup sets in history: 2
    35911.614 (14) Examining backup set with datestamp: 1419730117 (Dec 28 2014 01:28:37)
    35911.614 (14) 1419730117: db: this set includes a database (backup_2014-12-27-2028_my_weblog_36b445fb207e-db.gz); db count is now 1
    35911.615 (14) 1419730117: this backup set remains non-empty (1/1); will retain in history
    35911.616 (14) Examining backup set with datestamp: 1419697005 (Dec 27 2014 16:16:45)
    35911.616 (14) 1419697005: db: this set includes a database (backup_2014-12-27-1116_my_weblog_63a743f446f4-db.gz); db count is now 2
    35911.618 (14) 1419697005: this backup set remains non-empty (1/1); will retain in history
    35911.618 (14) Retain: saving new backup history (sets now: 2) and finishing retain operation
    35911.619 (14) Resume backup (36b445fb207e, 14): finish run
    35911.620 (14) There were no errors in the uploads, so the 'resume' event (15) is being unscheduled
    35911.622 (14) No email will/can be sent - the user has not configured an email address.
    35911.623 (14) The backup apparently succeeded (with warnings) and is now complete

    Uploaded all 24.zip backups this time.

    I used the one you e-mailed me. Seems like it works.

    -Raymond Day

    #74262
    udadmin
    Keymaster

    Hi Ray,

    Great – glad it worked for you!

    Waiting until scheduled time to retry because of errors

    When you see this, it’s best to wait until the job gives a final finishing state. UpdraftPlus has various re-try strategies, to try to overcome temporary conditions.

    Best wishes,
    David

Viewing 15 posts - 1 through 15 (of 15 total)
  • The topic ‘It could not upload it all!’ is closed to new replies.