updraft hangs at uploading files

[RyanParticipant]

I deleted the schedule on the 8th “resumption”.
Nothing has been uploaded to S3 and wp-cron.php is taking 100% cpu the entire time.

[udadminKeymaster]

Hi Ryan,

I have emailed you the next release of UD a few days early. It has a little more logging of unusual S3 conditions, which might help.

David

[RyanParticipant]

I installed that version but I dont see anything additional logged.
It still just ends with

1203.516 (1) Requesting upload of the files that have not yet been successfully uploaded (20)
1203.524 (1) Cloud backup selection: s3
1203.534 (1) Beginning dispatch of backup to remote (s3)

[RyanParticipant]

I deleted all files from the S3 storage and all files from wp-content/updraft
Then I did a settings wipe in updraft.

It still shows 5 existing backups (that do not actually exist)
I have re-scanned both local and remote.

I was trying to get a clean slate to start over with but it seems to be holding on to some sort of log or setting to make it think it has the last 5 backups still available.
If I try to delete any of them, it hangs trying to remove from remote storage. I left it running for about 45 minutes. CPU is at 100% during that time.
If I try to download it does the same thing, requesting file from remote storage.. hang.. cpu maxed.

The last entry in the logs is always in reference to connecting to S3.

[udadminKeymaster]

Hi Ryan,

UD doesn’t delete backup sets from its history that aren’t found in a scan of the remote storage if those backups were believed to be there, in case the remote storage is misbehaving/down – it gives you the opportunity to bring them back. To force-delete a backup set from UD, press the “Delete” button in the ‘Existing Backups’ tab.

The CPU usage and total lack of an error message is interesting. I wonder if the PHP engine or one of the PHP modules installed has a bug and is seg-faulting. If you re-try the command line run, but run it under strace, does it show a segfault?

Also… it’s possible to switch to a different code path for S3 entirely, which may well avoid whatever the issue is, by adding this early in your wp-config.php:

define(‘UPDRAFTPLUS_S3_OLDLIB’, true);

This will cause UD to use the old S3 library that was its preferred S3 library until February.

David

[RyanParticipant]

I’m running it through strace now.
Is there a function other than updraft_backup_all that I can use?
Something to only backup the database or at least exclude uploads?
It takes a long time just to zip up the uploads folder.

[udadminKeymaster]

Hi Ryan,

I think you can use updraft_backupnow_backup_database. Though, if your settings in your settings tab don’t include plugins, then those will be excluded when updraft_backup_all is used – the ‘all’ means “files (whichever ones are set) and database” rather than “everything you can”.

David

[RyanParticipant]

The command line finished but the updraft admin interface still shows it running. It looks like cron took it over.

The last line from strace was
exit_group(255) = ?

I will try the code change now

[udadminKeymaster]

Hi Ryan,

Are you able to give more of the last lines of the strace output than that? e.g. Around 25 lines.

David

[RyanParticipant]

when the interface is stuck on uploading, the log shows:

0019.751 (0) Requesting upload of the files that have not yet been successfully uploaded (3)
0019.775 (0) Cloud backup selection: s3
0019.784 (0) Beginning dispatch of backup to remote (s3)

strace tail at the same point:

close(8)                                = 0
access("/home/####/public_html/wp-content/plugins/updraftplus/oc/aws/Aws/Common/Enum/DateFormat.php", F_OK) = 0
lstat("/home/####/public_html/wp-content/plugins/updraftplus/oc/aws/Aws/Common/Enum/DateFormat.php", {st_mode=S_IFREG|0644, st_size=941, ...}) = 0
open("/home/####/public_html/wp-content/plugins/updraftplus/oc/aws/Aws/Common/Enum/DateFormat.php", O_RDONLY) = 8
fstat(8, {st_mode=S_IFREG|0644, st_size=941, ...}) = 0
fstat(8, {st_mode=S_IFREG|0644, st_size=941, ...}) = 0
fstat(8, {st_mode=S_IFREG|0644, st_size=941, ...}) = 0
mmap(NULL, 941, PROT_READ, MAP_SHARED, 8, 0) = 0x7fce48c92000
munmap(0x7fce48c92000, 941)             = 0
close(8)                                = 0
access("/home/####/public_html/wp-content/plugins/updraftplus/oc/guzzle/Guzzle/Http/Curl/RequestMediator.php", F_OK) = 0
lstat("/home/####/public_html/wp-content/plugins/updraftplus/oc/guzzle/Guzzle/Http/Curl/RequestMediator.php", {st_mode=S_IFREG|0644, st_size=4445, ...}) = 0
open("/home/####/public_html/wp-content/plugins/updraftplus/oc/guzzle/Guzzle/Http/Curl/RequestMediator.php", O_RDONLY) = 8
fstat(8, {st_mode=S_IFREG|0644, st_size=4445, ...}) = 0
fstat(8, {st_mode=S_IFREG|0644, st_size=4445, ...}) = 0
fstat(8, {st_mode=S_IFREG|0644, st_size=4445, ...}) = 0
mmap(NULL, 4445, PROT_READ, MAP_SHARED, 8, 0) = 0x7fce48c91000
munmap(0x7fce48c91000, 4445)            = 0
close(8)                                = 0
clock_gettime(CLOCK_MONOTONIC, {87204, 922141274}) = 0

[RyanParticipant]

I just checked the cpanel backups. They are using the same aws keys as updraft but a different bucket.
They are running fine each night.

[RyanParticipant]

And here is the rest of strace after it times out.

--- SIGPROF (Profiling timer expired) @ 0 (0) ---
setitimer(ITIMER_PROF, {it_interval={0, 0}, it_value={900, 0}}, NULL) = 0
rt_sigaction(SIGPROF, {0x93fb27, [PROF], SA_RESTORER|SA_RESTART, 0x33996326a0}, {0x93fb27, [PROF], SA_RESTORER|SA_RESTART, 0x33996326a0}, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [PROF], NULL, 8) = 0
setitimer(ITIMER_PROF, {it_interval={0, 0}, it_value={0, 0}}, NULL) = 0
sendto(6, "\1\1", 5, MSG_DONTWAIT, NULL, 0) = 5
close(6)                                = 0
close(7)                                = 0
rt_sigaction(SIGPIPE, NULL, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x33996326a0}, 8) = 0
rt_sigaction(SIGPIPE, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x33996326a0}, NULL, 8) = 0
rt_sigaction(SIGPIPE, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x33996326a0}, NULL, 8) = 0
rt_sigaction(SIGPIPE, NULL, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x33996326a0}, 8) = 0
rt_sigaction(SIGPIPE, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x33996326a0}, NULL, 8) = 0
rt_sigaction(SIGPIPE, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x33996326a0}, NULL, 8) = 0
munmap(0x7fce492fc000, 266240)          = 0
munmap(0x7fce48f5e000, 266240)          = 0
munmap(0x7fce49176000, 266240)          = 0
munmap(0x7fce491b7000, 266240)          = 0
munmap(0x7fce490f4000, 266240)          = 0
munmap(0x7fce4975c000, 266240)          = 0
munmap(0x7fce4971b000, 266240)          = 0
munmap(0x7fce4981f000, 266240)          = 0
munmap(0x7fce498e2000, 266240)          = 0
munmap(0x7fce498a1000, 266240)          = 0
munmap(0x7fce49a27000, 266240)          = 0
munmap(0x7fce499e6000, 266240)          = 0
munmap(0x7fce499a5000, 266240)          = 0
munmap(0x7fce49aa9000, 266240)          = 0
munmap(0x7fce4a3d7000, 266240)          = 0
close(2)                                = 0
close(1)                                = 0
munmap(0x7fce57ee9000, 4096)            = 0
close(0)                                = 0
munmap(0x7fce57eea000, 4096)            = 0
munmap(0x7fce4a314000, 266240)          = 0
munmap(0x7fce49923000, 266240)          = 0
munmap(0x7fce4a641000, 266240)          = 0
munmap(0x7fce4a600000, 266240)          = 0
munmap(0x7fce4a5bf000, 266240)          = 0
munmap(0x7fce4a355000, 266240)          = 0
munmap(0x7fce4a251000, 266240)          = 0
munmap(0x7fce49f57000, 266240)          = 0
munmap(0x7fce49f98000, 266240)          = 0
munmap(0x7fce49dd1000, 266240)          = 0
munmap(0x7fce49e12000, 266240)          = 0
munmap(0x7fce49bc9000, 266240)          = 0
munmap(0x7fce49964000, 266240)          = 0
munmap(0x7fce49135000, 266240)          = 0
munmap(0x7fce49072000, 266240)          = 0
munmap(0x7fce490b3000, 266240)          = 0
setitimer(ITIMER_PROF, {it_interval={0, 0}, it_value={0, 0}}, NULL) = 0
brk(0x3e2f000)                          = 0x3e2f000
brk(0x3caf000)                          = 0x3caf000
brk(0x3b2f000)                          = 0x3b2f000
brk(0x386f000)                          = 0x386f000
brk(0x3650000)                          = 0x3650000
brk(0x344f000)                          = 0x344f000
munmap(0x7fce48c93000, 266240)          = 0
munmap(0x7fce48cd4000, 266240)          = 0
munmap(0x7fce48d15000, 266240)          = 0
munmap(0x7fce48d56000, 266240)          = 0
munmap(0x7fce48d97000, 266240)          = 0
munmap(0x7fce48dd8000, 266240)          = 0
munmap(0x7fce48e19000, 266240)          = 0
munmap(0x7fce48e5a000, 266240)          = 0
munmap(0x7fce48e9b000, 266240)          = 0
munmap(0x7fce48edc000, 266240)          = 0
munmap(0x7fce48f1d000, 266240)          = 0
munmap(0x7fce48faf000, 266240)          = 0
munmap(0x7fce48ff0000, 266240)          = 0
munmap(0x7fce49031000, 266240)          = 0
munmap(0x7fce491f8000, 266240)          = 0
munmap(0x7fce49239000, 266240)          = 0
munmap(0x7fce4927a000, 266240)          = 0
munmap(0x7fce492bb000, 266240)          = 0
munmap(0x7fce4933d000, 266240)          = 0
munmap(0x7fce4937e000, 266240)          = 0
munmap(0x7fce493bf000, 266240)          = 0
munmap(0x7fce49400000, 266240)          = 0
munmap(0x7fce49441000, 266240)          = 0
munmap(0x7fce49482000, 266240)          = 0
munmap(0x7fce494c3000, 266240)          = 0
munmap(0x7fce49504000, 266240)          = 0
munmap(0x7fce49545000, 266240)          = 0
munmap(0x7fce49586000, 266240)          = 0
munmap(0x7fce495c7000, 266240)          = 0
munmap(0x7fce49608000, 266240)          = 0
munmap(0x7fce49649000, 266240)          = 0
munmap(0x7fce4968a000, 266240)          = 0
munmap(0x7fce496da000, 266240)          = 0
munmap(0x7fce4979d000, 266240)          = 0
munmap(0x7fce497de000, 266240)          = 0
munmap(0x7fce49860000, 266240)          = 0
munmap(0x7fce49a68000, 266240)          = 0
munmap(0x7fce49b06000, 266240)          = 0
munmap(0x7fce49b47000, 266240)          = 0
munmap(0x7fce49b88000, 266240)          = 0
munmap(0x7fce49c0a000, 266240)          = 0
munmap(0x7fce49c4b000, 266240)          = 0
munmap(0x7fce49c8c000, 266240)          = 0
munmap(0x7fce49ccd000, 266240)          = 0
munmap(0x7fce49d0e000, 266240)          = 0
munmap(0x7fce49d4f000, 266240)          = 0
munmap(0x7fce49d90000, 266240)          = 0
munmap(0x7fce49e53000, 266240)          = 0
munmap(0x7fce49e94000, 266240)          = 0
munmap(0x7fce49ed5000, 266240)          = 0
munmap(0x7fce49f16000, 266240)          = 0
munmap(0x7fce49fd9000, 266240)          = 0
munmap(0x7fce4a01a000, 266240)          = 0
munmap(0x7fce4a05b000, 266240)          = 0
munmap(0x7fce4a09c000, 266240)          = 0
munmap(0x7fce4a10c000, 266240)          = 0
munmap(0x7fce4a14d000, 266240)          = 0
munmap(0x7fce4a18e000, 266240)          = 0
munmap(0x7fce4a1cf000, 266240)          = 0
munmap(0x7fce4a210000, 266240)          = 0
munmap(0x7fce4a292000, 266240)          = 0
munmap(0x7fce4a2d3000, 266240)          = 0
munmap(0x7fce4a396000, 266240)          = 0
munmap(0x7fce4a439000, 266240)          = 0
munmap(0x7fce4a47a000, 266240)          = 0
munmap(0x7fce4a4bb000, 266240)          = 0
munmap(0x7fce4a4fc000, 266240)          = 0
munmap(0x7fce4a53d000, 266240)          = 0
munmap(0x7fce4a57e000, 266240)          = 0
munmap(0x7fce4a6a7000, 266240)          = 0
close(5)                                = 0
munmap(0x7fce4a6eb000, 104857600)       = 0
close(4)                                = 0
close(3)                                = 0
munmap(0x7fce50cf9000, 2760432)         = 0
munmap(0x7fce50f9b000, 2154416)         = 0
munmap(0x7fce511a9000, 2263856)         = 0
munmap(0x7fce4a686000, 135168)          = 0
munmap(0x7fce572b2000, 1052672)         = 0
munmap(0x7fce573b3000, 266240)          = 0
munmap(0x7fce513d2000, 323584)          = 0
exit_group(255)                         = ?

[RyanParticipant]

also, there are now 3 wp-cron.php processes at 100% cpu
updraft admin page shows uploading files 0%, waiting on the 4th retry

[udadminKeymaster]

Hi Ryan,

This suggests the possibility that the outgoing network traffic to S3 is being silently dropped, by a packet filtering firewall somewhere on the route to S3. (Hence no fatal errors – UD just continues waiting for network traffic, until it hits the PHP timeout limit and is killed).

Do you get the same result if you add this after the opening

[RyanParticipant]

I thought the same thing about the traffic being blocked, but my csf rules have not changed. 443 and 80 are both open along with the other default ports.

I added that define code to the top of my wp-config file but it didn’t seem to make a difference.

I have cpanel set to back up to S3 on this same server and it works fine. I just tested it today and it verified its connection with s3 right away. I even changed the auth key to the new one I created for updraft and it verified.

I’ll run another strace and share the file. Can I send it to your email address instead of making it public?

[Author]

Posts