backblaze confusing error messages due to backblaze design

[Andreas WagnerParticipant]

hey all,
i think this issue relates to a couple of entries here in the forum, and i am not sure how to easily solve it, but i think it should be addressed as it might be a relief on some of the support questions.

also, this is just a theory, so please correct me if i’m wrong. i might be totally off path here.

as i understand backblaze uses an orchestrator architecture to delegate requests to its actual internal vaults. that means that every request against their api will be answered by this orchestrator, rather than the actual backup storage server. in this setup it is quite common to receive some sort of 500 error, as it merely states the situation of the orchestrator, not the backup infrastructure.

what i am seeing in the logs quite often now are 503 errors, stating that the orchestrator is currently too busy, please try again shortly. the confusion i am talking about is how updraft handles this response. there are two things:
1. it triggers an error email (i have set email notifications to the choice: only when backup has errors)
2. it triggers the resume event, and as i can see, the plugin basically waits about 1500 seconds and then tries again.

this cycle repeats until eventually (occasionally after hours) all uploads have been successful. i only receive one email per upload cycle, so at least that is quite consistent, as it seems the plugin just gathers errors, without shooting an email on each single one of them.

i also believe that eventually it stops resuming, possibly after some arbitrary amount of retries (couldn’t find it in the code) if by that time it still couldn’t upload all the chunks it seems to me that the plugin will try uploading them on the next scheduled run, which is helpful. in the end i still seem to have complete updates on some bigger sites, if i just wait a day or two.

i see this post is getting a bit out of hand, but i guess the situation is just a bit complex.

here’s my two cents:
so in regards to updraft plugin, it really seems to work fine technically, but because of backblaze’s definition of error handling (503 is normal, just try again) and updraft’s interpretation of an error (uh oh, there’s a 503, something obviously went wrong) the end user gets mixed signals. maybe you can do something about that interpretation?
in regards to the retry mechanism for backblaze, i.e. updraft’s wait and resume option, maybe it would make sense to be able as a user under ‘expert settings’ to make this waiting time configurable between say a minimum of 10 seconds and a maximum of 1500 seconds? then a user could experiment with different scenarios, akin the size of upload chunks.

as an aside, since i have been using aws s3 before (waaaayy more expensive then backblaze): they use massive load balancers in front of their s3 vaults, so in a sense, any request will with near certainty be routed on the first try, if one is not in the unlucky situation of being that particular requester that makes the load balancer trip over (against which aws is probably shielded by early upscaling these single points of failure)

uff, that went somewhere. i hope anyone can follow this and make sense out of it.

hth

source: https://www.backblaze.com/blog/b2-503-500-server-error/

[Andreas WagnerParticipant]

this might be related: https://updraftplus.com/forums/topic/errors-during-backup-of-large-site-to-backblaze/

[udadminKeymaster]

> i only receive one email per upload cycle

You have apparently turned on “Debug mode” in the expert settings section (“Check this to receive more information and emails on the backup process – useful if something is going wrong.”). To not receive these emails, turn it off.

[Author]

Posts