Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Add support for 429 retry in S3 ResourcePath

Description

In PREOPS-1034 it is noted that we are getting some rate limiting errors from processing that should be handled by the S3 code. A 429 ClientError should trigger backoff but a while back we disabled automatic backoff support for generic ClientError – there are too many cases where trying again would be a waste of time. 429 error code explicitly indicates that we should retry. This will possibly need to be supported in many of the S3 APIs in ResourcePath.

Checklist

Lucidchart Diagrams

Issue Matrix

hide

Activity

Show:
Kian-Tat Lim
March 27, 2022 at 10:18 PM

Oops, never marked this reviewed, even though the PR had been.

Tim Jenness
March 24, 2022 at 6:15 PM

I've refactored the S3 code to allow the S3 download/upload file code path to catch a 429 ClientError and use backoff. I have no idea how to test it without merging it to main.

Tim Jenness
March 23, 2022 at 3:57 AM

Can you please explain to me what you are asking me to do? My reading from that GitHub discussion was that they are never going to support 429. I was under the impression that 429 was only supported in the legacy retry mode (which is what uses the JSON files).

I have reorganized the S3 code to try to catch 429 outside of boto but it's not pretty. I can't get Google to trigger a 429 from a notebook session since I can't seem to download files fast enough (writing files too fast does trigger the retry as shown above).

Colin Slater
March 23, 2022 at 12:01 AM

I did read the issue. I don't think anything there contradicts what I said. If you want to understand why boto is or is not retrying, you need to look at the response json, not just the HTTP status code.

Tim Jenness
March 22, 2022 at 11:42 PM

please read the referenced GitHub issue. They told me to go away because AWS will never return with a 429 but maybe you can convince them to catch 429 themselves.

Done
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Labels

Reviewers

Kian-Tat Lim

Story Points

RubinTeam

Components

Checklist

Created February 8, 2022 at 7:04 PM
Updated March 28, 2022 at 6:21 PM
Resolved March 28, 2022 at 6:21 PM