Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Contribute to GitLab
Sign in
Toggle navigation
C
client-s3
Project
Project
Details
Activity
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
李金钢
client-s3
Commits
174c2ef0
Commit
174c2ef0
authored
Mar 10, 2026
by
lijingang
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
增加超时时间并优化deleteS3Folder分批处理,避免连接超时
parent
48ae0e48
Pipeline
#300
failed with stages
Changes
1
Pipelines
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
31 additions
and
13 deletions
+31
-13
s3-client.js
lib/s3-client.js
+31
-13
No files found.
lib/s3-client.js
View file @
174c2ef0
...
@@ -25,9 +25,10 @@ try {
...
@@ -25,9 +25,10 @@ try {
secretAccessKey
:
"lOA7cJSzTY2nEgVeCR5WQhIf1tN9HvDjGmrpoP0w"
,
secretAccessKey
:
"lOA7cJSzTY2nEgVeCR5WQhIf1tN9HvDjGmrpoP0w"
,
},
},
forcePathStyle
:
true
,
// Must be enabled for RustFS compatibility
forcePathStyle
:
true
,
// Must be enabled for RustFS compatibility
maxAttempts
:
3
,
requestHandler
:
new
NodeHttpHandler
({
requestHandler
:
new
NodeHttpHandler
({
connectionTimeout
:
3000
,
connectionTimeout
:
10000
,
// Increased from 3000 to 10000 ms
socketTimeout
:
5000
,
socketTimeout
:
30000
,
// Increased from 5000 to 30000 ms
}),
}),
});
});
}
catch
(
error
)
{
}
catch
(
error
)
{
...
@@ -151,18 +152,35 @@ const deleteS3Folder = async (bucket, prefix) => {
...
@@ -151,18 +152,35 @@ const deleteS3Folder = async (bucket, prefix) => {
const
listResponse
=
await
s3
.
send
(
new
ListObjectsV2Command
(
listParams
));
const
listResponse
=
await
s3
.
send
(
new
ListObjectsV2Command
(
listParams
));
if
(
listResponse
.
Contents
&&
listResponse
.
Contents
.
length
>
0
)
{
if
(
listResponse
.
Contents
&&
listResponse
.
Contents
.
length
>
0
)
{
//
Add objects to delete lis
t
//
Convert objects to delete forma
t
objectsToDelete
=
listResponse
.
Contents
.
map
(
obj
=>
({
Key
:
obj
.
Key
}));
const
allObjects
=
listResponse
.
Contents
.
map
(
obj
=>
({
Key
:
obj
.
Key
}));
// Delete objects in batches (S3 allows up to 1000 objects per DeleteObjects request)
// Process in batches of max 1000 objects per request (S3 limit)
const
deleteParams
=
{
const
batchSize
=
1000
;
Bucket
:
bucket
,
for
(
let
i
=
0
;
i
<
allObjects
.
length
;
i
+=
batchSize
)
{
Delete
:
{
Objects
:
objectsToDelete
}
const
batch
=
allObjects
.
slice
(
i
,
i
+
batchSize
);
};
try
{
const
deleteResponse
=
await
s3
.
send
(
new
DeleteObjectsCommand
(
deleteParams
));
const
deleteParams
=
{
deletedCount
+=
deleteResponse
.
Deleted
?
deleteResponse
.
Deleted
.
length
:
0
;
Bucket
:
bucket
,
console
.
log
(
`Deleted
${
deleteResponse
.
Deleted
?
deleteResponse
.
Deleted
.
length
:
0
}
objects from
${
folderPrefix
}
`
);
Delete
:
{
Objects
:
batch
}
};
const
deleteResponse
=
await
s3
.
send
(
new
DeleteObjectsCommand
(
deleteParams
));
const
deletedInBatch
=
deleteResponse
.
Deleted
?
deleteResponse
.
Deleted
.
length
:
0
;
deletedCount
+=
deletedInBatch
;
console
.
log
(
`Deleted
${
deletedInBatch
}
objects from
${
folderPrefix
}
(batch
${
Math
.
floor
(
i
/
batchSize
)
+
1
}
)`
);
// Small delay between batches to avoid overwhelming the server
if
(
i
+
batchSize
<
allObjects
.
length
)
{
await
new
Promise
(
resolve
=>
setTimeout
(
resolve
,
100
));
}
}
catch
(
batchError
)
{
console
.
error
(
`Error deleting batch
${
Math
.
floor
(
i
/
batchSize
)
+
1
}
:`
,
batchError
.
message
);
// Continue with next batch instead of failing completely
console
.
log
(
'Continuing with remaining batches...'
);
}
}
}
}
continuationToken
=
listResponse
.
NextContinuationToken
;
continuationToken
=
listResponse
.
NextContinuationToken
;
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment