www
/
www-new
12
3
Fork 5
Code Issues 58 Pull Requests 42 Projects Releases Wiki Activity

Improve image optimization run-time and memory usage #457

Merged
a258wang merged 8 commits from amy-image-optimization-quick-fix into main 2022-06-04 19:31:02 -04:00
Conversation 1 Commits 8 Files Changed 1 +65 -36
a258wang commented 2022-06-04 15:12:04 -04:00
Owner
Copy Link

This PR aims to resolve the issues surrounding the optimize-images script, at least in the short term, in order to unblock other work.

Problems: Our optimize-images script was taking a very long time to run (~7 minutes in CI, when successful). This led to two problems:

  1. It was near impossible to run the script locally.
  2. CI jobs would often get killed on the optimize-images step.

Solutions:

  1. Resize the images in images/events prior to optimizing them, similar to what is done for the images in images/team. This solution on its own reduced the run-time of the script to ~30 seconds locally on Amy's laptop, which is comparable to the run-time of the script back when it was originally written.
  2. EDIT: Copy/resize/optimize the images in batches of 32 at a time. The reason why the CI job was being killed is because the script would run out of memory, however this change should resolve that while also keeping build times reasonable (~30 sec locally/when deploying on caffeine, ~3 min in CI). As a temporary fix, this PR also replaces the images in images/events with their resized + optimized versions. (For some unknown reason, Solution 1 is not sufficient to solve Problem 2.)

This PR also adds some logging to the script so we can get a (slightly) better sense of where the script is getting stuck.

Related issue: #456

This PR aims to resolve the issues surrounding the optimize-images script, at least in the short term, in order to unblock other work. **Problems:** Our optimize-images script was taking a very long time to run (~7 minutes in CI, when successful). This led to two problems: 1. It was near impossible to run the script locally. 2. CI jobs would often get killed on the optimize-images step. **Solutions:** 1. Resize the images in `images/events` prior to optimizing them, similar to what is done for the images in `images/team`. This solution on its own reduced the run-time of the script to ~30 seconds locally on Amy's laptop, which is comparable to the run-time of the script back when it was originally written. 2. EDIT: Copy/resize/optimize the images in batches of 32 at a time. The reason why the CI job was being killed is because the script would run out of memory, however this change should resolve that while also keeping build times reasonable (~30 sec locally/when deploying on caffeine, ~3 min in CI). ~~As a temporary fix, this PR also replaces the images in `images/events` with their resized + optimized versions. (For some unknown reason, Solution 1 is not sufficient to solve Problem 2.)~~ This PR also adds some logging to the script so we can get a (slightly) better sense of where the script is getting stuck. Related issue: #456
a258wang added 1 commit 2022-06-04 15:12:06 -04:00
continuous-integration/drone/push Build is failing Details
19c5aecb07 Resize events images and add logging
a258wang requested review from snedadah 2022-06-04 15:12:29 -04:00
a258wang added 1 commit 2022-06-04 15:19:44 -04:00
continuous-integration/drone/push Build is failing Details
2a62f0b508 Add resize logging
a258wang removed review request for snedadah 2022-06-04 15:48:27 -04:00
a258wang added 1 commit 2022-06-04 15:52:31 -04:00
continuous-integration/drone/push Build is passing Details
6a3b151ed2 Replace events images with optimized images
a258wang added 2 commits 2022-06-04 16:21:52 -04:00
continuous-integration/drone/push Build is failing Details
49c272efa3 Fix brackets
a258wang changed title from Improve image optimization run-time to WIP: Improve image optimization run-time (quick fix) 2022-06-04 16:24:05 -04:00
a258wang added 1 commit 2022-06-04 16:48:55 -04:00
continuous-integration/drone/push Build is passing Details
84eba39f7a Revert "Revert "Replace events images with optimized images"" 
This reverts commit 6860089d66.
a258wang changed title from WIP: Improve image optimization run-time (quick fix) to Improve image optimization run-time (quick fix) 2022-06-04 17:00:21 -04:00
a258wang requested review from snedadah 2022-06-04 17:00:37 -04:00
a258wang changed title from Improve image optimization run-time (quick fix) to Improve image optimization run-time - quick fix 2022-06-04 17:00:56 -04:00
a3thakra approved these changes 2022-06-04 17:56:17 -04:00
snedadah approved these changes 2022-06-04 18:36:04 -04:00
snedadah left a comment
Owner
Copy Link

Nice catch!! Makes sense why those specific images were taking so long now… this solution is great!

I think this solution is good enough to solve our problems for now. In the future, we should consider an alternative image storage mechanism anyway...

Nice catch!! Makes sense why those specific images were taking so long now… this solution is great! I think this solution is good enough to solve our problems for now. In the future, we should consider an alternative image storage mechanism anyway...
scripts/optimize-images.ts
@ -35,6 +37,8 @@ const ENCODER_OPTIONS: { [encoder: string]: Record<string, unknown> } = {
void optimizeImages();
export async function optimizeImages() {
const startTime = Date.now();
snedadah commented 2022-06-04 18:35:49 -04:00
Owner
Copy Link

By the way, JavaScript has a built in console.time()

https://developer.mozilla.org/en-US/docs/Web/API/console/time

By the way, JavaScript has a built in `console.time()` https://developer.mozilla.org/en-US/docs/Web/API/console/time
👀 1
a258wang added 2 commits 2022-06-04 19:07:36 -04:00
continuous-integration/drone/push Build is passing Details
0360382882 Use Promise.all on batches of 32
a258wang changed title from Improve image optimization run-time - quick fix to Improve image optimization run-time 2022-06-04 19:09:36 -04:00
a258wang changed title from Improve image optimization run-time to Improve image optimization run-time and memory usage 2022-06-04 19:19:02 -04:00
a258wang requested review from a3thakra 2022-06-04 19:19:35 -04:00
a258wang requested review from snedadah 2022-06-04 19:19:39 -04:00
a258wang commented 2022-06-04 19:20:49 -04:00
Author
Owner
Copy Link

Re-requested reviews because I made a pretty significant change - see PR description for update

Re-requested reviews because I made a pretty significant change - see PR description for update
snedadah approved these changes 2022-06-04 19:27:50 -04:00
snedadah left a comment
Owner
Copy Link

"Revert "Revert "Revert :)

`"Revert "Revert "Revert` :)
a258wang merged commit 443925190e into main 2022-06-04 19:31:02 -04:00
a258wang deleted branch amy-image-optimization-quick-fix 2022-06-04 19:31:02 -04:00
Sign in to join this conversation.
Reviewers
No reviewers
a3thakra
snedadah
Labels
Clear labels   
a11y

   
Backlog

   
Blocked

   
Bug

   
Content

   
Dependencies

   
Design

   
Feature Request

   
Good First Issue

   
In Progress

   
Performance

   
Priority - High

   
Priority - Low

   
Priority - Medium

   
Untriaged
No Label
a11y
Backlog
Blocked
Bug
Content
Dependencies
Design
Feature Request
Good First Issue
In Progress
Performance
Priority - High
Priority - Low
Priority - Medium
Untriaged
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: www/www-new#457
No description provided.
Delete Branch "amy-image-optimization-quick-fix"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?