www
/
www-new
12
3
Fork 5
Code Issues 58 Pull Requests 42 Projects Releases Wiki Activity

Reduce runtime of optimize-image script #456

Closed
snedadah wants to merge 18 commits from shahanneda/optimze-image-script into main
pull from: shahanneda/optimze-image-script
merge into: www:main
Conversation 2 Commits 18 Files Changed 138 +629 -8112
snedadah commented 2022-06-02 00:03:17 -04:00
Owner
Copy Link

Closes #441.

  • Changed optimize-images script to keep track of all the images it has already optimized (through a hash) in a already-optimized-images.json file.

  • Removed the public/images folder from gitignore so that optimized images will be tracked.

  • Added better logging for the script

  • Fixed issue with script ignoring images that end with ".JPG" (uppercase)

If reviewers want, I can try to revert the images commit so the number of files changed is reduced, however, I don't think its that bad, if you scroll down all the way once and click the "show more" button one time its pretty easy to get to the changed code files.

Usage:

Now, the ideal case is everytime someone adds a new image, they should run npm run build:images locally, and commit that.

However, if they forgot, that is fine, however, after a while if many images build up and we notice the build time going up again, someone should run npm run build:images and commit that to cache the optimized images.

staging: https://csclub.uwaterloo.ca/~a3thakra/csc/shahanneda/optimze-image-script/about/team/

in the future, we should consider properly storing images in a cdn

Closes #441. - Changed optimize-images script to keep track of all the images it has already optimized (through a hash) in a `already-optimized-images.json` file. - Removed the public/images folder from gitignore so that optimized images will be tracked. - Added better logging for the script - Fixed issue with script ignoring images that end with ".JPG" (uppercase) If reviewers want, I can try to revert the images commit so the number of files changed is reduced, however, I don't think its that bad, if you scroll down all the way once and click the "show more" button one time its pretty easy to get to the changed code files. Usage: Now, the ideal case is everytime someone adds a new image, they should run `npm run build:images` locally, and commit that. However, if they forgot, that is fine, however, after a while if many images build up and we notice the build time going up again, someone should run `npm run build:images` and commit that to cache the optimized images. staging: https://csclub.uwaterloo.ca/~a3thakra/csc/shahanneda/optimze-image-script/about/team/ in the future, we should consider properly storing images in a cdn
snedadah added 4 commits 2022-06-02 00:03:19 -04:00
continuous-integration/drone/push Build is failing Details
6f6510a5ee
Reduced number of workers
continuous-integration/drone/push Build is failing Details
5263c6880d
added console logging to image script
continuous-integration/drone/push Build is passing Details
4f79c73cd3
Ran optimize image script
snedadah added 2 commits 2022-06-02 00:12:33 -04:00
continuous-integration/drone/push Build is passing Details
9517e6d2c9
Updated naming for hashes
snedadah added 3 commits 2022-06-02 00:15:15 -04:00
snedadah added 2 commits 2022-06-02 02:01:37 -04:00
continuous-integration/drone/push Build is passing Details
2bff6684db
Testing warning
snedadah added 2 commits 2022-06-02 02:08:50 -04:00
continuous-integration/drone/push Build is passing Details
3d57bde02e
Testing warning on production
snedadah added 1 commit 2022-06-02 02:13:21 -04:00
continuous-integration/drone/push Build is passing Details
bead08ed09
updated warning
snedadah added 1 commit 2022-06-02 02:13:47 -04:00
continuous-integration/drone/push Build is passing Details
a3bd8da8cc
Updated env check
snedadah added 1 commit 2022-06-02 02:15:44 -04:00
continuous-integration/drone/push Build is passing Details
be527c382e
Updated ci check
snedadah added 1 commit 2022-06-02 02:37:37 -04:00
continuous-integration/drone/push Build is passing Details
b6129b3843
cleaned up
snedadah added 1 commit 2022-06-02 02:43:37 -04:00
continuous-integration/drone/push Build is passing Details
366e226520
Updated location of script
snedadah changed title from WIP: Reduce runtime of optimize-image script to Reduce runtime of optimize-image script 2022-06-02 02:56:40 -04:00
snedadah requested review from n3parikh 2022-06-02 02:56:57 -04:00
snedadah requested review from a3thakra 2022-06-02 02:56:58 -04:00
snedadah requested review from j285he 2022-06-02 02:56:58 -04:00
snedadah requested review from a258wang 2022-06-02 02:56:59 -04:00
a3thakra reviewed 2022-06-03 03:32:23 -04:00
scripts/optimize-images/optimize-images.ts
@ -0,0 +13,4 @@
import { ImagePool } from "@squoosh/lib";
import fse from "fs-extra";
import { default as getImageDimensions } from "image-size";
import SparkMD5 from "spark-md5";
a3thakra commented 2022-06-03 03:32:23 -04:00
Owner
Copy Link

Probably not very dangerous in this scenario, but we should just avoid md5 altogether since it's a broken hashing algorithm. Use a safer algorithm like sha256. You can use the nodejs builtin crypto library to do so. :)

Probably not very dangerous in this scenario, but we should just avoid md5 altogether since it's a broken hashing algorithm. Use a safer algorithm like sha256. You can use the nodejs builtin crypto library to do so. :)
snedadah commented 2022-06-06 20:12:49 -04:00
Author
Owner
Copy Link

Oh true, however my reasoning behind using MD5 is that it is one of the fastest hashing algorithms, and I doubt the unsafeness really effects our usecase.

Oh true, however my reasoning behind using MD5 is that it is one of the fastest hashing algorithms, and I doubt the unsafeness really effects our usecase.
a3thakra commented 2022-06-03 03:41:08 -04:00
Owner
Copy Link

The problem with checking in the public/images folder is that nothing is stopping anyone to add on to that folder.

It would be much better if we could:

  1. restore public/images artifact from a previous CI run before optimizing images
  2. run this optimize images script
  3. create a new public/images artifact and to overwrite the previous one

This was we are never storing 2x the images in our repo and it avoids all manual work while decreasing build times.

cc @r389li can we setup artifactory or something and add it to drone? https://plugins.drone.io/plugins/artifactory

The problem with checking in the public/images folder is that nothing is stopping anyone to add on to that folder. It would be much better if we could: 1. restore `public/images` artifact from a previous CI run before optimizing images 1. run this optimize images script 1. create a new `public/images` artifact and to overwrite the previous one This was we are never storing 2x the images in our repo and it avoids all manual work while decreasing build times. cc @r389li can we setup artifactory or something and add it to drone? https://plugins.drone.io/plugins/artifactory
r389li commented 2022-06-03 16:03:22 -04:00
Owner
Copy Link

@a3thakra we definitely don't need Artifactory. What we can do is put the images in /srv on Caffeine, and then just link to them using Apache URL rewrites. Let me know if you have other ideas.

@a3thakra we definitely don't need Artifactory. What we can do is put the images in `/srv` on Caffeine, and then just link to them using Apache URL rewrites. Let me know if you have other ideas.
👍 1
snedadah closed this pull request 2022-06-06 20:12:57 -04:00
All checks were successful
continuous-integration/drone/push Build is passing
Required
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No reviewers
n3parikh
a3thakra
j285he
a258wang
Labels
Clear labels   
a11y

   
Backlog

   
Blocked

   
Bug

   
Content

   
Dependencies

   
Design

   
Feature Request

   
Good First Issue

   
In Progress

   
Performance

   
Priority - High

   
Priority - Low

   
Priority - Medium

   
Untriaged
No Label
a11y
Backlog
Blocked
Bug
Content
Dependencies
Design
Feature Request
Good First Issue
In Progress
Performance
Priority - High
Priority - Low
Priority - Medium
Untriaged
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: www/www-new#456
No description provided.
Delete Branch "shahanneda/optimze-image-script"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?