Reduce runtime of optimize-image script
#456
Closed
snedadah
wants to merge 18 commits from shahanneda/optimze-image-script
into main
pull from: shahanneda/optimze-image-script
merge into: www:main
www:2022-07-08-MUA-Update
www:a258wang-events-2022-09-28
www:add-name-to-web-committe
www:amy-add-alt-tab-f22
www:amy-add-librarian-f22
www:amy-events-august-2022
www:amy-events-karaoke
www:amy-fix-vp-f22
www:amy-image-optimization-quick-fix-experiment
www:amy-news-2022-09-14
www:amy-random-shapes
www:amy-update-annie-headshot-f22
www:amy-update-community-reps
www:amy-update-internship-repo-f22
www:annies/add-circles-event
www:anniesun/add-myself-to-webstite
www:anniesun/update-blurbs
www:b72zhou-past-exec
www:bonnie-team-member-card
www:darren/add-class-profile-news
www:darren/add-myself-to-website
www:darren/add-news
www:darren/fix-dead-link
www:fix-oc-burger
www:j285he-add-news-09-07
www:j285he-ldap-test-example
www:jan-2023-constitution-changes
www:ldap-exec-test
www:main
www:neil-members
www:news-2022-09-08
www:news-2022-09-10
www:renovate/date-fns-tz-2.x
www:renovate/eslint-8.x
www:renovate/eslint-plugin-prettier-4.x
www:renovate/eslint-plugin-react-7.x
www:renovate/fs-extra-11.x
www:renovate/ldapts-4.x
www:renovate/mdx-js-loader-2.x
www:renovate/mdx-js-react-2.x
www:renovate/next-11.x
www:renovate/next-13.x
www:renovate/next-mdx-11.x
www:renovate/next-mdx-13.x
www:renovate/next-mdx-remote-3.x
www:renovate/node-18.x
www:renovate/node-19.x
www:renovate/npm-9.x
www:renovate/postcss-preset-env-8.x
www:renovate/react-18.x
www:renovate/react-dom-18.x
www:renovate/remark-14.x
www:renovate/remark-html-15.x
www:renovate/squoosh-lib-0.x
www:renovate/typescript-4.x
www:renovate/typescript-5.x
www:renovate/typescript-eslint-eslint-plugin-5.x
www:renovate/typescript-eslint-parser-5.x
www:richardshuai/karaoke
www:shahanneda-news-11jan2022
www:shahanneda/add-alt-tab-news
www:shahanneda/add-bot
www:shahanneda/add-code-party-1
www:shahanneda/add-code-party-23
www:shahanneda/add-event
www:shahanneda/add-events
www:shahanneda/add-myself
www:shahanneda/alt-tab-and-cp
www:shahanneda/fix-build-image-script
www:shahanneda/fix-name-parts
www:shahanneda/update-exec-blurbs
www:shahanneda/update-project-program
www:shahanneda/update-team
www:support-multiple-news-homepage
www:unix101-1
www:update-events
www:w23-update-team
www:warning-header
Reviewers
Request review
Labels
Apply labels
Clear labels
a11y
Backlog
Blocked
Bug
Content
Dependencies
Design
Feature Request
Good First Issue
In Progress
Performance
Priority - High
Priority - Low
Priority - Medium
Untriaged
No Label
a11y
Backlog
Blocked
Bug
Content
Dependencies
Design
Feature Request
Good First Issue
In Progress
Performance
Priority - High
Priority - Low
Priority - Medium
Untriaged
Milestone
Set milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Assign users
Clear assignees
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.
No due date set.
Dependencies
No dependencies set.
Reference: www/www-new#456
Reference in new issue
There is no content yet.
Delete Branch 'shahanneda/optimze-image-script'
Deleting a branch is permanent. It CANNOT be undone. Continue?
No
Yes
Closes #441.
Changed optimize-images script to keep track of all the images it has already optimized (through a hash) in a
already-optimized-images.json
file.Removed the public/images folder from gitignore so that optimized images will be tracked.
Added better logging for the script
Fixed issue with script ignoring images that end with ".JPG" (uppercase)
If reviewers want, I can try to revert the images commit so the number of files changed is reduced, however, I don't think its that bad, if you scroll down all the way once and click the "show more" button one time its pretty easy to get to the changed code files.
Usage:
Now, the ideal case is everytime someone adds a new image, they should run
npm run build:images
locally, and commit that.However, if they forgot, that is fine, however, after a while if many images build up and we notice the build time going up again, someone should run
npm run build:images
and commit that to cache the optimized images.staging: https://csclub.uwaterloo.ca/~a3thakra/csc/shahanneda/optimze-image-script/about/team/
in the future, we should consider properly storing images in a cdn
WIP: Reduce runtime of optimize-image scriptto Reduce runtime of optimize-image script 10 months agoimport { ImagePool } from "@squoosh/lib";
import fse from "fs-extra";
import { default as getImageDimensions } from "image-size";
import SparkMD5 from "spark-md5";
Probably not very dangerous in this scenario, but we should just avoid md5 altogether since it's a broken hashing algorithm. Use a safer algorithm like sha256. You can use the nodejs builtin crypto library to do so. :)
Oh true, however my reasoning behind using MD5 is that it is one of the fastest hashing algorithms, and I doubt the unsafeness really effects our usecase.
The problem with checking in the public/images folder is that nothing is stopping anyone to add on to that folder.
It would be much better if we could:
public/images
artifact from a previous CI run before optimizing imagespublic/images
artifact and to overwrite the previous oneThis was we are never storing 2x the images in our repo and it avoids all manual work while decreasing build times.
cc @r389li can we setup artifactory or something and add it to drone? https://plugins.drone.io/plugins/artifactory
@a3thakra we definitely don't need Artifactory. What we can do is put the images in
/srv
on Caffeine, and then just link to them using Apache URL rewrites. Let me know if you have other ideas.Reviewers