www-new/scripts/optimize-images.ts

139 lines
4.4 KiB
TypeScript
Raw Normal View History

/* eslint-disable @typescript-eslint/no-unsafe-assignment */
/* eslint-disable @typescript-eslint/no-unsafe-call */
/* eslint-disable @typescript-eslint/no-unsafe-member-access */
// TODO: upgrade libsquoosh once types are available: https://github.com/GoogleChromeLabs/squoosh/issues/1077
import { cpus } from "os";
import path from "path";
// eslint-disable-next-line @typescript-eslint/ban-ts-comment
// @ts-ignore
import { ImagePool } from "@squoosh/lib";
import fse from "fs-extra";
import { default as getImageDimensions } from "image-size";
const SOURCE_DIRECTORY = "images";
const DESTINATION_DIRECTORY = path.join("public", "images");
Improve image optimization run-time and memory usage (#457) This PR aims to resolve the issues surrounding the optimize-images script, at least in the short term, in order to unblock other work. **Problems:** Our optimize-images script was taking a very long time to run (~7 minutes in CI, when successful). This led to two problems: 1. It was near impossible to run the script locally. 2. CI jobs would often get killed on the optimize-images step. **Solutions:** 1. Resize the images in `images/events` prior to optimizing them, similar to what is done for the images in `images/team`. This solution on its own reduced the run-time of the script to ~30 seconds locally on Amy's laptop, which is comparable to the run-time of the script back when it was originally written. 2. EDIT: Copy/resize/optimize the images in batches of 32 at a time. The reason why the CI job was being killed is because the script would run out of memory, however this change should resolve that while also keeping build times reasonable (~30 sec locally/when deploying on caffeine, ~3 min in CI). ~~As a temporary fix, this PR also replaces the images in `images/events` with their resized + optimized versions. (For some unknown reason, Solution 1 is not sufficient to solve Problem 2.)~~ This PR also adds some logging to the script so we can get a (slightly) better sense of where the script is getting stuck. Related issue: #456 Co-authored-by: Amy <a258wang@uwaterloo.ca> Reviewed-on: https://git.csclub.uwaterloo.ca/www/www-new/pulls/457 Reviewed-by: Shahan Neda <snedadah@csclub.uwaterloo.ca>
2022-06-04 19:31:00 -04:00
// directories are relative to SOURCE_DIRECTORY
const TEAM_IMAGES_DIRECTORY = path.join("team", "");
Improve image optimization run-time and memory usage (#457) This PR aims to resolve the issues surrounding the optimize-images script, at least in the short term, in order to unblock other work. **Problems:** Our optimize-images script was taking a very long time to run (~7 minutes in CI, when successful). This led to two problems: 1. It was near impossible to run the script locally. 2. CI jobs would often get killed on the optimize-images step. **Solutions:** 1. Resize the images in `images/events` prior to optimizing them, similar to what is done for the images in `images/team`. This solution on its own reduced the run-time of the script to ~30 seconds locally on Amy's laptop, which is comparable to the run-time of the script back when it was originally written. 2. EDIT: Copy/resize/optimize the images in batches of 32 at a time. The reason why the CI job was being killed is because the script would run out of memory, however this change should resolve that while also keeping build times reasonable (~30 sec locally/when deploying on caffeine, ~3 min in CI). ~~As a temporary fix, this PR also replaces the images in `images/events` with their resized + optimized versions. (For some unknown reason, Solution 1 is not sufficient to solve Problem 2.)~~ This PR also adds some logging to the script so we can get a (slightly) better sense of where the script is getting stuck. Related issue: #456 Co-authored-by: Amy <a258wang@uwaterloo.ca> Reviewed-on: https://git.csclub.uwaterloo.ca/www/www-new/pulls/457 Reviewed-by: Shahan Neda <snedadah@csclub.uwaterloo.ca>
2022-06-04 19:31:00 -04:00
const EVENTS_IMAGES_DIRECTORY = path.join("events", "");
const IMAGE_MINIMUM_SIZE = 512;
const GET_ENCODER_FROM_EXTENSION: { [imageExtension: string]: string } = {
jpg: "mozjpeg",
jpeg: "mozjpeg",
Improve image optimization run-time and memory usage (#457) This PR aims to resolve the issues surrounding the optimize-images script, at least in the short term, in order to unblock other work. **Problems:** Our optimize-images script was taking a very long time to run (~7 minutes in CI, when successful). This led to two problems: 1. It was near impossible to run the script locally. 2. CI jobs would often get killed on the optimize-images step. **Solutions:** 1. Resize the images in `images/events` prior to optimizing them, similar to what is done for the images in `images/team`. This solution on its own reduced the run-time of the script to ~30 seconds locally on Amy's laptop, which is comparable to the run-time of the script back when it was originally written. 2. EDIT: Copy/resize/optimize the images in batches of 32 at a time. The reason why the CI job was being killed is because the script would run out of memory, however this change should resolve that while also keeping build times reasonable (~30 sec locally/when deploying on caffeine, ~3 min in CI). ~~As a temporary fix, this PR also replaces the images in `images/events` with their resized + optimized versions. (For some unknown reason, Solution 1 is not sufficient to solve Problem 2.)~~ This PR also adds some logging to the script so we can get a (slightly) better sense of where the script is getting stuck. Related issue: #456 Co-authored-by: Amy <a258wang@uwaterloo.ca> Reviewed-on: https://git.csclub.uwaterloo.ca/www/www-new/pulls/457 Reviewed-by: Shahan Neda <snedadah@csclub.uwaterloo.ca>
2022-06-04 19:31:00 -04:00
JPG: "mozjpeg",
png: "oxipng",
};
const ENCODER_OPTIONS: { [encoder: string]: Record<string, unknown> } = {
mozjpeg: {},
oxipng: {},
};
void optimizeImages();
export async function optimizeImages() {
Improve image optimization run-time and memory usage (#457) This PR aims to resolve the issues surrounding the optimize-images script, at least in the short term, in order to unblock other work. **Problems:** Our optimize-images script was taking a very long time to run (~7 minutes in CI, when successful). This led to two problems: 1. It was near impossible to run the script locally. 2. CI jobs would often get killed on the optimize-images step. **Solutions:** 1. Resize the images in `images/events` prior to optimizing them, similar to what is done for the images in `images/team`. This solution on its own reduced the run-time of the script to ~30 seconds locally on Amy's laptop, which is comparable to the run-time of the script back when it was originally written. 2. EDIT: Copy/resize/optimize the images in batches of 32 at a time. The reason why the CI job was being killed is because the script would run out of memory, however this change should resolve that while also keeping build times reasonable (~30 sec locally/when deploying on caffeine, ~3 min in CI). ~~As a temporary fix, this PR also replaces the images in `images/events` with their resized + optimized versions. (For some unknown reason, Solution 1 is not sufficient to solve Problem 2.)~~ This PR also adds some logging to the script so we can get a (slightly) better sense of where the script is getting stuck. Related issue: #456 Co-authored-by: Amy <a258wang@uwaterloo.ca> Reviewed-on: https://git.csclub.uwaterloo.ca/www/www-new/pulls/457 Reviewed-by: Shahan Neda <snedadah@csclub.uwaterloo.ca>
2022-06-04 19:31:00 -04:00
const startTime = Date.now();
const imagePaths = await getFilePathsInDirectory(SOURCE_DIRECTORY);
await fse.emptyDir(DESTINATION_DIRECTORY);
// maximum number of workers is 8 in order to avoid running out of memory
const numberOfWorkers = Math.min(cpus().length, 8);
Improve image optimization run-time and memory usage (#457) This PR aims to resolve the issues surrounding the optimize-images script, at least in the short term, in order to unblock other work. **Problems:** Our optimize-images script was taking a very long time to run (~7 minutes in CI, when successful). This led to two problems: 1. It was near impossible to run the script locally. 2. CI jobs would often get killed on the optimize-images step. **Solutions:** 1. Resize the images in `images/events` prior to optimizing them, similar to what is done for the images in `images/team`. This solution on its own reduced the run-time of the script to ~30 seconds locally on Amy's laptop, which is comparable to the run-time of the script back when it was originally written. 2. EDIT: Copy/resize/optimize the images in batches of 32 at a time. The reason why the CI job was being killed is because the script would run out of memory, however this change should resolve that while also keeping build times reasonable (~30 sec locally/when deploying on caffeine, ~3 min in CI). ~~As a temporary fix, this PR also replaces the images in `images/events` with their resized + optimized versions. (For some unknown reason, Solution 1 is not sufficient to solve Problem 2.)~~ This PR also adds some logging to the script so we can get a (slightly) better sense of where the script is getting stuck. Related issue: #456 Co-authored-by: Amy <a258wang@uwaterloo.ca> Reviewed-on: https://git.csclub.uwaterloo.ca/www/www-new/pulls/457 Reviewed-by: Shahan Neda <snedadah@csclub.uwaterloo.ca>
2022-06-04 19:31:00 -04:00
// process smaller batches in order to reduce memory usage
const batchSize = 32;
for (let i = 0; i < imagePaths.length; i += batchSize) {
// use a new ImagePool for each batch to reduce memory usage
const imagePool = new ImagePool(numberOfWorkers);
Improve image optimization run-time and memory usage (#457) This PR aims to resolve the issues surrounding the optimize-images script, at least in the short term, in order to unblock other work. **Problems:** Our optimize-images script was taking a very long time to run (~7 minutes in CI, when successful). This led to two problems: 1. It was near impossible to run the script locally. 2. CI jobs would often get killed on the optimize-images step. **Solutions:** 1. Resize the images in `images/events` prior to optimizing them, similar to what is done for the images in `images/team`. This solution on its own reduced the run-time of the script to ~30 seconds locally on Amy's laptop, which is comparable to the run-time of the script back when it was originally written. 2. EDIT: Copy/resize/optimize the images in batches of 32 at a time. The reason why the CI job was being killed is because the script would run out of memory, however this change should resolve that while also keeping build times reasonable (~30 sec locally/when deploying on caffeine, ~3 min in CI). ~~As a temporary fix, this PR also replaces the images in `images/events` with their resized + optimized versions. (For some unknown reason, Solution 1 is not sufficient to solve Problem 2.)~~ This PR also adds some logging to the script so we can get a (slightly) better sense of where the script is getting stuck. Related issue: #456 Co-authored-by: Amy <a258wang@uwaterloo.ca> Reviewed-on: https://git.csclub.uwaterloo.ca/www/www-new/pulls/457 Reviewed-by: Shahan Neda <snedadah@csclub.uwaterloo.ca>
2022-06-04 19:31:00 -04:00
await Promise.all(
imagePaths.slice(i, i + batchSize).map(async (imagePath) => {
const imageStartTime = Date.now();
const sourcePath = path.join(SOURCE_DIRECTORY, imagePath);
const destinationPath = path.join(DESTINATION_DIRECTORY, imagePath);
const fileExtension = imagePath.split(".").pop() ?? "";
const encoder = GET_ENCODER_FROM_EXTENSION[fileExtension];
if (!encoder) {
await fse.copy(sourcePath, destinationPath);
console.log(
`Copied ${imagePath} in ${getElapsedSeconds(imageStartTime)}s`
);
return;
}
const rawImageFile = await fse.readFile(sourcePath);
const ingestedImage = imagePool.ingestImage(rawImageFile);
const { width, height } = getImageDimensions(rawImageFile);
Improve image optimization run-time and memory usage (#457) This PR aims to resolve the issues surrounding the optimize-images script, at least in the short term, in order to unblock other work. **Problems:** Our optimize-images script was taking a very long time to run (~7 minutes in CI, when successful). This led to two problems: 1. It was near impossible to run the script locally. 2. CI jobs would often get killed on the optimize-images step. **Solutions:** 1. Resize the images in `images/events` prior to optimizing them, similar to what is done for the images in `images/team`. This solution on its own reduced the run-time of the script to ~30 seconds locally on Amy's laptop, which is comparable to the run-time of the script back when it was originally written. 2. EDIT: Copy/resize/optimize the images in batches of 32 at a time. The reason why the CI job was being killed is because the script would run out of memory, however this change should resolve that while also keeping build times reasonable (~30 sec locally/when deploying on caffeine, ~3 min in CI). ~~As a temporary fix, this PR also replaces the images in `images/events` with their resized + optimized versions. (For some unknown reason, Solution 1 is not sufficient to solve Problem 2.)~~ This PR also adds some logging to the script so we can get a (slightly) better sense of where the script is getting stuck. Related issue: #456 Co-authored-by: Amy <a258wang@uwaterloo.ca> Reviewed-on: https://git.csclub.uwaterloo.ca/www/www-new/pulls/457 Reviewed-by: Shahan Neda <snedadah@csclub.uwaterloo.ca>
2022-06-04 19:31:00 -04:00
await ingestedImage.decoded;
Improve image optimization run-time and memory usage (#457) This PR aims to resolve the issues surrounding the optimize-images script, at least in the short term, in order to unblock other work. **Problems:** Our optimize-images script was taking a very long time to run (~7 minutes in CI, when successful). This led to two problems: 1. It was near impossible to run the script locally. 2. CI jobs would often get killed on the optimize-images step. **Solutions:** 1. Resize the images in `images/events` prior to optimizing them, similar to what is done for the images in `images/team`. This solution on its own reduced the run-time of the script to ~30 seconds locally on Amy's laptop, which is comparable to the run-time of the script back when it was originally written. 2. EDIT: Copy/resize/optimize the images in batches of 32 at a time. The reason why the CI job was being killed is because the script would run out of memory, however this change should resolve that while also keeping build times reasonable (~30 sec locally/when deploying on caffeine, ~3 min in CI). ~~As a temporary fix, this PR also replaces the images in `images/events` with their resized + optimized versions. (For some unknown reason, Solution 1 is not sufficient to solve Problem 2.)~~ This PR also adds some logging to the script so we can get a (slightly) better sense of where the script is getting stuck. Related issue: #456 Co-authored-by: Amy <a258wang@uwaterloo.ca> Reviewed-on: https://git.csclub.uwaterloo.ca/www/www-new/pulls/457 Reviewed-by: Shahan Neda <snedadah@csclub.uwaterloo.ca>
2022-06-04 19:31:00 -04:00
const shouldResize =
(imagePath.startsWith(TEAM_IMAGES_DIRECTORY) ||
imagePath.startsWith(EVENTS_IMAGES_DIRECTORY)) &&
(width ?? 0) > IMAGE_MINIMUM_SIZE &&
(height ?? 0) > IMAGE_MINIMUM_SIZE;
Improve image optimization run-time and memory usage (#457) This PR aims to resolve the issues surrounding the optimize-images script, at least in the short term, in order to unblock other work. **Problems:** Our optimize-images script was taking a very long time to run (~7 minutes in CI, when successful). This led to two problems: 1. It was near impossible to run the script locally. 2. CI jobs would often get killed on the optimize-images step. **Solutions:** 1. Resize the images in `images/events` prior to optimizing them, similar to what is done for the images in `images/team`. This solution on its own reduced the run-time of the script to ~30 seconds locally on Amy's laptop, which is comparable to the run-time of the script back when it was originally written. 2. EDIT: Copy/resize/optimize the images in batches of 32 at a time. The reason why the CI job was being killed is because the script would run out of memory, however this change should resolve that while also keeping build times reasonable (~30 sec locally/when deploying on caffeine, ~3 min in CI). ~~As a temporary fix, this PR also replaces the images in `images/events` with their resized + optimized versions. (For some unknown reason, Solution 1 is not sufficient to solve Problem 2.)~~ This PR also adds some logging to the script so we can get a (slightly) better sense of where the script is getting stuck. Related issue: #456 Co-authored-by: Amy <a258wang@uwaterloo.ca> Reviewed-on: https://git.csclub.uwaterloo.ca/www/www-new/pulls/457 Reviewed-by: Shahan Neda <snedadah@csclub.uwaterloo.ca>
2022-06-04 19:31:00 -04:00
if (width && height && shouldResize) {
const smallerDimension = width < height ? "width" : "height";
Improve image optimization run-time and memory usage (#457) This PR aims to resolve the issues surrounding the optimize-images script, at least in the short term, in order to unblock other work. **Problems:** Our optimize-images script was taking a very long time to run (~7 minutes in CI, when successful). This led to two problems: 1. It was near impossible to run the script locally. 2. CI jobs would often get killed on the optimize-images step. **Solutions:** 1. Resize the images in `images/events` prior to optimizing them, similar to what is done for the images in `images/team`. This solution on its own reduced the run-time of the script to ~30 seconds locally on Amy's laptop, which is comparable to the run-time of the script back when it was originally written. 2. EDIT: Copy/resize/optimize the images in batches of 32 at a time. The reason why the CI job was being killed is because the script would run out of memory, however this change should resolve that while also keeping build times reasonable (~30 sec locally/when deploying on caffeine, ~3 min in CI). ~~As a temporary fix, this PR also replaces the images in `images/events` with their resized + optimized versions. (For some unknown reason, Solution 1 is not sufficient to solve Problem 2.)~~ This PR also adds some logging to the script so we can get a (slightly) better sense of where the script is getting stuck. Related issue: #456 Co-authored-by: Amy <a258wang@uwaterloo.ca> Reviewed-on: https://git.csclub.uwaterloo.ca/www/www-new/pulls/457 Reviewed-by: Shahan Neda <snedadah@csclub.uwaterloo.ca>
2022-06-04 19:31:00 -04:00
// specifying only one dimension maintains the aspect ratio
const preprocessOptions = {
resize: {
enabled: true,
[smallerDimension]: IMAGE_MINIMUM_SIZE,
},
};
Improve image optimization run-time and memory usage (#457) This PR aims to resolve the issues surrounding the optimize-images script, at least in the short term, in order to unblock other work. **Problems:** Our optimize-images script was taking a very long time to run (~7 minutes in CI, when successful). This led to two problems: 1. It was near impossible to run the script locally. 2. CI jobs would often get killed on the optimize-images step. **Solutions:** 1. Resize the images in `images/events` prior to optimizing them, similar to what is done for the images in `images/team`. This solution on its own reduced the run-time of the script to ~30 seconds locally on Amy's laptop, which is comparable to the run-time of the script back when it was originally written. 2. EDIT: Copy/resize/optimize the images in batches of 32 at a time. The reason why the CI job was being killed is because the script would run out of memory, however this change should resolve that while also keeping build times reasonable (~30 sec locally/when deploying on caffeine, ~3 min in CI). ~~As a temporary fix, this PR also replaces the images in `images/events` with their resized + optimized versions. (For some unknown reason, Solution 1 is not sufficient to solve Problem 2.)~~ This PR also adds some logging to the script so we can get a (slightly) better sense of where the script is getting stuck. Related issue: #456 Co-authored-by: Amy <a258wang@uwaterloo.ca> Reviewed-on: https://git.csclub.uwaterloo.ca/www/www-new/pulls/457 Reviewed-by: Shahan Neda <snedadah@csclub.uwaterloo.ca>
2022-06-04 19:31:00 -04:00
await ingestedImage.preprocess(preprocessOptions);
Improve image optimization run-time and memory usage (#457) This PR aims to resolve the issues surrounding the optimize-images script, at least in the short term, in order to unblock other work. **Problems:** Our optimize-images script was taking a very long time to run (~7 minutes in CI, when successful). This led to two problems: 1. It was near impossible to run the script locally. 2. CI jobs would often get killed on the optimize-images step. **Solutions:** 1. Resize the images in `images/events` prior to optimizing them, similar to what is done for the images in `images/team`. This solution on its own reduced the run-time of the script to ~30 seconds locally on Amy's laptop, which is comparable to the run-time of the script back when it was originally written. 2. EDIT: Copy/resize/optimize the images in batches of 32 at a time. The reason why the CI job was being killed is because the script would run out of memory, however this change should resolve that while also keeping build times reasonable (~30 sec locally/when deploying on caffeine, ~3 min in CI). ~~As a temporary fix, this PR also replaces the images in `images/events` with their resized + optimized versions. (For some unknown reason, Solution 1 is not sufficient to solve Problem 2.)~~ This PR also adds some logging to the script so we can get a (slightly) better sense of where the script is getting stuck. Related issue: #456 Co-authored-by: Amy <a258wang@uwaterloo.ca> Reviewed-on: https://git.csclub.uwaterloo.ca/www/www-new/pulls/457 Reviewed-by: Shahan Neda <snedadah@csclub.uwaterloo.ca>
2022-06-04 19:31:00 -04:00
console.log(
`Resized ${sourcePath} in ${getElapsedSeconds(imageStartTime)}s`
);
}
Improve image optimization run-time and memory usage (#457) This PR aims to resolve the issues surrounding the optimize-images script, at least in the short term, in order to unblock other work. **Problems:** Our optimize-images script was taking a very long time to run (~7 minutes in CI, when successful). This led to two problems: 1. It was near impossible to run the script locally. 2. CI jobs would often get killed on the optimize-images step. **Solutions:** 1. Resize the images in `images/events` prior to optimizing them, similar to what is done for the images in `images/team`. This solution on its own reduced the run-time of the script to ~30 seconds locally on Amy's laptop, which is comparable to the run-time of the script back when it was originally written. 2. EDIT: Copy/resize/optimize the images in batches of 32 at a time. The reason why the CI job was being killed is because the script would run out of memory, however this change should resolve that while also keeping build times reasonable (~30 sec locally/when deploying on caffeine, ~3 min in CI). ~~As a temporary fix, this PR also replaces the images in `images/events` with their resized + optimized versions. (For some unknown reason, Solution 1 is not sufficient to solve Problem 2.)~~ This PR also adds some logging to the script so we can get a (slightly) better sense of where the script is getting stuck. Related issue: #456 Co-authored-by: Amy <a258wang@uwaterloo.ca> Reviewed-on: https://git.csclub.uwaterloo.ca/www/www-new/pulls/457 Reviewed-by: Shahan Neda <snedadah@csclub.uwaterloo.ca>
2022-06-04 19:31:00 -04:00
const encodeOptions = { [encoder]: ENCODER_OPTIONS[encoder] };
await ingestedImage.encode(encodeOptions);
Improve image optimization run-time and memory usage (#457) This PR aims to resolve the issues surrounding the optimize-images script, at least in the short term, in order to unblock other work. **Problems:** Our optimize-images script was taking a very long time to run (~7 minutes in CI, when successful). This led to two problems: 1. It was near impossible to run the script locally. 2. CI jobs would often get killed on the optimize-images step. **Solutions:** 1. Resize the images in `images/events` prior to optimizing them, similar to what is done for the images in `images/team`. This solution on its own reduced the run-time of the script to ~30 seconds locally on Amy's laptop, which is comparable to the run-time of the script back when it was originally written. 2. EDIT: Copy/resize/optimize the images in batches of 32 at a time. The reason why the CI job was being killed is because the script would run out of memory, however this change should resolve that while also keeping build times reasonable (~30 sec locally/when deploying on caffeine, ~3 min in CI). ~~As a temporary fix, this PR also replaces the images in `images/events` with their resized + optimized versions. (For some unknown reason, Solution 1 is not sufficient to solve Problem 2.)~~ This PR also adds some logging to the script so we can get a (slightly) better sense of where the script is getting stuck. Related issue: #456 Co-authored-by: Amy <a258wang@uwaterloo.ca> Reviewed-on: https://git.csclub.uwaterloo.ca/www/www-new/pulls/457 Reviewed-by: Shahan Neda <snedadah@csclub.uwaterloo.ca>
2022-06-04 19:31:00 -04:00
const encodedImage = await ingestedImage.encodedWith[encoder];
await fse.outputFile(destinationPath, encodedImage.binary);
Improve image optimization run-time and memory usage (#457) This PR aims to resolve the issues surrounding the optimize-images script, at least in the short term, in order to unblock other work. **Problems:** Our optimize-images script was taking a very long time to run (~7 minutes in CI, when successful). This led to two problems: 1. It was near impossible to run the script locally. 2. CI jobs would often get killed on the optimize-images step. **Solutions:** 1. Resize the images in `images/events` prior to optimizing them, similar to what is done for the images in `images/team`. This solution on its own reduced the run-time of the script to ~30 seconds locally on Amy's laptop, which is comparable to the run-time of the script back when it was originally written. 2. EDIT: Copy/resize/optimize the images in batches of 32 at a time. The reason why the CI job was being killed is because the script would run out of memory, however this change should resolve that while also keeping build times reasonable (~30 sec locally/when deploying on caffeine, ~3 min in CI). ~~As a temporary fix, this PR also replaces the images in `images/events` with their resized + optimized versions. (For some unknown reason, Solution 1 is not sufficient to solve Problem 2.)~~ This PR also adds some logging to the script so we can get a (slightly) better sense of where the script is getting stuck. Related issue: #456 Co-authored-by: Amy <a258wang@uwaterloo.ca> Reviewed-on: https://git.csclub.uwaterloo.ca/www/www-new/pulls/457 Reviewed-by: Shahan Neda <snedadah@csclub.uwaterloo.ca>
2022-06-04 19:31:00 -04:00
console.log(
`Optimized ${sourcePath} in ${getElapsedSeconds(imageStartTime)}s`
);
})
);
await imagePool.close();
}
Improve image optimization run-time and memory usage (#457) This PR aims to resolve the issues surrounding the optimize-images script, at least in the short term, in order to unblock other work. **Problems:** Our optimize-images script was taking a very long time to run (~7 minutes in CI, when successful). This led to two problems: 1. It was near impossible to run the script locally. 2. CI jobs would often get killed on the optimize-images step. **Solutions:** 1. Resize the images in `images/events` prior to optimizing them, similar to what is done for the images in `images/team`. This solution on its own reduced the run-time of the script to ~30 seconds locally on Amy's laptop, which is comparable to the run-time of the script back when it was originally written. 2. EDIT: Copy/resize/optimize the images in batches of 32 at a time. The reason why the CI job was being killed is because the script would run out of memory, however this change should resolve that while also keeping build times reasonable (~30 sec locally/when deploying on caffeine, ~3 min in CI). ~~As a temporary fix, this PR also replaces the images in `images/events` with their resized + optimized versions. (For some unknown reason, Solution 1 is not sufficient to solve Problem 2.)~~ This PR also adds some logging to the script so we can get a (slightly) better sense of where the script is getting stuck. Related issue: #456 Co-authored-by: Amy <a258wang@uwaterloo.ca> Reviewed-on: https://git.csclub.uwaterloo.ca/www/www-new/pulls/457 Reviewed-by: Shahan Neda <snedadah@csclub.uwaterloo.ca>
2022-06-04 19:31:00 -04:00
console.log(`TOTAL DURATION: ${getElapsedSeconds(startTime)}s`);
}
async function getFilePathsInDirectory(directory: string): Promise<string[]> {
const entries = await fse.readdir(directory, { withFileTypes: true });
return (
await Promise.all(
entries.map(async (entry) => {
if (entry.isDirectory()) {
const subdirectory = path.join(directory, entry.name);
const subentries = await getFilePathsInDirectory(subdirectory);
return subentries.map((subentry) => path.join(entry.name, subentry));
}
return entry.name;
})
)
).flat();
}
Improve image optimization run-time and memory usage (#457) This PR aims to resolve the issues surrounding the optimize-images script, at least in the short term, in order to unblock other work. **Problems:** Our optimize-images script was taking a very long time to run (~7 minutes in CI, when successful). This led to two problems: 1. It was near impossible to run the script locally. 2. CI jobs would often get killed on the optimize-images step. **Solutions:** 1. Resize the images in `images/events` prior to optimizing them, similar to what is done for the images in `images/team`. This solution on its own reduced the run-time of the script to ~30 seconds locally on Amy's laptop, which is comparable to the run-time of the script back when it was originally written. 2. EDIT: Copy/resize/optimize the images in batches of 32 at a time. The reason why the CI job was being killed is because the script would run out of memory, however this change should resolve that while also keeping build times reasonable (~30 sec locally/when deploying on caffeine, ~3 min in CI). ~~As a temporary fix, this PR also replaces the images in `images/events` with their resized + optimized versions. (For some unknown reason, Solution 1 is not sufficient to solve Problem 2.)~~ This PR also adds some logging to the script so we can get a (slightly) better sense of where the script is getting stuck. Related issue: #456 Co-authored-by: Amy <a258wang@uwaterloo.ca> Reviewed-on: https://git.csclub.uwaterloo.ca/www/www-new/pulls/457 Reviewed-by: Shahan Neda <snedadah@csclub.uwaterloo.ca>
2022-06-04 19:31:00 -04:00
function getElapsedSeconds(startTime: number) {
return (Date.now() - startTime) / 1000;
}