public
/
mirror
10
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
mirror/merlin
History
Andrew Wang 39fc29a0a8 force overrides trace and update x.org repo 2022-08-05 01:25:18 +00:00
..
arthur refactor sync 2022-07-28 02:09:36 +00:00
cmd/arthur modify arthur status message 2022-07-02 23:35:26 +00:00
config finish adding all projects 2022-07-30 02:27:49 +00:00
logger Bump version, add verbosity and dry run options 2022-06-23 02:43:42 +00:00
sync force overrides trace and update x.org repo 2022-08-05 01:25:18 +00:00
test fix gitignore 2021-09-09 00:31:28 -04:00
.gitignore write merlin.go 2021-09-17 00:31:49 -04:00
Handoff.md force overrides trace and update x.org repo 2022-08-05 01:25:18 +00:00
README.md minor fixes 2022-08-03 03:58:48 +00:00
go.mod Bump version, add verbosity and dry run options 2022-06-23 02:43:42 +00:00
go.sum Bump version, add verbosity and dry run options 2022-06-23 02:43:42 +00:00
merlin-config-all.ini fix up csc-sync-debian 2022-07-24 17:18:12 +00:00
merlin-config-test.ini fix up csc-sync-debian 2022-07-24 17:18:12 +00:00
merlin-config.ini force overrides trace and update x.org repo 2022-08-05 01:25:18 +00:00
merlin-go.service finalize move and add report_mirror 2022-07-15 04:31:43 +00:00
merlin.go minor fixes 2022-08-03 03:58:48 +00:00

README.md

Merlin

Build Status

This folder contains the code for merlin (which does the actual syncing) and arthur (which sends commands to merlin).

Check out the the mirror env for a testing environment

Usage

go build merlin.go

Then configure merlin-config.ini and run using ./merlin

Nice Features To Add

  • detect if an rsync process is stuck (watch the stdout/stderr of the rsync processes)
  • detect if config file is changed and automatically reload (fsnotify)
  • sort arthur status by last time synced
  • respond to arthur without blocking main thread
  • state files appear to be getting extra newlines
  • log rotation for log files (also maybe state reporting to prometheus)
  • split off arthur with a more featureful arthur for debugging
    • last sync runtime, time until next sync, due to sync (true/false for repo would sync if could)
    • get expected rsync command or repo config
    • details for last 10 syncs (avg time, success rate, data read/written)

Completed

  • add bwlimit option for each rsync process
  • write process manager
  • save state (last attempted time, last attempted status) for each repo, and restore state on startup (e.g. use JSON/INI file for each repo)
  • calculate difference between the scheduled time of a job and the time at which it actually ran; log this
  • add all repos to merlin-config.ini (*)
  • handle termination signals in merlin (SIGINT, SIGTERM); close stopChan for this
  • listen on Unix socket in merlin
  • implement arthur.go (commands: sync and status)
  • allow dynamic reloading in merlin
  • use separate log file for each child process (currently sharing stdout/stderr with parent)
  • implement zfssync in merlin (just invoke the existing Python script)