mirror/merlin

Merlin

This folder contains the code for merlin (which does the actual syncing) and arthur (which sends commands to merlin).

Check out the the mirror env for a testing environment

Usage

go build merlin.go

Then configure merlin-config.ini and run using ./merlin

Nice Features To Add

• detect if an rsync process is stuck (watch the stdout/stderr of the rsync processes)
• detect if config file is changed and automatically reload (fsnotify)
• sort arthur status by last time synced
• respond to arthur without blocking main thread
• state files appear to be getting extra newlines
• log rotation for log files (also maybe state reporting to prometheus)
• split off arthur with a more featureful arthur for debugging
  • last sync runtime, time until next sync, due to sync (true/false for repo would sync if could)
  • get expected rsync command or repo config
  • details for last 10 syncs (avg time, success rate, data read/written)

Completed

• add bwlimit option for each rsync process
• write process manager
• save state (last attempted time, last attempted status) for each repo, and restore state on startup (e.g. use JSON/INI file for each repo)
• calculate difference between the scheduled time of a job and the time at which it actually ran; log this
• add all repos to merlin-config.ini (*)
• handle termination signals in merlin (SIGINT, SIGTERM); close stopChan for this
• listen on Unix socket in merlin
• implement arthur.go (commands: sync and status)
• allow dynamic reloading in merlin
• use separate log file for each child process (currently sharing stdout/stderr with parent)
• implement zfssync in merlin (just invoke the existing Python script)