continuous-integration/drone/push Build is passing
Details
Add an API for members to create their own virtual hosts. Co-authored-by: Max Erenberg <> Reviewed-on: #35 Co-authored-by: Max Erenberg <merenber@csclub.uwaterloo.ca> Co-committed-by: Max Erenberg <merenber@csclub.uwaterloo.ca> |
||
---|---|---|
.drone | ||
ceo | ||
ceo_common | ||
ceod | ||
debian | ||
docs | ||
etc | ||
one_time_scripts | ||
tests | ||
.drone.yml | ||
.gitignore | ||
MANIFEST.in | ||
Makefile | ||
README.md | ||
VERSION.txt | ||
clear_cache.sh | ||
dev-requirements.txt | ||
docker-compose.yml | ||
docker-entrypoint.sh | ||
requirements.txt | ||
setup.cfg | ||
setup.py |
README.md
pyceo
CEO (CSC Electronic Office) is the tool used by CSC to manage club accounts and memberships. See docs/architecture.md for an overview of its architecture.
The API documentation is available as a plain HTML file in docs/redoc-static.html.
Development
Docker
If you are not modifying code related to email or Mailman, then you may use Docker containers instead, which are much easier to work with than the VM.
First, make sure you create the virtualenv:
docker run --rm -v "$PWD:$PWD" -w "$PWD" -u $(id -u):$(id -g) python:3.7-buster \
sh -c 'python -m venv venv && . venv/bin/activate && pip install -r requirements.txt -r dev-requirements.txt'
Then bring up the containers:
docker-compose up -d # or without -d to run in the foreground
This will create some containers with the bare minimum necessary for ceod to run, and start ceod on each of phosphoric-acid, mail, and coffee container. You can check the containers status using:
docker-compose logs -f
To use ceo, run the following:
docker-compose exec phosphoric-acid bash
su ctdalek
. venv/bin/activate
python -m ceo # the password is krb5
This should bring up the TUI.
Normally, ceod should autoamtically restart when the source files are changed. To manually restart the service, run:
docker-compose kill -s SIGHUP phosphoric-acid
To stop the containers, run:
docker-compose down
Alternatively, if you started docker-compose in the foreground, just press Ctrl-C.
VM
If you need the full environment running in VM, follow the guide on syscom dev environment. This will setup all of the services needed for ceo to work. You should clone this repo in the phosphoric-acid container under ctdalek's home directory; you will then be able to access it from any container thanks to NFS.
Once you have the dev environment setup, there are a few more steps you'll need to do for ceo.
Kerberos principals
First, you'll need ceod/<hostname>
principals for each of phosphoric-acid,
coffee and mail. (coffee is taking over the role of caffeine for the DB
endpoints). For example, in the phosphoric-acid container:
kadmin -p sysadmin/admin
<password is krb5>
addprinc -randkey ceod/phosphoric-acid.csclub.internal
ktadd ceod/phosphoric-acid.csclub.internal
Do this for coffee and mail as well. You need to actually be in the
appropriate container when running these commands, since the credentials
are being added to the local keytab.
On phosphoric-acid, you will additionally need to create a principal
called ceod/admin
(remember to addprinc and ktadd).
Database
Note: The instructions below apply to the dev environment only; in production, the DB superusers should be restricted to the host where the DB is running.
Attach to the coffee container, run mysql
, and run the following:
CREATE USER 'mysql' IDENTIFIED BY 'mysql';
GRANT ALL PRIVILEGES ON *.* TO 'mysql' WITH GRANT OPTION;
(In prod, the superuser should have '@localhost' appended to its name.)
Now open /etc/mysql/mariadb.conf.d/50-server.cnf and comment out the following line:
bind-address = 127.0.0.1
Then restart MariaDB:
systemctl restart mariadb
Install PostgreSQL in the container:
apt install -y postgresql
Modify the superuser postgres
for password authentication and restrict new users:
su postgres
psql
ALTER USER postgres WITH PASSWORD 'postgres';
REVOKE ALL ON SCHEMA public FROM public;
GRANT ALL ON SCHEMA public TO postgres;
Create a new pg_hba.conf
:
cd /etc/postgresql/<version>/<branch>/
mv pg_hba.conf pg_hba.conf.old
# new pg_hba.conf
# TYPE DATABASE USER ADDRESS METHOD
local all postgres peer
host all postgres 0.0.0.0/0 md5
local all all peer
host all all localhost md5
local sameuser all peer
host sameuser all 0.0.0.0/0 md5
Warning: in prod, the postgres user should only be allowed to connect locally, so the relevant snippet in pg_hba.conf should look something like
local all postgres md5
host all postgres localhost md5
host all postgres 0.0.0.0/0 reject
host all postgres ::/0 reject
Add the following to postgresql.conf:
listen_addresses = '*'
Now restart PostgreSQL:
systemctl restart postgresql
In prod, users can login remotely but superusers (postgres
and mysql
) are only
allowed to login from the database host.
Mailman
You should create the following mailing lists from the mail container:
/opt/mailman3/bin/mailman create syscom@csclub.internal
/opt/mailman3/bin/mailman create syscom-alerts@csclub.internal
/opt/mailman3/bin/mailman create exec@csclub.internal
/opt/mailman3/bin/mailman create ceo@csclub.internal
See https://git.uwaterloo.ca/csc/syscom-dev-environment/-/tree/master/mail for instructions on how to access the Mailman UI from your browser.
If you want to actually see the archived messages, you'll need to tweak the settings for each list from the UI so that non-member messages get accepted (by default they get held).
Dependencies
Next, install and activate a virtualenv:
sudo apt install libkrb5-dev libpq-dev python3-dev
python3 -m venv venv
. venv/bin/activate
pip install -r requirements.txt
pip install -r dev-requirements.txt
Running the application
ceod is a distributed application, with instances on different hosts offering different services. Therefore, you will need to run ceod on multiple hosts. Currently, those are phosphoric-acid, mail and caffeine (in the dev environment, caffeine is replaced by coffee).
To run ceod on a single host (as root, since the app needs to read the keytab):
export FLASK_APP=ceod.api
export FLASK_ENV=development
flask run -h 0.0.0.0 -p 9987
Sometimes changes you make in the source code don't show up while Flask
is running. Stop the flask app (Ctrl-C), run clear_cache.sh
, then
restart the app.
Interacting with the application
To use the TUI:
python -m ceo
To use the CLI:
python -m ceo --help
Alternatively, you may use curl to send HTTP requests.
ceod uses SPNEGO for authentication, and TLS for confidentiality and integrity. In development mode, TLS can be disabled.
First, make sure that your version of curl has been compiled with SPNEGO support:
curl -V
Your should see 'SPNEGO' in the 'Features' section.
Here's an example of making a request to an endpoint which writes to LDAP:
# Get a Kerberos TGT first
kinit
# Make the request
curl --negotiate -u : --service-name ceod --delegation always \
-d '{"uid":"test_1","cn":"Test One","given_name":"Test","sn":"One","program":"Math","terms":["s2021"]}' \
-X POST http://phosphoric-acid:9987/api/members
Packaging
First, I strongly recommend running the build in a Docker/Podman container to avoid screwing up your main system:
podman run -it --name pyceo-packaging -v "$PWD":"$PWD" -w "$PWD" debian:buster bash
Important: Make sure to use a container image for the same distribution which you're packaging. For example, if you're creating a package for bullseye, you should be using the debian:bullseye Docker image (this is because the virtualenv symlinks python to the OS' version of python).
Here are some of the prerequisites you'll need to build the deb files:
apt install devscripts debhelper git-buildpackage
Make sure to also install all of the packages in the 'Build-Depends' section in debian/control.
There are two important files to change before creating a new package: debian/changelog
(which can be edited by running dch -i
), and VERSION.txt.
Make sure you git commit your changes before building the packages.
To build unsigned packages:
gbp buildpackage --git-upstream-branch=master -uc -us
To build signed packages (for uploading), you need to have your GPG key ready, and it should also be in the CSC mirror keyring. Once you have done that, replace '-uc -us' by '-k<your_gpg_key_id>', e.g.
gbp buildpackage --git-upstream-branch=master -k8E5568ABB0CF96BC367806ED127923BE10DA48DC
This will create a bunch of files (deb, dsc, tar.gz, etc.) in the parent directory.
To clean the packages:
rm ../*.{xz,gz,dsc,build,buildinfo,changes,deb}
Uploading
Ask a syscom member for their dupload.conf file, and place it in your ~/.dupload.conf. Then, from a CSC machine, upload the changes file from the parent directory, e.g.
dupload ceo_1.0.0-buster1_amd64.changes