Python CSC Electronic Office
Go to file
Max Erenberg f84965c8e1
continuous-integration/drone/push Build is passing Details
reload all NGINX servers after adding a vhost (#90)
Currently, only the NGINX server on biloba is reloaded after adding a new vhost or renewing an SSL certificate. The NGINX server on chamomile should also be reloaded, since chamomile is a warm standby for biloba.

This PR adds a new config option in ceod.ini to specify the shell command to reload the web servers.

Reviewed-on: #90
Co-authored-by: Max Erenberg <merenber@csclub.uwaterloo.ca>
Co-committed-by: Max Erenberg <merenber@csclub.uwaterloo.ca>
2023-01-22 17:20:55 -05:00
.drone use bullseye for base container (#91) 2023-01-22 17:15:40 -05:00
ceo Show groups in member for API, CLI and TUI (#82) 2022-11-26 20:09:05 -05:00
ceo_common Use the admin creds in the HTTPClient when necessary (#85) 2022-11-06 15:23:27 -05:00
ceod reload all NGINX servers after adding a vhost (#90) 2023-01-22 17:20:55 -05:00
debian merenber signs the packages 2022-10-23 22:04:32 -04:00
docs Show groups in member for API, CLI and TUI (#82) 2022-11-26 20:09:05 -05:00
etc reload all NGINX servers after adding a vhost (#90) 2023-01-22 17:20:55 -05:00
one_time_scripts Unsubscribe/resubscribe members when they're expired and renewed (#53) 2022-06-02 02:06:49 -04:00
tests reload all NGINX servers after adding a vhost (#90) 2023-01-22 17:20:55 -05:00
.drone.yml use bullseye for base container (#91) 2023-01-22 17:15:40 -05:00
.gitignore Revert "Simplify packaging" 2022-10-23 22:00:48 -04:00
MANIFEST.in include Jinja templates in MANIFEST.in 2021-11-02 01:53:53 -04:00
Makefile Revert "Simplify packaging" 2022-10-23 22:00:48 -04:00
PACKAGING.md Revert "Simplify packaging" 2022-10-23 22:00:48 -04:00
README.md use bullseye for base container (#91) 2023-01-22 17:15:40 -05:00
VERSION.txt Release 1.0.24 2022-10-23 19:50:38 -04:00
clear_cache.sh Add debian packaging (#32) 2021-10-28 20:52:19 -04:00
dev-requirements.txt Add support for using number in member terms renwewal API (#77) 2022-10-07 07:58:23 -04:00
docker-compose.yml use bullseye for base container (#91) 2023-01-22 17:15:40 -05:00
docker-entrypoint.sh add option to use Docker instead of VM (#16) 2021-09-17 22:39:27 -04:00
requirements.txt Disable inactive club sites (#68) 2022-07-22 23:51:59 -04:00
setup.cfg implement renewal reminders (#61) 2022-06-30 20:02:06 -04:00
setup.py use port 465 for smtps 2021-11-02 02:46:05 -04:00

README.md

pyceo

Build Status

Main TUI view

CEO (CSC Electronic Office) is the tool used by CSC to manage club accounts and memberships. See docs/architecture.md for an overview of its architecture.

The API documentation is available as a plain HTML file in docs/redoc-static.html.

Development

Docker

If you are not modifying code related to email or Mailman, then you may use Docker containers instead, which are much easier to work with than the VM.

First, make sure you create the virtualenv:

docker run --rm -v "$PWD:$PWD:z" -w "$PWD" python:3.9-bullseye sh -c 'apt update && apt install -y libaugeas0 && python -m venv venv && . venv/bin/activate && pip install -r requirements.txt -r dev-requirements.txt'

Then bring up the containers:

docker-compose up -d  # or without -d to run in the foreground

This will create some containers with the bare minimum necessary for ceod to run, and start ceod on each of phosphoric-acid, mail, and coffee container. You can check the containers status using:

docker-compose logs -f

To use ceo, run the following:

docker-compose exec phosphoric-acid bash
su ctdalek
. venv/bin/activate
python -m ceo  # the password is krb5

This should bring up the TUI.

Normally, ceod should autoamtically restart when the source files are changed. To manually restart the service, run:

docker-compose kill -s SIGHUP phosphoric-acid

To stop the containers, run:

docker-compose down

Alternatively, if you started docker-compose in the foreground, just press Ctrl-C.

VM

If you need the full environment running in VM, follow the guide on syscom dev environment. This will setup all of the services needed for ceo to work. You should clone this repo in the phosphoric-acid container under ctdalek's home directory; you will then be able to access it from any container thanks to NFS.

Once you have the dev environment setup, there are a few more steps you'll need to do for ceo.

Kerberos principals

First, you'll need ceod/<hostname> principals for each of phosphoric-acid, coffee and mail. (coffee is taking over the role of caffeine for the DB endpoints). For example, in the phosphoric-acid container:

kadmin -p sysadmin/admin
<password is krb5>
addprinc -randkey ceod/phosphoric-acid.csclub.internal
ktadd ceod/phosphoric-acid.csclub.internal

Do this for coffee and mail as well. You need to actually be in the appropriate container when running these commands, since the credentials are being added to the local keytab. On phosphoric-acid, you will additionally need to create a principal called ceod/admin (remember to addprinc and ktadd).

Database

Note: The instructions below apply to the dev environment only; in production, the DB superusers should be restricted to the host where the DB is running.

Attach to the coffee container, run mysql, and run the following:

CREATE USER 'mysql' IDENTIFIED BY 'mysql';
GRANT ALL PRIVILEGES ON *.* TO 'mysql' WITH GRANT OPTION;

(In prod, the superuser should have '@localhost' appended to its name.)

Now open /etc/mysql/mariadb.conf.d/50-server.cnf and comment out the following line:

bind-address = 127.0.0.1

Then restart MariaDB:

systemctl restart mariadb

Install PostgreSQL in the container:

apt install -y postgresql

Modify the superuser postgres for password authentication and restrict new users:

su postgres
psql

ALTER USER postgres WITH PASSWORD 'postgres';
REVOKE ALL ON SCHEMA public FROM public;
GRANT ALL ON SCHEMA public TO postgres;

Create a new pg_hba.conf:

cd /etc/postgresql/<version>/<branch>/
mv pg_hba.conf pg_hba.conf.old
# new pg_hba.conf
# TYPE  DATABASE        USER            ADDRESS                 METHOD
local   all             postgres                                peer
host    all             postgres        0.0.0.0/0               md5

local   all             all                                     peer
host    all             all             localhost               md5

local   sameuser        all                                     peer
host    sameuser        all             0.0.0.0/0               md5

Warning: in prod, the postgres user should only be allowed to connect locally, so the relevant snippet in pg_hba.conf should look something like

local   all             postgres                                md5
host    all             postgres        localhost               md5
host    all             postgres        0.0.0.0/0               reject
host    all             postgres        ::/0                    reject

Add the following to postgresql.conf:

listen_addresses = '*'

Now restart PostgreSQL:

systemctl restart postgresql

In prod, users can login remotely but superusers (postgres and mysql) are only allowed to login from the database host.

Mailman

You should create the following mailing lists from the mail container:

/opt/mailman3/bin/mailman create syscom@csclub.internal
/opt/mailman3/bin/mailman create syscom-alerts@csclub.internal
/opt/mailman3/bin/mailman create exec@csclub.internal
/opt/mailman3/bin/mailman create ceo@csclub.internal

See https://git.uwaterloo.ca/csc/syscom-dev-environment/-/tree/master/mail for instructions on how to access the Mailman UI from your browser.

If you want to actually see the archived messages, you'll need to tweak the settings for each list from the UI so that non-member messages get accepted (by default they get held).

Dependencies

Next, install and activate a virtualenv:

sudo apt install libkrb5-dev libpq-dev python3-dev
python3 -m venv venv
. venv/bin/activate
pip install -r requirements.txt
pip install -r dev-requirements.txt

Running the application

ceod is a distributed application, with instances on different hosts offering different services. Therefore, you will need to run ceod on multiple hosts. Currently, those are phosphoric-acid, mail and caffeine (in the dev environment, caffeine is replaced by coffee).

To run ceod on a single host (as root, since the app needs to read the keytab):

export FLASK_APP=ceod.api
export FLASK_ENV=development
flask run -h 0.0.0.0 -p 9987

Sometimes changes you make in the source code don't show up while Flask is running. Stop the flask app (Ctrl-C), run clear_cache.sh, then restart the app.

Interacting with the application

To use the TUI:

python -m ceo

To use the CLI:

python -m ceo --help

Alternatively, you may use curl to send HTTP requests.

ceod uses SPNEGO for authentication, and TLS for confidentiality and integrity. In development mode, TLS can be disabled.

First, make sure that your version of curl has been compiled with SPNEGO support:

curl -V

Your should see 'SPNEGO' in the 'Features' section.

Here's an example of making a request to add a user (in the Docker container):

# If you're root, switch to the ctdalek user first
su ctdalek
# Get a Kerberos TGT (password is krb5)
kinit
# Make the request
curl --negotiate -u : --service-name ceod --delegation always \
    -d '{"uid":"test_1","cn":"Test One","given_name":"Test","sn":"One","program":"Math","terms":["s2021"]}' \
    -X POST http://phosphoric-acid:9987/api/members

# To delete the user:
curl --negotiate -u : --service-name ceod --delegation always \
    -X DELETE http://phosphoric-acid:9987/api/members/test_1

# In prod, use the following base URL instead:
# https://phosphoric-acid.csclub.uwaterloo.ca:9987

Packaging

See PACKAGING.md.