69 lines
2.1 KiB
Markdown
69 lines
2.1 KiB
Markdown
|
# leylines
|
||
|
|
|
||
|
|
this repo enables managing a [dask](https://dask.org) cluster using
|
||
|
|
[wireguard](https://www.wireguard.com/) to link nodes which may be separated by WAN[^1] and includes
|
||
|
|
an opinionated mini wireguard manager (on the server side, workers use wg-quick) that doubles as an
|
||
|
|
[ansible](https://www.ansible.com/) inventory plugin. finally, ansible playbooks can run setup and
|
||
|
|
deployment for dask nodes
|
||
|
|
|
||
|
|
## how to
|
||
|
|
|
||
|
|
```bash
|
||
|
|
(cd leylines-monocypher && pip3 install --user .)
|
||
|
|
(cd leylines && pip3 install --user .)
|
||
|
|
mkdir -p ~/.config/leylines
|
||
|
|
leylines init -n myserver -i 1.2.3.4 -k path/to/ssh-key
|
||
|
|
leylines add -n worker-0 -k path/to/ssh-key
|
||
|
|
...
|
||
|
|
leylines add -n worker-n -k path/to/ssh-key
|
||
|
|
```
|
||
|
|
|
||
|
|
optionally copy the database to your laptop so you can run ansible locally (there will be some
|
||
|
|
actual API soon but not right now)
|
||
|
|
|
||
|
|
start a privileged shell (there is no service for the wireguard stuff yet -- coming soon)
|
||
|
|
```bash
|
||
|
|
systemd-run -tS --uid $(id -u) --gid $(id -g) -pAmbientCapabilities=CAP_NET_ADMIN
|
||
|
|
```
|
||
|
|
|
||
|
|
sync wireguard settings
|
||
|
|
```bash
|
||
|
|
leylines sync
|
||
|
|
```
|
||
|
|
|
||
|
|
get status
|
||
|
|
```bash
|
||
|
|
leylines status
|
||
|
|
```
|
||
|
|
|
||
|
|
get config for a node
|
||
|
|
```bash
|
||
|
|
leylines get-conf <id>
|
||
|
|
```
|
||
|
|
|
||
|
|
manually copy that config to your worker node, `/etc/wireguard/leyline-wg.conf` and then
|
||
|
|
`systemctl enable --now wg-quick@leyline-wg`
|
||
|
|
|
||
|
|
currently the wireguard topology is a star. this doesn't actually work optimally for my config,
|
||
|
|
where some nodes are colocated and should have direct connections to each other and others should go
|
||
|
|
over WAN to reach distant nodes. this will be changed (you may be sensing a pattern with the amount
|
||
|
|
of TODO)
|
||
|
|
|
||
|
|
run the ansible playbook
|
||
|
|
```bash
|
||
|
|
cd leylines-ansible
|
||
|
|
ansible-playbook -i leylines_inv.py playbook-setup.yml
|
||
|
|
```
|
||
|
|
|
||
|
|
the first run will take a while. it builds python 3.9.5 and installs it, then builds a virtualenv
|
||
|
|
with python dependencies in it, and then installs and starts systemd user services for the scheduler
|
||
|
|
and workers
|
||
|
|
|
||
|
|
now you can open `<your server's wireguard ip>:31336` to view the dask dashboard
|
||
|
|
|
||
|
|
use the cluster with
|
||
|
|
```python
|
||
|
|
from dask.distributed import Client
|
||
|
|
client = Client("<your server's wireguard ip>:31337")
|
||
|
|
```
|