Proof Of Concepts

I migrated my Linode instance to Hetzner. During the migration, I realized most of the services I'm self-hosting are no longer useful to me. At the same time, AI tooling in my workplace has opened my eyes to the possibility of AI-assisted development. This is a guide for what I did to set up Hermes on a Hetzner VPS, and how I plan to use it for vibe-coding and rearchitecting my toy-application, a bill-spitting Telegram application called Nanasplits. It may be scant on details, but it should put all the resources in one place for anyone who wants to do something similar. # Setting up Hetzner ## Buying a shared server Hetzner currently has the cheapest option for a VPS for its specs. For my current setup, 2 vCPU and 4GB of RAM with 40GB of storage costs me €5.59/month. Selection options to take note of: - Image: Went with Ubuntu 24.04 for a batteries-included experience, but would have gone with something more efficient Fedora next time. - Networking: Public IPv4 is a **must**. [Cloning from Github require IPv4 addresses.](https://github.com/orgs/community/discussions/151477) - SSH Keys: Set up SSH keys and use your terminal to access the VPS. Hetzner's web console is absolute atrocious and pasting from clipboard mangles the paste content. It will be hell to just log in. - Firewall: There are 2 layers of firewall, Hetzner's own and Ubuntu's UFW. We will set up and duplicate the rules on both firewalls; Hetzner's has the benefit of blocking traffic before it reaches your VPS, and we use UFW as a baseline. ## Firewall rules Firstly, we need to set up the firewall rules on Hetzner's dashboard to allow incoming traffic on the necessary ports. In case this is not present as the default, you will minimally need: - Port 22, ICMP: This lets us SSH into the server. Subsequently, we will set up other rules for Tailscale and HTTPS, to be covered later. ## User setup Assuming the VPS has started up, `ssh` into the server as the root user. ```bash ssh root@<IP_ADDRESS> ``` Once in, do an update of the system. We shall also install some necessary utilities. ```bash apt update && apt upgrade -y apt install -y curl git ufw unzip ``` ### Create and run Hermes as a non-root user It is generally a good practice to run applications as a non-root user for security reasons. ```bash adduser hermes usermod -aG sudo hermes ``` Copy your SSH public key to the new user's `authorized_keys` file to allow SSH access. ```bash cp ~/.ssh/authorized_keys /home/hermes/.ssh/authorized_keys chown -R hermes:hermes /home/hermes/.ssh/authorized_keys chmod 600 /home/hermes/.ssh/authorized_keys ``` Now, test that you can log in as the new user. ```bash ssh hermes@<IP_ADDRESS> ``` ## Firewall setup ### Tailscale private VPN Instead of exposing SSH to the public, we shall set up a private VPN using Tailscale. This means that only devices authenticated to the Tailscale network can access the VPS; more security for us. Follow the [official installation guide](https://tailscale.com/docs/install/cloud/hetzner) for Hetzner > Note that this adds a firewall rule to allow UDP port 41641 ## Hermes installation Follow the [official installation guide](https://hermes-agent.nousresearch.com/docs/getting-started/installation) to install Hermes on the VPS. Straightforward, nothing much to add here. ### Use on telegram The [official instructions](https://hermes-agent.nousresearch.com/docs/user-guide/messaging/telegram) will set you up for using Hermes on Telegram. > One thing to note is that you will need to **turn off** privacy mode in bot settings, but enable **require_mention** in the Hermes config, in order to trigger the bot with @bot mentions in a group chat. We can keep the long polling setup, as using the webhook entails exposing a public endpoint. I found this unnecessary, but you will need to make changes the to Caddy and Cloudflare DNS setup later to enable this. ## Public facing web app In order to serve the official and development version of the Telegram mini-app, we need to provide a public-facing endpoint. We should have a lone instance of the development version at some port (3000), and the official version will run as multiple instances as a systemd service with PM2, and be reverse-proxied by Caddy. The reason to have multiple instances of the production app is to able to do zero-downtime deployment, and also to maintain availability of the service in case one instance crashes. ### Cloudflare DNS I set up a subdomain `nanasplits.ding.gg` that points to the VPS' public IP address. For TLS, we will use Cloudflare's free SSL certificate, and set the SSL/TLS mode to "Full (strict)". This means that the traffic between Cloudflare and the VPS will also be encrypted. In Hetzner's firewall, allow incoming traffic on port 443 only for HTTPS. ### Caddy reverse proxy With Ubuntu, installing Caddy via `apt` will automatically set up Caddy as a systemd service. Simply edit the Caddyfile to add a new site configuration for `nanasplits.ding.gg` that reverse proxies to the production app instances. ```caddy frame="code" title="/etc/caddy/Caddyfile" # Simple test endpoint to verify the server and Caddy is up and running. ping.ding.gg { respond "PONG" 200 } # Refer to the Caddy docs for more information: # https://caddyserver.com/docs/caddyfile # production app nanasplits.ding.gg { reverse_proxy [::1]:3001 [::1]:3002 [::1]:3003 { lb_policy round_robin } } # development app nanasplitsdev.ding.gg { reverse_proxy [::1]:3000 } ``` > Localhost addresses are in IPV6 format, hence the `[::1]` since Hetzner VPS do not come with an IPV4 loopback interface. # Setting up a public-facing web service The goal I want to achieve is to be able to guide the development of _**and test**_ new features on Nanasplits with Hermes from my phone, while I'm out and about. The idea goes something like this: 1. dev server is exposed to the public internet and accessed via Telegram 2. Hermes makes changes to the local copy of codebase. 3. Dev version of Telegram mini-app gets hot-reloaded with the new code, and I can test the changes on the spot. 4. Iterate and repeat. Once the feature is ready, I can then merge the code to master. Build is done on the VPS and deployment is just a `systemctl reload` of the `./dist` folder. Essentially the VPS instance is the source of truth while Github is just a mirror. ##

An idea popped into my head while checking out this search solution for statically generated sites: [PageFind](https://pagefind.app/). Pagefind works by building an index of your site's content at build time, and then uses that index to perform searches on the client side. Data bandwidth is saved by using an ordered index, so that only a portion of the index needs to be downloaded, based on metadata. This inspired me to try building a static vector search solution. I have also been meaning to try out running gpu accelerated code in the browser. For large sites, parallelization and gpu acceleration will greatly enhance the user experience, if it is possible. ## Sketch There are quite a few Approximate Nearest Neighbor (ANN) algorithms. But the one that seems to be the most popular is HNSW, which is basically a skip-list in higher dimensions. Unfortunately, each layer strictly increases in size, so there is no way to save bandwidth by only downloading a portion of the index with the traditional algorithm. Idea: For each layer, each node in a layer maps to a distinct layer which contains the neighbourhood of the node in the previous layer. This allows us to prune the search space by fixing the size of each layer, akin to a B+ tree, but for higher dimensions. This is also amenable to parallelization, as all the nodes in a layer can be compared to the query in parallel. There are also some build time optimizations that are possible with parallelization, but this will be fleshed out in the future. ### Evaluation I think the main metric would be recall and precision compared to IVF-PQ, followed by latency and bandwidth usage. ## Does it even make sense? Generating the embedding for your search term is already going to require a model or an external call. What is the point of the static vector search, if you need a server to handle the embedding generation? You might as well just do the search on the server side as well, and return the results. Calling a managed external service also doesn't make sense since you will be shipping your API key to the client! The alternative is to run the embedding on the client. The smallest model I have seen is > 600MB, which completely negates the bandwidth optimizations of the chunked index. However, I believe that we will reach a point where on-device models become more ubiquitous, which makes the idea feasible then. We see some direction of this with Apple's [Foundation Models Framework](https://developer.apple.com/documentation/FoundationModels) and Google's [Built-in AI](https://developer.chrome.com/docs/ai/built-in). This shall be an early proof of concept.