Back

Lubook Chapter 1. Setting up the Server

My first steps into my biggest project yet, a full-stack application for a comic-sharing website like Wattpad, Webtoon, etc. My first time ever setting up the server on a self-hosted VPS.

Uploaded on: Sat Nov 30 2024

Hello, this is my development process and logs of the project Lubook. (I put a GitHub link, instead of a website, etc. since there’s a chance you’re reading this after I stopped paying for the hosting services).

Motivation

As a project for the course Introduction to Software Engineering, our group decided on a manga reading website, but their version is about pirating and sharing illegally obtained mangas, by other manga artists (named Openbook). Myself, I have no problem with piracy as long as it’s for personal use, but here it is being used in a professional environment, namely an educational institution, which is why I do not support its development.

As a team member and a student, I’ll have to fulfill tasks to make sure I pass the subject, but as an artist it’s very difficult to bring myself to work on such a project without leaving snide comments within like, even in software official documents submitted to the professor, I will still leave these kind of comments. It may not be professional, but the project itself isn’t professional in the first place:

/**
 * GET /api/authors: Retrieves all authors we have stolen from.
 */
const getAuthors = asyncHandler((req, res) => ...);

Therefore, as a rather small retaliation, I’ll use this opportunity as a way to learn more, and recreate the project designed for 4 members, by myself, deciding and designing all by myself, titled Lubook (because my name is Luna and I’m uncreative, what a shame as an artist).

Prologue: Project Structure

This project is meant to be a monorepo, where the backend and the frontend portions of the application are both on the same repository. The overall tech stack was designed to be MERN by our group leader (MongoDB, ExpressJS, ReactJS, NodeJS). MongoDB is hosted by the M0 cluster on its own Atlas Cloud, but I don’t have any idea what he plans for where Node should be hosted yet, as of now, it’s only theoretical so far.

I go a little bit different to make it interesting. Even though, the core stuff is very similar, I use different technologies. I’m going to go with PostgreSQL, Vue, and same thing with Express and Node. Which I argue, is “similar” enough.

Setting up the Backend

First thing that came to mind, where to host the actual Express API, for the front end to interact with? At the concept stage, I was leaning towards NextJS with its API routes handling, or Astro API routes, and then publish it on a serverless host like Vercel. But that wouldn’t let me learn as much as what I choose to do now.

I want to keep going with Express, Postgres and Vue. So, the next option would be a VPS, and host providers like AWS EC2, Digital Ocean came to mind. I plan to keep both the Postgres and the express API on the same server, so only the express app can access the database through localhost, and open a port for the express API to accept HTTP requests.

I have never really deployed on a VPS before, so I whipped up a cheap and simple $5 VPS, setup Ubuntu, NPM, Git and PostgreSQL on it. I had some trouble with Postgres’s user postgres not accepting any passwords I want to after logging in to it. Google searches told me to go in the database, and do a ALTER USER query, but that didn’t work. I did a simple logout command, then from root, change the password of the user postgres instead. This might not be the best way to do it, but it worked.

Next thing to do would be setting up SSL, to allow HTTPS request to the express server. I used Let’s Encrypt’s certbot for generating an SSL, after pointing my domains to this VPS server. If you’re reading this while the project is still being paid for, the domain is api.lubook.club.

Funny thing, I used ufw utils to setup a firewall, and not regarding that that means that the SSH port is also blocked. I got blocked out of my VPS. Make sure to do ufw allow ssh, lol.

First success on CURL

Retracing Steps

After a whole grueling day of installing and configuring random stuff that just breaks because I don’t know how to work, an Express server has been setup correctly under the domain. In hindsight, the only problem was making sure firewall allows ssh before doing anything. Just in case someone else is having trouble, or even myself in the future, I’ll list out the steps I did.

Setup Firewalls with ufw, making sure to allow SSH (port 22), HTTP (port 80) and HTTPS (port 443). Or any other ports (5432 for Postgres) if you need remote access to database.
Install Postgres. Login to postgres, create a new role (basically a user), with a password. Create a new database for your app with SQL. Grant your user’s access to that database.
Allow connections to Postgres? For me, this was not necessary, because by default, Postgres only allows connections from localhost, and I’m hosting both Express and PSQL on same server, it doesn’t really matter. But if you split it, you need to tell Postgres to allow connections from certain IPs, or all IPs. Check this too.
1. Basically, you need to find the config file /etc/postgresql/<version>/main/postgresql.conf, and change the value of listen_addresses.
2. Edit the pg_hba.conf file, this tells how to authenticate (with passwords or something like that).
3. If you haven’t allowed port 5432 for Postgres, you want to do it now.
4. Make sure to restart the service if you change configs, obviously.
Download code: The simplest way to download the code to the VPS would be cloning git, setting up nvm/npm, and just running it like your local machine. Or you can have your local machine send over with scp (not the foundation, it meant Secure copy protocol), or rsync commands. Usually you want to also install npm and pm2 (process manager) from npm, to allow background running node processes, or it gets awkward trying to run commands with it still running.
Profits? You can try checking if your server is setup properly by having a simple /ping route, and on another machine run curl to see.
Let’s Encrypt! If you want to serve your server under HTTPS, you need to setup SSL on the machine. You can checkout certbot for a free solution to generating SSL. To allow Express or your server of choice to take the certificates, refer to that library’s documentation. For Node/Express, it is creating a server under package https, and provide the ssl_cert and ssl_key files.

  const options = {
    key: readFileSync(process.env.SSL_KEY || ""),
    cert: readFileSync(process.env.SSL_CERT || ""),
  };

  // Server for listening.
  https.createServer(options, app).listen(443, () => {
    console.log("Server started with HTTPS port 443.");
  });

  // Server to redirect back to https.
  http
    .createServer(async (req, res) => {
      res.writeHead(301, { Location: "https://" + req.headers.host + req.url });
      res.end();
    })
    .listen(80, () => {
      console.log("Redirecting server started with HTTP port 80.");
    });

Backend Testing

When using MongoDB, it was very simple to just swap real server for a mock server with mongodb-memory-server, with postgres,… it’s awkward. The best way to test this seems to be containerizing postgres, and run tests like that.

Okay, fine, after a while of messing around, I think the best way to be able to test postgres queries and controllers is with a docker container. I have tried with pg-mem for a similar version of mongodb-memory-server, but it isn’t very straight forward.

The solution here is just to install PostgreSQL on your own machine and test it yourself, there’s really no other way with how Postgres is. You can use Docker Containers also, for example on GitHub Actions. I’m currently using Actions for checking, also to keep this information for myself, here’s the config file that worked for me (I’m using Drizzle ORM, might be different for other ORMs).

  test:
    # Docker is only available on Linux runners
    runs-on: ubuntu-latest

    services:
      db:
        image: postgres:17

        # Basically, just setting up user/password/database.
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: test

        # Make sure that postgres is setup before continuing.
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

        # Maps out and portforwards ports to connect. I don't know
        # the details for this exactly. But it says "Maps TCP 5432 to Postgres"
        ports:
          - 5432:5432

    name: Run backend tests
    steps:

      # Just normal steps for CI to tests.
      - name: Checkout
        uses: actions/checkout@v4
        with:
          fetch-depth: 1
          sparse-checkout: |
            server/
          sparse-checkout-cone-mode: false

      # I use PNPM instead of NPM, isn't that different.
      - name: Setup PNPM
        uses: pnpm/action-setup@v4
        with:
          version: 9
      - name: Setup Node
        uses: actions/setup-node@v4
        with:
          node-version: latest
          cache: pnpm
          cache-dependency-path: ./server/pnpm-lock.yaml
      - name: Install dependencies
        working-directory: ./server
        run: pnpm i

      # Put the schema on the database before running tests.
      - name: Migrate schema
        working-directory: ./server
        run: pnpm drizzle-kit push
        env:
          POSTGRES_HOST: localhost
          POSTGRES_PORT: 5432
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: postgres
          POSTGRES_DATABASE: test

      # Run the tests!
      - name: Test with coverage
        working-directory: ./server
        run: pnpm test
        env:
          MODE: CI
          JWT_SECRET: ${{ secrets.JWT_SECRET }}
          POSTGRES_HOST: localhost
          POSTGRES_PORT: 5432
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: postgres
          POSTGRES_DATABASE: test

Also, as I was using ID with SERIAL type as primary key, when deleting table, make sure to use TRUNCATE TABLE table RESTART IDENTITY CASCADE;, and include RESTART IDENTITY part to make the table start back at 1, or it kept going up as you run tests on your machine, which is kinda awkward, not bad, but awkward.

Test Data

Users test data is simple enough as you can easily generate random named accounts to test with. But what about the content itself? This is when I hit a huge roadblock that I really should have thought of before but somehow i just did not. As I’m not infringing on anyone’s copyright, the only options available to me would be public domain comic books, or freely distributable works, but in a way, I still think that’s a bit awkward.

I have a few stories I wrote that might work with 1 chapter just to have some content available on the server, that’s also easier said then done. At the time of writing this, my iPad and Apple Pencil age is 2 days old, I love painting and writing, but I’m not familiar with digital painting at all. Even then, I’ll still choose to learn and adapt to digital painting (now that I can due to the birthday gift of an iPad, thanks dad!), rather than ripping off others’ work. I’m quite excited, a random project out of spite manages to become one of the biggest projects, involving so many different fields and technologies all at once.