# Week 9, 10 & 11 - CircuitVerse@GSOC'23

This blog is coming after a long time \[almost 3 weeks\].

Throughout this week, the main tasks involved were -

* Update the docker setup to make it more convenient
    
* Test the docker setup in all OS \[Linux, Mac, Windows\]
    
* Rewrite Documentation
    

#### Revamp Docker Setup

The main problems with the current setup are -

1. In the docker file, it copies all the source code and installs dependencies
    
2. The image and process inside the container run as root user
    
3. For this reason, whenever some dependencies or codebase changes the docker image rebuild and which takes lots of time
    
4. As the process runs as a root user, some file changes from the container give permission errors in a code editor.
    
5. As we install all the dependencies at the time of building the image, there is no possibility to cache the dependencies
    

**Solution -**

The solution is inspired by the **GitHub Codespaces** working principle.

**👀 How do codespaces work?**

1. It has a Dockerfile of the application
    
2. It starts the container with a `sleep infinity` to prevent it from terminating after the start
    
3. Then, it uses `attach` or `exec` to attach the container
    
4. Then, it runs a `setup.sh` to install all dependencies and prepare the4 environment
    
5. Then it runs `boot.sh` to start the application
    

**🏄 Let's do the same for CircuitVerse project**

1. Prepare the Dockerfile. This will only contain the ruby and nodejs environment. As well as it will have a non-root user with the same user id and group id of the host.
    
    We will provide host's group id and user id by build arguments
    
    ```dockerfile
    FROM ruby:3.2.1
    
    # Args
    ARG NON_ROOT_USER_ID
    ARG NON_ROOT_GROUP_ID
    ARG NON_ROOT_USERNAME
    ARG NON_ROOT_GROUPNAME
    ARG OPERATING_SYSTEM
    
    # Check mandatory args
    RUN test -n "$NON_ROOT_USER_ID"
    RUN test -n "$NON_ROOT_GROUP_ID"
    RUN test -n "$OPERATING_SYSTEM"
    RUN test -n "$NON_ROOT_USERNAME"
    RUN test -n "$NON_ROOT_GROUPNAME"
    
    # Create app directory
    RUN mkdir /circuitverse
    # Create non-root user directory
    RUN mkdir /home/${NON_ROOT_USERNAME}
    # Create non-root vendor directory
    RUN mkdir /home/vendor
    RUN mkdir /home/vendor/bundle
    # set up workdir
    WORKDIR /circuitverse
    
    # Set shell to bash
    SHELL ["/bin/bash", "-c"]
    
    # install dependencies
    RUN apt-get update -qq && \
     apt-get install -y imagemagick shared-mime-info libvips sudo make cmake netcat libnotify-dev git chromium-driver chromium --fix-missing && apt-get clean
    
    # Setup nodejs and yarn
    RUN curl -sL https://deb.nodesource.com/setup_16.x | bash \
     && apt-get update && apt-get install -y nodejs && rm -rf /var/lib/apt/lists/* \
     && curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - \
     && echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list \
     && apt-get update && apt-get install -y yarn && rm -rf /var/lib/apt/lists/*
    
    # If OPERATING_SYSTEM is Linux, create non-root user
    RUN if [[ "$OPERATING_SYSTEM" == "linux" ]]; then \
        # create non-root user with same uid:gid as host non-root user
        groupadd -g ${NON_ROOT_GROUP_ID} -r ${NON_ROOT_GROUPNAME} && useradd -u ${NON_ROOT_USER_ID} -r -g ${NON_ROOT_GROUPNAME} ${NON_ROOT_USERNAME} \
        && chown -R ${NON_ROOT_USERNAME}:${NON_ROOT_GROUPNAME} /circuitverse \
        && chown -R ${NON_ROOT_USERNAME}:${NON_ROOT_GROUPNAME} /home/${NON_ROOT_USERNAME} \
        && chown -R ${NON_ROOT_USERNAME}:${NON_ROOT_GROUPNAME} /home/vendor \
        && chown -R ${NON_ROOT_USERNAME}:${NON_ROOT_GROUPNAME} /home/vendor/bundle \
        # Provide sudo permissions to non-root user
        && adduser ${NON_ROOT_USERNAME} sudo \
        && echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers ;\
    fi
    
    # Switch to non-root user
    USER ${NON_ROOT_USERNAME}
    ```
    
    > For `macOS` and `windows` we don't need non-root users because the volume mounting is worked as shared storage. So the permission error will not come
    
2. Next, we create a `setup.sh` which will install dependencies and migrate the database
    
    ```bash
    # !/bin/sh
    
    # Remove tmp folder
    rm -rf /circuitverse/tmp
    
    # Install ruby dependencies
    gem install bundler
    bundle config set --local without "production"
    bundle config set --local path "/home/vendor/bundle"
    bundle install
    # Install node dependencies
    yarn
    # Setup database
    bundle exec rails db:create
    bundle exec rails db:schema:load
    bundle exec rails db:migrate
    bundle exec rails db:seed
    # generate key-pair for jwt-auth
    # if private.pem and public.pem does not exists
    if [ ! -f "/circuitverse/config/private.pem" ] && [ ! -f "/circuitverse/config/public.pem" ]; then
      openssl genrsa -out /circuitverse/config/private.pem 2048
      openssl rsa -in /circuitverse/config/private.pem -outform PEM -pubout -out /circuitverse/config/public.pem
    fi
    ```
    
3. Then we create a `boot.sh` to initiate `setup.sh` and then start the server
    
    ```bash
    #!/bin/bash
    
    # Delete server.pid if it exists
    rm -f /circuitverse/tmp/pids/server.pid 2>&1
    
    # Run setup
    echo "Setup project"
    ./bin/docker/setup
    
    if [[ "$OPERATING_SYSTEM" == "linux" || "$OPERATING_SYSTEM" == "mac" ]]; then
      if [ ! -d "${HOST_CURRENT_DIRECTORY%/*}" ]; then
        sudo mkdir -p "${HOST_CURRENT_DIRECTORY%/*}"
      fi
      # Setup symbolic link for solargraph
      if [ ! -L "$HOST_CURRENT_DIRECTORY" ]; then
        sudo ln -s -T /circuitverse "$HOST_CURRENT_DIRECTORY"
      fi
      # Start solargraph server in background
      bundle exec solargraph socket --host="0.0.0.0" --port=3002 &> /dev/null &
    fi
    
    # Start server
    ./bin/dev
    
    # Start bash if previous command exits
    /bin/bash
    ```
    
4. Now, we need to prepare the `docker-compose.yml`. As you can see still now we haven't copied the source code in the image or anywhere.
    
    We will mount the codebase to the container at **/circuitverse** location to make it easy to edit files and keep all files in sync
    
    ```dockerfile
    ....
      web:
        build:
          context: .
          dockerfile: Dockerfile
          args:
            - NON_ROOT_USER_ID=${CURRENT_UID}
            - NON_ROOT_GROUP_ID=${CURRENT_GID}
            - OPERATING_SYSTEM=${OPERATING_SYSTEM}
            - NON_ROOT_USERNAME=${NON_ROOT_USERNAME}
            - NON_ROOT_GROUPNAME=${NON_ROOT_GROUPNAME}
        command: sleep infinity
        volumes:
          - .:/circuitverse:rw
          - ./config/database.docker.yml:/circuitverse/config/database.yml:rw
          - ruby_bundle:/home/vendor/bundle:rw
        cap_add:
          - SYS_ADMIN
        ports:
          - "3000:3000"
          - "3001:3001"
          - "3002:3002"
          - "3035:3035"
          - "3036:3036"
        depends_on:
          - db
          - redis
        environment:
          REDIS_URL: "redis://redis:6379/0"
          CIRCUITVERSE_USE_SOLR: "false"
          DOCKER_ENVIRONMENT: "true"
          NODE_ENV: "development"
          HOST_CURRENT_DIRECTORY: $PWD
          OPERATING_SYSTEM: $OPERATING_SYSTEM
    ....
    ```
    
5. Everything is ready. But there is an issue, we can't run this using `docker-compose up`. We need to set the `NON_ROOT_USER_ID` `NON_ROOT_GROUP_ID` `OPERATING_SYSTEM` environment variables. Also, need to start the server by attaching to it. So write a script for that
    
    ```dockerfile
    #!/bin/bash
    
    # Detect operating system [linux, macos] with uname
    DETECTED_OS=$(uname -s | tr '[:upper:]' '[:lower:]')
    
    # If operating system is linux
    if [ "$DETECTED_OS" = "linux" ]; then
      # Set environment variables temporary
      export CURRENT_GID=$(id -g)
      export CURRENT_UID=$(id -u)
      export NON_ROOT_USERNAME="user"
      export NON_ROOT_GROUPNAME="user"
      export OPERATING_SYSTEM="linux"
      # Check if docker and docker compose are installed
      if ! command -v docker &>/dev/null; then
        echo "Docker is not installed. Install docker : https://docs.docker.com/engine/install/"
        exit 0
      fi
      # Check if rootless docker is available
      if docker ps &>/dev/null; then
        # Run docker-compose up
        docker compose up -d --build
        # Run docker-compose exec web bash
        docker compose exec web bin/docker/boot
        # Run docker-compose down
        docker compose down
      else
        # Run docker-compose up as root user
        sudo docker compose up -d --build
        # Run docker-compose exec web bash as root user
        sudo docker compose exec web bin/docker/boot
        # Run docker-compose down as root user
        sudo docker compose down
      fi
    fi
    
    # If operating system is macos
    if [ "$DETECTED_OS" = "darwin" ]; then
      # Set environment variables temporary
      export CURRENT_GID="0"
      export CURRENT_UID="0"
      export NON_ROOT_USERNAME="root"
      export NON_ROOT_GROUPNAME="root"
      export OPERATING_SYSTEM="mac"
    
      # Check if docker and docker compose are installed
      if ! command -v docker &>/dev/null; then
        echo "Docker is not installed. Install docker : https://docs.docker.com/engine/install/"
        exit 0
      fi
    
      # Run docker-compose up
      docker compose up -d --build
      # Run docker-compose exec web bash
      docker compose exec web bin/docker/boot
      # Run docker-compose down
      docker compose down
    fi
    ```
    
    This one script is responsible to fetch user id and group id and set temporary environment variables.
    
    This will start the container and then run `bin/docker/boot` inside the container. And after the container got exited, it run `docker compose down` to stop other containers as well.
    
6. 💁 That's all.
    

#### Test docker setup

We have tested this docker setup in

* Windows \[both Hyper-V and WSL2\]
    
* Linux
    
* MacOS
    

It works on all os without any issues.

For Windows, we have written a PowerShell script to support it as well.

```powershell
# Set environment variables
$env:NON_ROOT_USERNAME = "root";
$env:NON_ROOT_GROUPNAME = "root";
$env:OPERATING_SYSTEM = "windows";
$env:PWD = (Get-Location).Path;
$env:CURRENT_UID = "0";
$env:CURRENT_GID = "0";

# Check if docker is available
if (Get-Command -Name "docker" -ErrorAction SilentlyContinue) {
    # Run docker-compose up
    docker compose up -d --build
    # Run docker-compose exec to boot the image
    docker compose exec web bin/docker/boot
    # Run docker-compose down
    docker compose down
}
else {
    Write-Output "Docker is not available. Follow this documentation to install docker: https://docs.docker.com/desktop/install/windows-install/"
}
```

You can get more details in this PR - [https://github.com/CircuitVerse/CircuitVerse/pull/3913/](https://github.com/CircuitVerse/CircuitVerse/pull/3913/)

#### Rewrite Documentation

Many changes were made to the project for improving the development experience. So it's very important to update the documentation as well.

We have written the documentation for Remote Development Platforms, as well as separate documentation for each operating system for setting up a docker-based environment + local installation.

**Additionally, working on the reviewed PRs. Making some changes on the PR to make it ready to merge.**

---

That's all for this blog. If you liked this subscribe to the newsletter.

---

🎉 Finally, my project `SwiftWave` has been published on GitHub.

SwiftWave is a self-hosted lightweight PaaS solution to deploy and manage your applications on any VPS without any hassle

**GitHub Link** \- [https://github.com/swiftwave-org/swiftwave](https://github.com/swiftwave-org/swiftwave)

If you like the initiative, star⭐ the project on GitHub.

---