Intro: What is this?

This is my work in progress repo for building what is basically PoeStack v2. I started building PoeStack a year ago and I've learned a lot of things and the v1 has over 100k users. While the v1 has been really successful and accomplished pretty much everything I set out to solve it has also exposed further issues both in the poe tool space and within the project that the v1 was not designed to tackle. I've experimented a lot with how these challenges could be overcome in the v1 and my work trying has further cemeted the idea that some high level changes are needed to take the project to its next stage.

V1 Problems and Solutions for Sage:

Here I'll lay out the problems that the V1 has ran into and the solutions that I plan for Sage to offer going forward.

- Server Costs

Problem:

This is by far the biggest issue with the V1 but it's probably not an issue the way most people would initially imagine. At the time of writing this the v1 full cost to operate is about $650 per month this is totally fine as the v1 makes about $1k per month ($850 Patreon + $150 ads). So what's the issue? Well this cost is pretty low for the size of the app with 100k users and about 20k DAU the project needs to stay highly optimized to keep costs this low. The users are also very active and site is computationally heavy compared to an average site. These optimizations are fine and they work but they make development complicated and each additional feature needs a lot of thought put into how it will work while conserving resources. This makes it hard to add new features quickly at this point and in many ways limits the types of features that could be added.

Solution:

While the site is computationally heavy on the server this is because there are thousands of people using it most of the time. The work required for any one user is actually pretty small and thanks to a large redesign of my pricing models I now believe it is possible for this work to happen on a desktop app. By moving the v2 from a website to a Electron app most of the work could take place on a users computer instead of PoeStacks servers. This would massively reduce the cost of operating PoeStack and would allow for a much wider array of features with much less concern for tiny CPU/memory/storage operations that only matter with thousands of users calculations need to occur server side.

- Complexity for Community Developers

Problem:

Poe is blessed to have a community of people so ready to contribute their time to community tools. PoeStack was designed and built to be open source with the intention that people who aren't me could easily contribute features that they were passionate about. Since so much of the v1's code was server side this added a lot of complexity on top of the issues described above made it very hard for for people who don't have access to PoeStack's dev/prod env to run the server and thus be able to develop a full feature slice (server/api/frontend). While many people have shown interest and many have contributed code to the v1 I believe this limitation has hampered the ability for contributors to truly "own" a section or feature reducing their desire to get deeper into development.

Solution:

By moving to a desktop app for most of the features running the code without access to dev/prod will be MUCH easier. I've already started designing the code in the repo to be runnable from either DBs/GGG-api or from local data files. I've started setting up exports of the DBs that combined with Docker files should allow the now very small amount of server code needed to be spun up locally. Finally inspired by Runelite I've started designing the desktop app to be plugin based. This will allow a core set of plugins (stash-view, tft bulk tools, tft live listings, ect) to be built in this repo but developers and users will be free to develop/run/publish/install plugins without my assistance. This model has worked great fro Runelite and I really believe it can work even better in our community given how amazing the tool developers here are.

//TODO there are more things to go over soon tm.

Roadmap

For now I'll just keep this as a general todo list.

Steps needed to enter pre-alpha (build only):

  • Setup user profile/authentication
  • Add leagues, characters, profile to the api
  • Add Client.txt event tailing
  • Add PoeStack pricing info to the api
  • Add top/bottom bar to the UI
  • Add visible version to the UI
  • Switch to icons only in the sidebar
  • Add a general guard panel for ensuring connection
  • Test the Windows build

Steps needed to enter alpha (releases):

  • Figure out how our Sqlite implementation should be used by scripts
  • Move these docs to mdBook
  • Make the scroll bars on windows look better
  • Remove the Window top bar and add a better looking one
  • Add automatic updating for install versions
  • Add a button to the v1 to make authentication not require copy/paste
  • Add a basic stash tracking/character tracking plugin
  • Add a plugin management plugin to enabled/disable and install/uninstall/update plugins
  • Work more on theming, add a theme selector
  • Test the Windows installer

Steps needed to enter beta:

  • Add ability to connect a Discord account
  • Feature parity for stash tracking plugin
  • Setup new TFT integration on the backend
  • Add TFT integration to the api
  • Add TFT live search/bulk tool plugins

Long term ideas

  • Add API for sending users Discord messages
  • Add API for interacting with local item filters, it would be nice to have a way to enable filters to automatically adjust to econmoic data or active account
  • Add overlay API to allow plugins to overlay POE
  • Add API for interacting with locally installed POB

Technology/Design Choices

This document aims to explain why I made certain technologies/design descisions for use in the Sage project.

Plugin Platform Based Application

The goal of Sage is mainly to allow people in the POE community to easily create features they are passionate about. POE is really lucky to have a community that makes tons of cool tools. That being said most people are not very passionate about setting up Auth flows, building basic UI, wrapping the GGG api, implementing caching layers, ect. To make it easier for people in the community to build cool tools Sage is a plugin first application. The Sage project accomplishes these goals in two parts.

  • Sage implements a common libarary and plugin interface that allows plugins to easily access any of the data needed to create features. This layer handles the authentication, API rate limits, comes with logical defaults for caching, and provides easy ways to bind this data to React. Using this library it is easy to write plugins that have advanced functionality with a relativiely small amount of code (my example stash viewer plugin is around 60 total lines of React). It also lowers the bar that is required to start building a feature while also exposing these features to a wider audiance than most developers would have when starting their first POE tool.
  • Sage also implements an Electron UI that allows users to authenticate, and then install plugins. The UI handles plugin life cycles and displaying the plugins pages to the user. Sage also handles keeping the base library/plugins updated on user machines and collecting metrics/logs for debugging purposes.

Since plugin development is the goal of sage all core functionality that can be a plugin should be a plugin.

Monorepo

Sage is a monorepo that contains all projects that go into running the Sage application. This includes:

  • Common libraries for building plugins
  • The Electron application
  • Example plugins
  • The GGG api wrapper
  • Backend code for:
    • Tracking POE pricing information
    • Handling user auth
    • TFT integrations
  • All core and published plugins

Okay so what? What are the advantages of keeping all of these projects in one repo? The main benifit is that at its core Sage is a shared library and a shared platform. Monorepos have a couple of properties that make building shared code easier. The first benifit is that having all of the code in one place makes sharing the code easier. In our case if all plugins are being developed within the main repo they will all have local access to the compiled common library, electron app, and GGG api right inside of the same directory they are doing plugin development in.

This leads into the next benifit which is versioning. What if as they are making a change to their plugin they want to make a change to the common library to enable a new feature. Since this is a mono repo they can just make the change and submit one PR that contains the new feature in the common repo and the change to their plugin that uses the new feature. When this change is merged the SHA of the squashed commit is effecively both the plugin and library version in one this enables versioning quite easily since old versions of the Electron app that are running an older version of the shared library can still easily access the plugins code as it existed before the change was made while newer versions can access the newer version. This can be accomplished without a monorepo but even in this simple case the monorepo makes it easier.

What if our change to the common repo wasn't just a new feature but a change to an existing feature that would require plugins to do some sort of migration? Well if we want to make that kind of change in the mono repo all of the published plugins exist inside the project so for the projects build to pass not just the common library must be buildable but all the plugins in the monorepo also need to build. So if we change the name of a function in the common library we must make sure all the plugins recieve the same change or the PR build will fail. In this manner once this PR is passing and is merge we know that at this time all plugins that live in the monorepo already build with whatever change we made. It isn't possible to make a change to the common library that would cause a published plugin to not compile because compiling the published plugins is a part of the build of the common library, and again we know that for the resulting git hash all published plugins were built with the matching version of the library and all hashes before that one were built with the previous version since it's all built together at once it always all matches up. Without using a monorepo this is much harder to accomplish and it is much easier to push changes to a common lib that will break downstream consumers.

Finally the mono repo helps keep all pugins updated. If we made the breaking change described above and published a new library version plugin developers would need to update their dependancy and publish a new version of their plugin. When they did this we would need to make sure that people running the newest version of the Electron app pull in the newest version of the plugin while people running an older version of the Electron app need to pull in the older version of the plugin. The mono repo simplfies this all by making sure that when something is merge into the mono repo it must build and pass tests with the current version of the common libraries, and when the common libraries are updated that PR must also pass build and tests for all the consumers. In this way everyone stays in sync.

React/Electron

One of Sage's main goals is to make the building of POE tools easier and more accessible to most people it can. React/Electron are a great combination for this goal. Electron be built to run on pretty much any computer that can also run POE. React is the most used web framework by far. In my opinion this combination will give the average person the easiest on-ramp to building thier first script while also giving Sage access to the most community support during development.

RxJS

RxJs might seem like a bit of a werid choice as this does add some base complexity to working with Sage. RxJs does have a learning curve and is a technology not everyone is familiar with but it helps solve some problems that are highly relevant to Sage. The main issue it solves is that when building the Sage common libraries we need to consider that any combition of existing and future plugins could be running at the same time. As someone working on the common library we relatively little control over how the plugins will use the common library. As an example lets consider rate limits for stash tabs. There are many potential plugins that are interested in stash tab data. As a user clicks through their plugin tab each plugin faces descisions about if they should load stash data, access the common cache, access thier own cache, ect. These descisions are complicated and become more complicated the more plugins and different implmentations of these descsions that enter the mix. Sage prefers a different approach. The plugins make none of these descisons, in this example they can request tab data as often as they even on each render and the common library will transparently handle returning cached data, returning stale data while refreshing the data, and push events to other subscribers even if they did not make the request. Accomplishing this in a way that is stable and generic enough for use across many apis is made much easier by building the common library in an event based architecture. Rather than writing my own event busses/publishers/consumers I choose RxJs as while it does take some getting used too at least people who have used it before will not have to learn a totally custom solution. Using prebuilt bindings the actual use of RxJs in plugins can be minimized keeping the complexity more contained to the common library.

TailwindCss

Tailwind was also choosen to complement the plugin based architechture of the project. Using Tailwind themes and common library components we can pretty easily achieve a unified look across plugins while side stepping the complexity that comes with dynamically loading custom css files and layering them correctly as different plugins would surely end up using the same names as other plugins without knowing.

Getting Started

This document will help you get started with Sage development. By the end of the document you will have a development enviroment setup and a test plugin running.

Prerequisites

To effecitvely work on a Sage plugin or the common library it would be helpful if you had some experience with the following:

  • Typescript/Javascript - All the plugins are written in Typescript/Javascript these are not hard languages to pick up but you might struggle a bit at the start.
  • React - All the UI is built with React you don't need to be a React wizard but some understanding will go a long way.
  • Tailwindcss - All of the CSS in the project is done with Tailwind again you don't need much experience but understanding what it is would help a lot.

As you get into more advanced features it might also be helpful to have experience with:

  • RxJs - The common library uitilizes this heavily you will need a basic understanding to work inside the common library, while writing scripts you can use RxJs but most of it can also be abstracted with prebuilt bindings/promises.
  • Node/Electron - For creating scripts you don't need to know much about these but as you get into more advanced features or working with the common library these will become useful.

System Requirements

Before moving on make sure you have the following installed:

  • NodeJS - node -v and npm -v should work
  • Java - java -version should show a jdk higher than version 8 (technically to build plugins you do not need this but it will make it much more complicated)
  • Git - git -v should work

Project Setup

Follow these instructions to clone and setup the project:

  • Clone the Sage project git clone https://github.com/PoeStack/poestack-sage You should now have a directory containing the code from this repo
  • Inside the repo run the following command ./gradlew npmInstall or ./gradlew.bat npmInstall on Windows. This will install all the npm modules for all sub projects in the repo
  • Next run ./gradlew npmBuild or ./gradlew.bat npmBuild on Windows. NOTE: on some systems this command stalls the first time it is ran if it has been running a bit kill it and run it again
  • Next navigate to the src/echo-app directory cd ./src/echo-app
  • Execute the following command npm run start
  • A Electron window should open. Congratulations you have correctly installed Sage for local development

Debugging:

  • Make sure you have the newest version of node installed
  • Mac consideration here

Creating Your First Plugin

This section will go over creating a simple plugin from scratch. We'll be creating a "hideout is lava" plugin that will show stopwatch whenever we enter our hideout to remind us to get back to mapping. For this section some file management commands might not work on all systems if they don't just make the directory/copy files manually.

The first step is to create a new directory for our plugin in /src/echo-plugins. mkdir src/echo-plugins/hideout-is-lava

Lets cd into the new directory so our working directory is our new plugins home. `cd src/echo-plugins/hideout-is-lava

Now that we have a directory to work in we'll need some base files. Lets copy these from an example plugin to give us a starting point. rsync -av --exclude=node_modules ../../echo-plugin-examples/poe-log-plugin/ . (Copy everything from the echo-plugin-examples/poe-log-plugin into the current directory except for node_modules)

We should now have a package.json with some deps, a src directory containing the example plugin, and some supporting files.

To start lets rename our plugin. Open package.json in your edtior and change "name": "example-poe-log-plugin" to "name": "hideout-is-lava-plugin"

Now run npm install to install the plugin's deps

Next lets make some changes to the plugins src/entry.tsx file. Open it in an editor and change plugin: 'example-log-plugin-stash' to plugin: 'hideout-is-lava-plugin'

Lets also change the icon in the side bar Change DocumentTextIcon to FireIcon and change the import to import { FireIcon } from '@heroicons/react/24/outline'

Now lets update the app change the contents of src/App.tsx to

const App = () => {
  return <>Hello from our new plugin</>
}

export default App

Now run npm run build to build our new plugin

Next open a termninal in src/echo-app run npm run start to launch the Electron app. In the side bar you should now see the fire icon. Click it and you should see our test message.

Leave the Electron app open and change our test message to <>Wow our plugin is cool</>. Run npm run build in the plugin directory and in the Electron app press cmd + r or F5 on Windows to refresh the page. Go back to the fire icon and you should see the new next without having recompiled/restarted the Electron app.

Okay we've now got all the basics out of the way lets start building our actual plugin.

Sage Development Discord

Below you will find the invite to the Sage development welcome channel. To gain access to the actual development channels you need to read the below requirements and meet one of them.

Requirements - At-least one of the following:

  • Have completed the development getting started tutorial. Send a link to your fork and a picture of the compiled app in the #apply-for-dev-access channel.
  • Be a developer/contributor to an existing POE tool. Send a link to your github showing contribution in the #apply-for-dev-access channel.
  • Have the tool dev role or higher in either TFT or the Path of Exile Discord mention having it in the #apply-for-dev-access channel.
  • Have a significant Github history contributing to open source software. Send a link to your Github in the #apply-for-dev-access channel.
  • Are a Patreon supporter. Mention this in the #apply-for-dev-access channel.

Once your application is approved you will be given a role that will grant you access to the discussion channels.

If you meet any of the above and would like to talk about development feel free to join: https://discord.gg/DNb6Axbvnv

How Docs Are Built

The docs are built and compiled to html using mdbook. The mdbook is hosted by github pages.

Local dependencies

  • install rust and cargo
  • cargo install mdbook
  • cargo install mdbook-pagetoc
  • cargo install mdbook-theme

Build

  • Run mdbook serve ./src/sage-docs
  • navigate to http://localhost:3000/ in a web browser

References

  • https://katopz.medium.com/how-to-build-mdbook-with-github-actions-eb9899e55d7e for actions setup
  • https://rust-lang.github.io/mdBook/index.html
  • https://github.com/slowsage/mdbook-pagetoc