Plan | Player Analytics 5.6 build 2965

5.5 build 1951 - Change log
This release contains update to Java 11, Sponge 8, Fabric 1.19.1, as well as Webserver rewrite to Jetty. In addition it contains new features and improvements, and more progress on Frontent BETA in the form of rewritten server & network pages.

Special thanks to Vankka, Kopo, Drex & stashenko for contributions to this update

Change Log
Breaking changes
This version updates config so backup your config in case you need to revert back to 5.4!

• Java 11: Plan now requires Java 11 or newer to run. Java 8 is no longer supported.
• Sponge API 8: Sponge API 7 is no longer supported.
• Fabric 1.19.1: Fabric 1.18 is no longer supported

If you can't update, you can continue using 5.4 build 1722 even if connected to same database as newer versions. (No database schema changes!)

Supported APIs

• Updated to Java 11, which was required by Jetty and Sponge API 8.
• Vankka implemented Sponge API 8 changes.
• Drex implemented support for Fabric 1.19.1

New features
Jetty Webserver
New webserver implementation speeds up response transmission times considerably

• Replaced Sun Webserver with Jetty (https://www.eclipse.org/jetty/), an open source embedded webserver library.
• Plan now supports HTTP/2
• Plan now supports TLSv1.3 for HTTPS, and browsers have a larger selection of cipher suites they can use.
  • Users of PlanCert.jks (Included in the jar) are encouraged to create their own self signed cert. I had to implement some extra code to support expired certificate (Since PlanCert.jks inside the jar has expired), and that may not be as secure as creating a new self-signed .p12 - Instructions will be added to the wiki, but the plugin also logs the instructions if you use that cert.
• Fixed gzip compression for responses (speeds up transmission)

Access log
Access logs are now stored in Plan database in plan_access_log table.

• You can also log access to console with Webserver.Security.Access_log.Print_to_console (default false) in the config. This is useful for debugging reverse-proxy issues.
• The IP of each request is stored in the database - as it is necessary for security (a valid legal basis for use with GDPR).
• The request logs in database are removed after Webserver.Security.Access_log.Remove_logs_after_days (default 30)

Swagger
Plan json endpoints are now documented with Swagger available on the webserver itself. You can access the Swagger documentation from /docs on your Plan webserver after updating.

Data Gathering

• Fabric should now gather join addresses correctly. Previously player IPs were gathered by mistake. To clear the invalid data use /plan db removejoinaddresses {servername} once you install the plugin.
• Added 'Data_gathering.Preserve_join_address_case' setting (default false) that attempts to preserve join address case. Note that some servers/clients may lowercase addresses regardless.
• Possibly fixed an issue where Quit event is processed before Join, leading to long incorrect sessions being stored on next join/shutdown
• Possibly fixed an issue where saving sessions that ended on shutdown failed when another join had just stored one of the join addresses that were previously unseen. (It would retry next reload and probably succeed - so low priority issue.)

Frontend BETA progress
Enable with Plugin.Frontend_BETA setting in the config. Note that Html Customization and Export are not yet supported in the beta.

• Rewrote /login page in React
• Rewrote /server page in React
  • You can now switch between visualizations for 'current playerbase' and 'latest join address' pie-graphs
    
  • Join addresses by day graph is now available to analyze incoming join addresses further.
    
• Rewrote /network page in React
  • You can now switch between Map projections in Geolocations
    
  • You can now sort servers in server list by specific values
    
  • Join addresses by day graph also available for network page
  • Added Players online to network performance tab - data from multiple servers can be viewed in one graph on network performance tab
• Fixed various issues and cleaned up overall appearance

You can track progress of frontend beta here

Bugfixes

• Fixed lag caused by large tab completion results if server had a lot of players and someone tab completed /plan player .
• Fixed empty cookies preventing Plan login due to error
• Reduced maximum join address length to 191 characters due to limitation of old MySQL versions

Configuration

• Added plan.command permission (default: true) on Spigot servers to allow hiding /plan command from tab completion by negating this permission
• Config structure was modified to be valid yaml so that other programs can validate it. The config is updated automatically with current values of old config versions.
• Increased default inactive player removal threshold 180 days -> 3650 days (10 years). This change is not applied automatically to old configs.
• stashenko updated RU locale
• Added /plan db migrate_to_online_uuids command to help migrating from an offline-mode server to online-mode
  • Moves data of offline uuids to online uuids (Looks up online uuid from Mojang)
  • Combines data of offline uuids and online uuids if both exist
  • See /plan db for more help

Extensions

• Plugin method calls should no longer be made during Plan disable.
• Updated PlotSquared extension, should now register correctly
• Updated Floodgate extension, should now register correctly
• Updated Nucleus extension for Sponge 8
• Removed RedProtect support for Sponge (No longer supported in Sponge 8)

Automation

• Dev builds for every commit is now available DEV build pipeline (Click on title and scroll down to Artifacts, note that you need to log in to Github.)

Click to expand...

Change log 5.4 build 1722
This update fixes fabric 1.19 support and join address gathering.
Special thanks to Kopo for contributions to this update.
Change log
Fix join address gathering (No longer 'Unknown')
An oversight in the way the player leave listener code was structured caused the player join address to be removed from memory before its linking to the session that was ending.
It is now fixed.
Fabric 1.19 support
Kopo fixed fabric pipeline for 1.19 - Unfortunately the changes are incompatible with older versions of fabric. If you need older version of fabric use older version of Plan.
Frontend BETA

• /players page rewritten in React now available when opted in to frontend beta
  • Note that no fixes to functionality were added to it yet, just implementation in React.
• Server page rewrite is making good progress and is about 75% complete.

Server identification

• Possibly fixed ServerInfoFile.yml clearing and server duplicating if MySQL is busy during enable

5.4 build 1690
This update fixes some annoying bugs and prepares the plugin for the move to 5.5 and Java 11 while keeping this version compatible for those unable to update their servers to Java 11.
If no further critical bugs surface next update will be 5.5.
Change log
Plugin stability

• Fixed issue where Plan would prevent server enable from continuing if database was not responsive during enable.
• Fixed 2 crashes when database write latency is high
  • Fixed thread starvation crash during high latency when using ViaVersion or ProtocolSupport
  • Reduced chance of out of memory crash when during high latency by starting to drop plugin data transactions for a while if queue grows over 500 transactions. (The transactions continue if queue returns to <50)

Database
Join address schema change

• Join addresses are now stored in plan_join_address table and id referencing that is added to plan_sessions table for time based analysis implementation later.
  • Backwards compatible schema change - old versions will insert 'unknown' as each sessions address.
  • plan_user_info.join_address column was not removed, but its use is discouraged for 3rd parties as its use may be removed in the future.

Fixes to patches

• Fixed UserInfoOptimizationPatch resetting join_address data when applied
• Fixed typo in RemoveDanglingServerDataPatch that caused it to fail
• Possibly improved speed of RemoveDanglingUserDataPatch

Other fixes to database

• Prevented transactions from executing if database ran into fatal error. This reduces amount of errors logged on a failed patch
• Fixed error related to user_id constraint violation for Ping and Session store transactions. If they execute faster than the user register transaction the player will not have up-to-date information until the proper transaction executes, but the storage of other data will no longer fail.
• Prevented ActiveCookieStore related DbOpException if plugin failed to enable

Website

• Server page > Performance now lists Average players under Performance as Numbers.

Frontend BETA

• Implemented player ping graph on the player page when opted into frontend beta
• Fixed nickname list not showing colors properly
• Fixed terminal icon not loading when user registered with console
• Fallback to using relative address if Alternative_IP is misconfigured to reduce confusion. Previously nothing would load if address was different, now there's a chance something loads.

Placeholders

• Wrote tests for placeholders
  • Fixed player_recent_kill_{1-10} and player_recent_death_{1-10} off-by-one mistake and related error for n=10.
  • Fixed player_unique_players_today erroring if there had been no sessions during today.
• New placeholders
  • %plan_player_current_session_length%: The length of active session (Online time)
  • %plan_player_current_session_length_raw%: unformatted ms
  • %plan_player_latest_session_length%: The length of active session or previous session if not online
  • %plan_player_latest_session_length_raw%: unformatted ms
  • %plan_player_previous_session_length%: The length of previous non-active session even if online
  • %plan_player_previous_session_length_raw%: unformatted ms
• player_ placeholders now support adding layername or :uuid at the end of placeholder to show data for another player
  • Note that if the player by given identifier is not found the placeholder is not replaced.

Other bugfixes

• Console will now log colors properly on Bukkit and Bungee based servers
• Fixed some _raw placeholders from being formatted
• Fixed Error related to GriefDefender Extension

Bugfixes - Change log 5.4 build 1633

This update brings a couple important bugfixes to build 1628. Please read change log of 1628 if you're running older build than 1628.

Change log
Database

• Fixed databases failing to patch schema if plan_users or plan_servers were missing rows that had uuids in other tables.
  This affected roughly ~1 in 80 servers based on reports and metrics.
  This problem could occur if server had shutdown at a specific time during player join, or if someone had deleted rows manually from plan_users or plan_servers.
  • The fix deletes dangling data so that the rest of the patches can execute properly.
  • If you have run into this problem on your server this update should fix your database automatically.
• Fixed SQL errors in ViaVersion Extension and ProtocolSupport Extension

Click to expand...

Frontend BETA and Database optimizations - Change log 5.4 build 1628

Hello! This update brings some great database optimizations, beginning of Frontend BETA (Opt-in), as well as small fixes. Read more below

This update is backwards incompatible due to database schema changes.
Servers with old Plan versions should not be enabled after starting a server with this update installed.

How to prepare for this update

Networks: Make sure to update all servers that are connected to the database!
If an old version starts with the new database schema it will attempt to reverse some of the new changes.

Click to expand...

Special thanks to Kopo, Pingger & KasperiP for contributions to this update.

Change log
Database optimizations (Schema changes)
A lot of work was put into analyzing bottlenecks in queries and slow query logs sent in by users.

Benchmarks using 17k players are promising, as /players page now loads in 5s vs 2+min before.

Database patches will apply on first enable of the update to modify the database schema. This can take up to 4 minutes. Networks: Only enable one Plan server and wait for it to patch database before starting other servers.

Changed database schema to optimize query speed and table size

If you're using these columns via Query API you can replace your use case:
If you need uuid in ResultSet, use SELECT u.uuid ... JOIN plan_users u on u.id=<table>.user_id
If you have uuid or server uuid in WHERE, use server_id=(SELECT id FROM plan_servers WHERE uuid='<server uuid>')
Some queries that were identified as slow were optimized to perform better by changing the query
Geolocations queries
Ping country table query
Player table query
Other queries are also sped up by the uuid -> user_id and server_uuid -> server_id changes, as comparing integers is faster than Strings.
Extension data insert was made faster by keeping icon ids in memory for short duration
MySQL insert performance may be improved since auto-commit was turned off
Query page queries now use user ids instead of uuids - this fixes 'Max SQL size exceeded' issues.
Added a new index to plan_ping table to speed up ping queries.
Removed redundant uuid indexes that are now replaced with foreign key indexes for user_id and server_id, which reduces database size.
Frontend BETA (Opt-in)
Frontend is being rewritten in React one page at a time. This is to have more control over what is displayed on the website, since a lot of requested features will require more interaction or hiding features partially. - While doable with vanilla javascript, code would quickly become unmaintainable.

The goal is to get feature parity with the current frontend implementation by rewriting one page at a time and then switch over completely.

Some features will not use React frontend until everything is complete.

To Opt-in set Plugin.Frontend_BETA to true
Player page has been rewritten in React, other pages still use previous implementation.
Html Customization is currently incompatible with the beta and is disabled automatically.
Export is currently incompatible with the beta and old pages continue to be exported even if beta is enabled.
Automated browser testing was improved a lot and all pages are now tested for basic javascript errors.
Data gathering
Fixed AFK time not being gathered on Fabric, the movement detector was incorrectly firing every time a player position packet arrived (Happens once per second).
Config
Default Player head image address was changed from cravatar to crafatar. Remove the whole row from the config and reload Plan if you wish to use the new setting.
Locale
Kopo did a massive locale system improvement in order to support translations on the frontend side, implementing yml conversion, namespacing all of the existing locale keys and writing json endpoints for frontend to load locale - as well as the code responsible for loading those using i18next.

Locale files now use .yml format. Any locale files you may be using are converted automatically.
The new React Frontend allows changing language on the frontend on a per-user basis with a dropdown menu - This is currently only available on the rewritten pages (player page)

KasperiP fixed a typo in Finnish locale
Export
Players page is now exported more frequently
Exported pages should no longer show the Logout button
Extensions
Essentials Economy was split from Essentials Extension. Now there are 2 extensions managed separately. This was done since the economy calls were triggering updates too frequently. If you have disabled Essentials Extension you may need to disable EssentialsEco Extension separately
Fixed NPE related to LuckPerms Extension
Fixed SQL syntax error related to LuckPerms Extension
Fixed issues with KingdomsX Extension when using newer version of KingdomsX (They changed names of some of their packages)
Pingger added support for 1.18 stone blocks to LogBlock Extension
QoL changes
/plan servers command now lists Plan version on each server so it is easier to keep track of servers being out of date.
Bugfixes
Fixed negative register dates in player dat files breaking the database
Fixed ConcurrentModificationException related to AFK tracker
Prevented use of incorrectly formatted WorldAlias regex settings
Fixed Plan failing to enable if database query took too long during enable.

Click to expand...

5.4 build 1583 - Hotfix for MySQL issue
This release contains a hotfix for MySQL driver unloading issues on disable/reload, causing the plugin to spew a really long error message and failing to reload when using MySQL.
Driver is no longer unloaded during reload to avoid any issues.
This bug was introduced in dev build 1536 on January 6th, but it was not caught or reported.

5.4 build 1581 - Bugfixes
This build contains a heck of a lot of bugfixes and a bunch of small features.
Special thanks to HexedHero and Kopo for contributions to this update & thanks to everyone who has been reporting issues!
Change log
Data gathering

• Kopo fixed not logging deaths on fabric when killer was not a player

Config

• Throw more detailed error when Unit of setting is incorrect
• Added Display_options.Player_head_img_url config setting for the player page head image

Stability

• Fixed SpongeForge crashing with Plan on enable. This was caused by a bug in SQLite.
• Possibly fixed OutOfMemory crash where server ran out of threads. Amount of tasks registered by Plan over time was reduced.
• Possibly fixed error on server shutdown related to class loading ("zip file closed")
• Possibly fixed duplicate servers being stored in the database when database is busy during enable.

Website

• Added Join address to /player pages
• Fixed player pages with 2 different UUIDs showing same player's data when both players had same name (In cases where offline and online UUIDs are present due to user error)
• Fixed ISO code for South Korea and North Korea so that they show up properly on the map
• Internal error web pages now have error context
• Added support for Hex color codes in player display names on the website
• Fixed Query page 'to main page' link on reverse-proxy setups using subdirectory
• Fixed performance tab not loading on /network page if locale was not English

Extensions

• Fixed ArithmeticException related to Libertybans extension
• Fixed wrong block count for netherrack in Logblock extension
• Fixed "Connected from same IP" data from Litebans not including player names that had the same IP
• Kopo fixed Litebans extension not getting new data when there were over 5000 rows of specific punishments.
• Disabled Litebans extension automatically on game servers if it is already installed on Proxy to avoid duplicate data
• Clean up data of extension disabled in the config when plugin is reloaded to avoid confusion when a plugin does not disappear from the website after disabling them.
• Luckperms extension now updates its data when player's group is changed / group is deleted even if the player is not online.

Placeholders

• Added new placeholders
  • %plan_server_players_registered_{total/day/week/month}%
  • %plan_network_players_registered_{total/day/week/month}%
  • %plan_server_players_unique_{total/day/week/month}%
  • %plan_network_players_unique_{total/day/week/month}%
  • %plan_player_is_afk%, output true/false
  • %plan_player_is_afk_badge%, output 'AFK'/''
  • %plan_top_{category}_{time}_{n}_value%, output the value giving the player their top spot
• Fixed group by error related to top category placeholder queries
• Placeholders in the Plan config are now refreshed more often and earlier during login to improve chance they are displayed when the other plugin uses server thread for placeholders.

Export

• Hid refresh buttons on exported web files since refreshing does not update data on exported files

Commands

• Fixed fabric specific command issues
  • Fixed /plan reload
  • Fixed commands requiring confirmation expiring the confirmation immediately
  • Fixed console getting wrong kind of messages from command output

Other

• Cleaned up extra error messages from browser dev tools console
• Fixed index out of bounds exception related to trying to log errors after it failed previously (leaving behind empty file)
• Reduced log level of "Webserver disabled (disabled in config)" from warning to info
• HexedHero added more instructions to SQLite exception when temp directory is out of space for when user is using Pterodactyl

5.4 build 1516 - Year-in-review
Hello! So, today marks 5 years since Plan development started. That's a pretty long time to be honest. As has become customary, there's a survey and a year-in-review post below this updates change log. Thanks for being awesome!
Special thanks to zax71, KairyByte & OLEGSHA for contributions to this update!
Change log

• Updated HighCharts to newer version since they have fixed performance issue, in addition I removed ordinal: false workaround in Plan that was used to avoid the performance issue. If you liked the odd gaps in your data that may have been present due to the workaround, you can set Display_options.Graphs.Show_gaps_in_data to true in the Plan config to have them visible.
• Fixed NPE related to null gamemode on some platforms
• Kairubyte added version number to javascript file loading addresses so that they are no longer cached by cloudflare between updates. (Which caused issues in the past)
• zax71 and OLEGSHA added some missing plugin dependencies to softdepend in plugin.yml

Survey
Here's a link to this year's survey: https://forms.gle/cpc6trn6wfJCpg8E9
Read on below for a look back on what has been going on with Plan this year & what's in store in the future.
Year-in-review: Year 5
This year I started working as a full-time developer so the updates have been less frequent. Non-the-less, surprising to me, the development speed has not really slowed that much. Thanks to the past years of structuring the code, it is still easy to maintain and add new features to Plan.
Here are the "main" features that landed in the last year.

• The Query page
• Player join address gathering
• 15+ new Extensions
• Fabric support
• Network performance tab
• Server uptime
• Database driver downloading

In terms of features I'd say this year has been very good compared to previous years. I can't claim all the credit, since a lot of this has been made possible by the numerous contributors!
I decided to list all the contributors from 2021 (in order of appearance) here, thank you to: Guinness_Akihiko, galexrt, QuakyCZ, MrFriggo, vacoup, Kopo, Creeperface01, WolverStones, hallo1142, Aurelien30000, BruilsiozPro, WolverStones, BrainStone, AppleMacOS, 4drian3d, EyuphanMandiraci, 10935336, 洛伊, Sander0542, portlek, rymiel, mbax, untuned, PerchunPak, Shadowhackercz, HexedHero, Vankka, DrexHD, zisunny104, SkipM4, MAXOUXAX, adhg6 and BratishkaErik, zax71, KairyByte and OLEGSHA.

This year in numbers

• From build 1062 to 1515: 457 commits released as updates
• From #8 to #11 in top spigot plugins by rating: The reduced update frequency has likely impacted relative score compared to other plugins
• From 2500 servers to over 4000 servers
• From 49k lines to 59k lines of code (not counting extensions or web files)
• From 43,7% test coverage to 51.3% coverage

Last year around this time I thought there was a storm brewing in terms of maintainability, but this year has been rather smooth sailing - probably thanks to the reduced update frequency allowing longer testing times for dev builds catching more bugs before they end up in a main release.
Even though a bunch of tickets get closed all the time (260 tickets closed this year), they never run out. Luckily the new project tracker has turned out to be successful in reducing the ticket mass to manageable chunks that can be worked on whenever I have the extra energy to do so.
To turn towards the future, and adding a slight personal note, I have found a functional balance where things progress, but in bit smaller chunks than before. That said, I don't have nearly as much time for Plan as I used to before I got a job (25h/week vs ~3h/week now), so larger features take quite a long time. In an ideal world I would love to work on Plan full-time, but since Plan covers 1.5% of my monthly living expenses, it is simply not feasible.
Right now I'm also writing a Master's thesis, so Plan development will likely stay on the a backseat at least until I have finished that endeavor. Don't worry though - as you can see from the releases this year, being on the backseat has not really slowed things down
Thanks for reading, and for using Plan! I hope you have a wonderful Christmas and Happy new year!

5.4 build 1494 - Hotfix to previous build

• Fixed plugins using MySQL that started after Plan from breaking due to Plan unregistering the MySQL driver (Thanks for the fix Vankka!)

There were three things that lead to this issue not being caught:

• The driver unregistering previously dealt with other plugins using Plan driver was forgotten when writing driver download code
• The driver downloading changed so that the mysql driver was not relocated to plan.com.mysql.driver.Driver which caused the main MySQL driver to be unregistered instead
• There was no test environment in which a plugin using MySQL started after Plan

Sorry for any trouble this may have caused.

5.4 build 1490 - Bugfixes to “Further”

• Fixed Ore version checker giving 403 error due to missing User-Agent
• Released version now the same on all release platforms