Damon Otter

Mad Otter Games
Developer
Hi,
We are aware that the Halloween Build that went Live last week had many new bugs. We are sorry for all of the inconvenience and frustration this has caused to all of our players.

I want to give an update on why these bugs occurred, and where we are with fixing them.

We pushed a server-only build yesterday that (hopefully) fixed the main bug we were seeing: players were getting stuck when changing zones. So far, it seems to be working, but it's a little early to declare it is certainly fixed. We have only seen one bug report over the last 24 hours about a problem when zoning; whereas before we fixed the build we were getting 20 or so bug reports per day.

The second most reported bug is about problems when crafting items, talking to some Npcs (vendors), and some other related issues. We are certain we have a fix for this bug. The fix is currently on Test Server, and once we feel the build on Test has been tested enough, we plan to push it Live- either on Saturday or Sunday.

How did this buggy build get live?
We have long wanted to fix server-side lag in the game. We embarked on a major project to refactor some of the deep underlying systems that the game relies on. Changes like this are fundamental to how the game code runs, so we were aware that they could introduce major bugs. Of course, we did a lot of internal testing to find and fix as many bugs as we could-- and we also held the build on TEST server for a long healthy test cycle. We fixed the bugs that were reported. The players on Test were excellent in reporting bugs that they found.

So, how did we miss the bugs? It's because these bugs are rare, and some of them require a lot more load and stress on the game than we can get on the Test server. Test may have 20 or 30 people on at a time, but Live servers have 20x this amount.

The Good News is that server-side lag is GREATLY IMPROVED. This will make a huge difference going forward. Basically, it's like the foundation of a house-- we had to replace the foundation. Now it's robust and sturdy and we can build a good frame on top of it.

I'm aware that some players are still reporting "lag"- but we have investigated these issues, and they are primarily frame rate issues caused by graphics-- for example, they occur in large battles in Raids or against Elder Bosses or in the Event Zone. All the special effects can really impact frame rate. Now that server-side lag is greatly improved, we can now focus on client-side fixes like improving frame rate issues caused by excessive graphical effects.

So, in summary, I think the short-term pain (about a week to 10 days) was certainly frustrating, but in the long-run, it will make the game much better.

Best,
Damon
 

BlueDevil

Great Adventurer
Characters
BlueDevil, BluDemon, FatesOrb, BluAngel, Baelfiree, Catching Shade
Platform
Android
When spells go green (On PC) in rift - for example, we had 8 people running last night. On that token, since rift is separate it shouldn't be "laggy"? I believe it's server-side because when the EZ is calm (bosses not spawning when we've run rift before) we don't have lag. People can jump up and down, but can't hit spells or move. When people are killing bosses in the EZ, there are events going on outside of where we are - we don't get any lag until they occur, and that is what happened with our small group last night. Both hops (1&2) spawned and were subsequently killed at the same time.
So in summary the server-side build pushed yesterday was to fix this issue?
 
Reactions: Damon Otter
  • Likex1

Damon Otter

Mad Otter Games
Developer
When spells go green (On PC) in rift - for example, we had 8 people running last night. On that token, since rift is separate it shouldn't be "laggy"? I believe it's server-side because when the EZ is calm (bosses not spawning when we've run rift before) we don't have lag. People can jump up and down, but can't hit spells or move. When people are killing bosses in the EZ, there are events going on outside of where we are - we don't get any lag until they occur, and that is what happened with our small group last night. Both hops (1&2) spawned and were subsequently killed at the same time.
So in summary the server-side build pushed yesterday was to fix this issue?
Thanks for the feedback. What you describe and what your group experienced certainly sounds like server-side lag. The reason I believe this is because:
1. Your group only had 8 people
2. You could move around normally at full speed, but you couldn't get spells off - so that means that your frame rate was fine, but the server was falling behind processing commands (it would be helpful if you let us know if you actually check "fps" in game when you are seeing lag, and if it's good (20 or more) or not.
3. You believe it was correlated to when EZ is busy.

We are still working on fixing lag, and I'll make sure the programmers are aware of your post. That said, I can say that despite your experience, lag is much better than before. The reason is because there are diagnostics available showing when one of the world servers is delayed, and we can see the pauses/delays are a) much fewer in number, and b) shorter on average.

Oh, and finally, the server-build we pushed yesterday was not to fix lag per se... it was to fix the zone load freeze bug that was introduced in the builds prior (which were to fix server side lag).

We will continue to work on lag for a few more weeks.
Best, Damon
 

Damon Otter

Mad Otter Games
Developer
p.s.
For those of you interested in technical details, the improvements to anti-lag performance were mainly of two types:
1. Reducing garbage - data that is no longer being used (referenced by anything) must be collected. Garbage collectors are notoriously slow since they have to "walk-the-tree" of references to determine what is actually no longer being referenced, and if so, delete it. Sometimes the tree of references can have cycles (loops) so walking the tree must avoid going in circles.
2. Time-expensive tasks are moved to a thread (a separate process) so they don't delay the main thread (which processes all game events, like spells, attacks, items, etc. etc.). Basically, think of a thread as a helper: the more things that the main worker thread can hand-off to helpers, the faster the main worker thread can run without delays. Time-expensive tasks are things like synching a player character when it enters/leaves a zone, checking credentials, persisting data for permanent backup, etc. These are simple examples, and these were already in threads, but there are more obscure things that we've been tracking down and moving to threads.

Finally, we have another strategy we're going to complete before the end of the year that will make it so that when there is lag, it won't cause players to take damage and die during this period of time when they can't heal, etc.
 

BlueDevil

Great Adventurer
Characters
BlueDevil, BluDemon, FatesOrb, BluAngel, Baelfiree, Catching Shade
Platform
Android
Thanks for the feedback. What you describe and what your group experienced certainly sounds like server-side lag. The reason I believe this is because:
1. Your group only had 8 people
2. You could move around normally at full speed, but you couldn't get spells off - so that means that your frame rate was fine, but the server was falling behind processing commands (it would be helpful if you let us know if you actually check "fps" in game when you are seeing lag, and if it's good (20 or more) or not.
3. You believe it was correlated to when EZ is busy.

We are still working on fixing lag, and I'll make sure the programmers are aware of your post. That said, I can say that despite your experience, lag is much better than before. The reason is because there are diagnostics available showing when one of the world servers is delayed, and we can see the pauses/delays are a) much fewer in number, and b) shorter on average.

Oh, and finally, the server-build we pushed yesterday was not to fix lag per se... it was to fix the zone load freeze bug that was introduced in the builds prior (which were to fix server side lag).

We will continue to work on lag for a few more weeks.
Best, Damon
Thank you kindly for clarifying!
 
Reactions: Zazie
  • Likex1

Ekimie

Hunter
Royal Guardian
Platform
Steam
Finally, we have another strategy we're going to complete before the end of the year that will make it so that when there is lag, it won't cause players to take damage and die during this period of time when they can't heal, etc.
This is especially nice! Looking forward to it's implementation. Thank you for your explanation, I always love these little dev side views, I learn something that way
 
Reactions: Deri and Zazie
  • Likex2

BFurry

Wizard
Platform
Android
Some players are getting stuck in the event zone... bug reports sent. please help them
IMG_0918.png
 
Reactions: Cthulhu
  • Likex1

Joleen

BRA Member
Royal Guardian
Characters
Kavita, Katona, Chephirah, Avani, Lash LaRue
Platform
Android, PC
Finally, we have another strategy we're going to complete before the end of the year that will make it so that when there is lag, it won't cause players to take damage and die during this period of time when they can't heal, etc.
That is most excellent!
 

ParmaV

Hunter
Platform
Android
I am now unable to enter the EZ on any of my characters and the one who was in the EZ cannot get back in because it goes to traveling screen then goes to Load zone failed. Internal error occurred while entering the zone. So my character who is collecting for event gear can't get back into the zone to carry on. I have been trying since early this morning as in around 07:00 hrs BST. I have sent bug reports but am posting here as it basically crashes the game whenever I try so I'm just not logging on now.
 

Damon Otter

Mad Otter Games
Developer
Hi Everyone,
We are now pushing a build that fixes the crafting issue.

It's now clear that the zone load stuck bug is not fixed- we are getting many reports. The build we just pushed today will unstick all stuck players. We're working on a fix for it now. In the meantime, we will be rebooting the servers every day until it's fixed-- the bug doesn't show up for 2 days after a reboot, so we believe this will prevent players from getting stuck.
Best, Damon
 

Cosmickitty

Bog Frog
Platform
iOS
Any chance we can get a temporary boost to plenties till fixed? They don’t spawn till 24 hrs after reboot, so there won’t be any plenties till this is fixed
 
Reactions: Squonk and Zazie
  • Likex2

CaraLox

Jack Of All Trades
Characters
10
Platform
iOS, PC
We played for a long time without plenti’s, so to go without plentis to fix a bigger problem I can deal with.
to waste time developing a temporary fix for plentis takes away development time to work on more important stuff.
 
Reactions: Majenta and Ekimie
  • Likex2

Sarah Otter

Mad Otter Games
Developer
Hi, Everyone,

We are getting much closer now to a new hotfix for the zoning/loading/stuck issue. In the meantime, for those of you who find yourselves getting stuck, one way to avoid the bug is when switching characters to wait about twenty seconds before switching. The error is happening during the switching character process -- login info for the first character is still being cleared up when login for the next character starts. As Damon mentioned, this is related to the changes we've been making to reduce lag, which is why we did not have this bug before.
 

Phergus

Master Gatherer
Platform
PC
Any chance we can get a temporary boost to plenties till fixed? They don’t spawn till 24 hrs after reboot, so there won’t be any plenties till this is fixed
Where do you get this idea that plenties don't spawn for 24 hours after a server reboot?

We had a reboot on US3 this morning and I just did a gathering run finding plenties in several zones.
 

Ozul

Citizen
Platform
iOS
Where do you get this idea that plenties don't spawn for 24 hours after a server reboot?

We had a reboot on US3 this morning and I just did a gathering run finding plenties in several zones.
Those are Plenties that were there before the reset so are still activated. When a reboot happens, new plenty cycles now take 16 to 24 hours to respawn at random Locations.
 

Cosmickitty

Bog Frog
Platform
iOS
Where do you get this idea that plenties don't spawn for 24 hours after a server reboot?

We had a reboot on US3 this morning and I just did a gathering run finding plenties in several zones.
ozul pretty much answered it, but to my understanding… a certain number are spawned after server reset. I was told it used to be higher but people thought it was too high. It takes time for regular spawning of plenties to pick back up.

also changing the plenty spawn should be simply changing one variable, I’m not asking them to rewrite the entire plenty code 🤷‍♀️
 
Top