SphereServer BugTracker - SphereServer
View Issue Details
IDProjectCategoryView StatusDate SubmittedLast Update
0002496SphereServerexecutable - windows buildpublic21-03-15 12:3825-03-15 16:25
Reporternolok 
Assigned ToXuN 
PrioritynormalSeveritycrashReproducibilitysometimes
StatusresolvedResolutionfixed 
PlatformOSOS Version
Product Version 
Target VersionFixed in Version0.56c Nightly 
Nightly VersionAutomated (specify build number)
Experimental FlagsNone
Option FlagsNone
Internal Build Number
Summary0002496: Crash probably due to highly crowded areas
DescriptionI'm using sphere to decorate my world with other guys, so i have sectors with 1200-1300 items. There isn't a big lag, but i think that sphere has difficulty to manage these sectors, in fact walking from a crowded sector to another makes the server crash very often and loop generating enormous logs, until i manually kill the server.

I tested with nightly builds 0002150 and 0002215.

Example log:
20:40:5:'x' commands 't'=1 (it's like .tele)
20:41:DEBUG:__ thread (1624) __ | # | _____ function _____________ | ticks passed from previous function start ______
20:41:DEBUG:>> 1624 | 0 | NetworkThread::tick | +0
20:41:DEBUG:>> 1624 | 1 | NetworkOutput::processOutput | +15
20:41:DEBUG:>> 1624 | 2 | NetworkOutput::processPacketQueue | +0 <-- exception catch point (below is guessed and could be incorrect!)
20:41:DEBUG:>> 1624 | 3 | NetworkOutput::sendPacket | +0
20:41:DEBUG:__ thread (2280) __ | # | _____ function _____________ | ticks passed from previous function start ______
20:41:DEBUG:>> 2280 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)
20:41:CRITICAL:"Access Violation" (0xfffcf153), in NetworkManager::Tick() 0000001 "cleaning queues"
20:41:CRITICAL:Exception, in NetworkOutput::processPacketQueue() 0000001 "sending"
20:41:DEBUG:id='5', pri='2', packet '2' of '50' to send, length '52' of '20000'
20:41:DEBUG:__ thread (1624) __ | # | _____ function _____________ | ticks passed from previous function start ______
20:41:DEBUG:>> 1624 | 0 | NetworkThread::tick | +0
20:41:DEBUG:>> 1624 | 1 | NetworkOutput::processOutput | +15
20:41:DEBUG:>> 1624 | 2 | NetworkOutput::processPacketQueue | +0 <-- exception catch point (below is guessed and could be incorrect!)
20:41:DEBUG:>> 1624 | 3 | NetworkOutput::sendPacket | +16
20:41:DEBUG:id='5', pri='2', packet '3' of '50' to send, length '78' of '20000'
20:41:DEBUG:__ thread (1624) __ | # | _____ function _____________ | ticks passed from previous function start ______
20:41:DEBUG:>> 1624 | 0 | NetworkThread::tick | +0
20:41:DEBUG:>> 1624 | 1 | NetworkOutput::processOutput | +15
20:41:DEBUG:>> 1624 | 2 | NetworkOutput::processPacketQueue | +0 <-- exception catch point (below is guessed and could be incorrect!)
20:41:DEBUG:>> 1624 | 3 | NetworkOutput::sendPacket | +16
20:41:DEBUG:__ thread (2280) __ | # | _____ function _____________ | ticks passed from previous function start ______
20:41:DEBUG:>> 2280 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)
20:41:DEBUG:id='5', pri='2', packet '4' of '50' to send, length '104' of '20000'
20:41:DEBUG:__ thread (1624) __ | # | _____ function _____________ | ticks passed from previous function start ______
20:41:DEBUG:>> 1624 | 0 | NetworkThread::tick | +0
20:41:DEBUG:>> 1624 | 1 | NetworkOutput::processOutput | +15
20:41:DEBUG:>> 1624 | 2 | NetworkOutput::processPacketQueue | +0 <-- exception catch point (below is guessed and could be incorrect!)
20:41:DEBUG:>> 1624 | 3 | NetworkOutput::sendPacket | +32
20:41:DEBUG:id='5', pri='2', packet '5' of '50' to send, length '130' of '20000'
20:41:DEBUG:__ thread (1624) __ | # | _____ function _____________ | ticks passed from previous function start ______
20:41:DEBUG:>> 1624 | 0 | NetworkThread::tick | +0
20:41:DEBUG:>> 1624 | 1 | NetworkOutput::processOutput | +15
20:41:DEBUG:>> 1624 | 2 | NetworkOutput::processPacketQueue | +0 <-- exception catch point (below is guessed and could be incorrect!)
20:41:DEBUG:>> 1624 | 3 | NetworkOutput::sendPacket | +47
20:41:DEBUG:id='5', pri='2', packet '6' of '50' to send, length '156' of '20000'
20:41:DEBUG:__ thread (1624) __ | # | _____ function _____________ | ticks passed from previous function start ______
20:41:DEBUG:>> 1624 | 0 | NetworkThread::tick | +0
20:41:DEBUG:>> 1624 | 1 | NetworkOutput::processOutput | +15
20:41:DEBUG:>> 1624 | 2 | NetworkOutput::processPacketQueue | +0 <-- exception catch point (below is guessed and could be incorrect!)
20:41:DEBUG:>> 1624 | 3 | NetworkOutput::sendPacket | +47
20:41:DEBUG:id='5', pri='2', packet '7' of '50' to send, length '182' of '20000'
20:41:DEBUG:__ thread (2280) __ | # | _____ function _____________ | ticks passed from previous function start ______
20:41:DEBUG:>> 2280 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)
20:41:DEBUG:__ thread (1624) __ | # | _____ function _____________ | ticks passed from previous function start ______
20:41:DEBUG:>> 1624 | 0 | NetworkThread::tick | +0
20:41:DEBUG:>> 1624 | 1 | NetworkOutput::processOutput | +15
20:41:DEBUG:>> 1624 | 2 | NetworkOutput::processPacketQueue | +0 <-- exception catch point (below is guessed and could be incorrect!)
20:41:DEBUG:>> 1624 | 3 | NetworkOutput::sendPacket | +63
20:41:DEBUG:id='5', pri='2', packet '8' of '50' to send, length '208' of '20000'
20:41:DEBUG:__ thread (1624) __ | # | _____ function _____________ | ticks passed from previous function start ______
20:41:DEBUG:>> 1624 | 0 | NetworkThread::tick | +0
20:41:DEBUG:>> 1624 | 1 | NetworkOutput::processOutput | +15
20:41:DEBUG:>> 1624 | 2 | NetworkOutput::processPacketQueue | +0 <-- exception catch point (below is guessed and could be incorrect!)
20:41:DEBUG:>> 1624 | 3 | NetworkOutput::sendPacket | +78
20:41:DEBUG:id='5', pri='2', packet '9' of '50' to send, length '234' of '20000'

And so on, for a nice 11 MB log (server crashed in the night, i killed it in the morning).


In addition, i think those sectors are a problem also when nobody is there.
A guy was working in another town, with very few dynamic items inside, and the server crashed.
Here's the log:

23:39:5:'y' commands 'nudgeup 10'=1
23:39:'y' commands uid=040003d1d (paper lantern) to 'nudgeup 10'=1
23:39:DEBUG:__ thread (2352) __ | # | _____ function _____________ | ticks passed from previous function start ______
23:39:DEBUG:>> 2352 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)
23:39:CRITICAL:"Access Violation" (0xfffcf153), in NetworkManager::Tick() 0000001 "cleaning queues"
23:39:DEBUG:__ thread (2352) __ | # | _____ function _____________ | ticks passed from previous function start ______
23:39:DEBUG:>> 2352 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)
23:39:DEBUG:__ thread (2352) __ | # | _____ function _____________ | ticks passed from previous function start ______
23:39:DEBUG:>> 2352 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)
23:39:DEBUG:__ thread (2352) __ | # | _____ function _____________ | ticks passed from previous function start ______
23:39:DEBUG:>> 2352 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)
23:39:DEBUG:__ thread (2352) __ | # | _____ function _____________ | ticks passed from previous function start ______
23:39:DEBUG:>> 2352 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)
23:39:DEBUG:__ thread (2352) __ | # | _____ function _____________ | ticks passed from previous function start ______
23:39:DEBUG:>> 2352 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)
23:39:DEBUG:__ thread (2352) __ | # | _____ function _____________ | ticks passed from previous function start ______
23:39:DEBUG:>> 2352 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)
23:39:DEBUG:__ thread (2352) __ | # | _____ function _____________ | ticks passed from previous function start ______
23:39:DEBUG:>> 2352 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)
23:39:DEBUG:__ thread (2352) __ | # | _____ function _____________ | ticks passed from previous function start ______
23:39:DEBUG:>> 2352 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)
23:39:DEBUG:__ thread (2352) __ | # | _____ function _____________ | ticks passed from previous function start ______
23:39:DEBUG:>> 2352 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)
23:39:DEBUG:__ thread (2352) __ | # | _____ function _____________ | ticks passed from previous function start ______
23:39:DEBUG:>> 2352 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)
23:39:DEBUG:__ thread (2352) __ | # | _____ function _____________ | ticks passed from previous function start ______
23:39:DEBUG:>> 2352 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)
23:39:DEBUG:__ thread (2352) __ | # | _____ function _____________ | ticks passed from previous function start ______
23:39:DEBUG:>> 2352 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)

Then the server closed itself, so the error part of the log is that, i didn't cut anything.

We all are using the Enhanced Client and FeatureSA = 0, but i don't think that's relevant.
I also tried to increase in the ini MaxPacketsPerTick from 25 to 35 and 50, MaxQueueSize from 50 to 30, 40 and 60, MaxSizePerTick from 12000 to 13000 and 15000, but it didn't prevent server from crashing, instead in some cases it increased the lag (in the areas with 1200+ items).
In the ini i also have UseAsyncNetwork=2, UsePacketPriority=1, UseExtraBuffer=1.
Steps To Reproduce
Additional InformationOptionFlags=01|08|080|0100|0200
Experimental=04|08|040|080|0400|0800|0200|080000|020000

I wrote them here because the list below isn't up to date.
TagsNo tags attached.
Relationships
Attached Files
Issue History
Date ModifiedUsernameFieldChange
21-03-15 12:38nolokNew Issue
22-03-15 01:43ShaminoNote Added: 0002744
23-03-15 01:32CorujaNote Added: 0002745
23-03-15 15:41XuNNote Added: 0002746
23-03-15 16:41nolokNote Added: 0002750
24-03-15 08:20XuNNote Added: 0002752
24-03-15 22:01nolokNote Added: 0002753
25-03-15 16:24XuNNote Added: 0002754
25-03-15 16:24XuNStatusnew => resolved
25-03-15 16:24XuNFixed in Version => 0.56c Nightly
25-03-15 16:24XuNResolutionopen => fixed
25-03-15 16:24XuNAssigned To => XuN

Notes
(0002744)
Shamino   
22-03-15 01:43   
Uhmmm, maybe try with UseAsyncNetwork=0 and test.
(0002745)
Coruja   
23-03-15 01:32   
try using UseAsyncNetwork=0, this feature can make the server unstable because it have some unknown errors

but are you using the latest nightly? yesterday build got some important changes which directly affect enhanced clients (the "open gump" packet) and also it fixes some "network" console errors too
(0002746)
XuN   
23-03-15 15:41   
As it's been said in the other replies, switch to 0 AsyncNetwork setting.

Increasing values in this settings will also help you:

// Maximum number of packets to send per tick
MaxPacketsPerTick=25 // try increasing this to 300

// Maximum number of packets before lowering packet priorities (0 for no limit)
MaxQueueSize=50 // same here

// Maximum number of bytes to send per tick (also governs maximum size of outgoing packets)
MaxSizePerTick=12000 // increase this to 1MB: 1000000

These values may be reduced by your network capacities, but Sphere will perform better anyway... however having such amount of items is impossible to handle in a live server no matter the emulator/game you play... I would recomend to put them into statics (if you were not planning to).
(0002750)
nolok   
23-03-15 16:41   
Yup, i'm planning to freeze them. Are there other tools than multool to do this?
Anyway, when I'll have a couple of minutes (a couple of days i think) I'll test your solutions, but why do I get errors if I am not in that sector?
(0002752)
XuN   
24-03-15 08:20   
This errors are related to this setting, which never worked as it should, I disabled it in the source in my last commit until it's fixed to avoid any problem more.
(0002753)
nolok   
24-03-15 22:01   
Tested with last nightly (with AsyncNetwork=0): no crash, even in sectors with 1500 items, that's nice :) Thanks guys
(0002754)
XuN   
25-03-15 16:24   
anytime