Main | My View | View Issues | Change Log | Roadmap
  

View Issue Details [ Jump to Notes ] [ Issue History ] [ Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0002496SphereServerexecutable - windows buildpublic21-03-15 12:3825-03-15 16:25
Reporternolok 
Assigned ToXuN 
PrioritynormalSeveritycrashReproducibilitysometimes
StatusresolvedResolutionfixed 
PlatformOSOS Version
Product Version 
Target VersionFixed in Version0.56c Nightly 
Summary0002496: Crash probably due to highly crowded areas
DescriptionI'm using sphere to decorate my world with other guys, so i have sectors with 1200-1300 items. There isn't a big lag, but i think that sphere has difficulty to manage these sectors, in fact walking from a crowded sector to another makes the server crash very often and loop generating enormous logs, until i manually kill the server.

I tested with nightly builds 0002150 and 0002215.

Example log:
20:40:5:'x' commands 't'=1 (it's like .tele)
20:41:DEBUG:__ thread (1624) __ | # | _____ function _____________ | ticks passed from previous function start ______
20:41:DEBUG:>> 1624 | 0 | NetworkThread::tick | +0
20:41:DEBUG:>> 1624 | 1 | NetworkOutput::processOutput | +15
20:41:DEBUG:>> 1624 | 2 | NetworkOutput::processPacketQueue | +0 <-- exception catch point (below is guessed and could be incorrect!)
20:41:DEBUG:>> 1624 | 3 | NetworkOutput::sendPacket | +0
20:41:DEBUG:__ thread (2280) __ | # | _____ function _____________ | ticks passed from previous function start ______
20:41:DEBUG:>> 2280 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)
20:41:CRITICAL:"Access Violation" (0xfffcf153), in NetworkManager::Tick() 0000001 "cleaning queues"
20:41:CRITICAL:Exception, in NetworkOutput::processPacketQueue() 0000001 "sending"
20:41:DEBUG:id='5', pri='2', packet '2' of '50' to send, length '52' of '20000'
20:41:DEBUG:__ thread (1624) __ | # | _____ function _____________ | ticks passed from previous function start ______
20:41:DEBUG:>> 1624 | 0 | NetworkThread::tick | +0
20:41:DEBUG:>> 1624 | 1 | NetworkOutput::processOutput | +15
20:41:DEBUG:>> 1624 | 2 | NetworkOutput::processPacketQueue | +0 <-- exception catch point (below is guessed and could be incorrect!)
20:41:DEBUG:>> 1624 | 3 | NetworkOutput::sendPacket | +16
20:41:DEBUG:id='5', pri='2', packet '3' of '50' to send, length '78' of '20000'
20:41:DEBUG:__ thread (1624) __ | # | _____ function _____________ | ticks passed from previous function start ______
20:41:DEBUG:>> 1624 | 0 | NetworkThread::tick | +0
20:41:DEBUG:>> 1624 | 1 | NetworkOutput::processOutput | +15
20:41:DEBUG:>> 1624 | 2 | NetworkOutput::processPacketQueue | +0 <-- exception catch point (below is guessed and could be incorrect!)
20:41:DEBUG:>> 1624 | 3 | NetworkOutput::sendPacket | +16
20:41:DEBUG:__ thread (2280) __ | # | _____ function _____________ | ticks passed from previous function start ______
20:41:DEBUG:>> 2280 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)
20:41:DEBUG:id='5', pri='2', packet '4' of '50' to send, length '104' of '20000'
20:41:DEBUG:__ thread (1624) __ | # | _____ function _____________ | ticks passed from previous function start ______
20:41:DEBUG:>> 1624 | 0 | NetworkThread::tick | +0
20:41:DEBUG:>> 1624 | 1 | NetworkOutput::processOutput | +15
20:41:DEBUG:>> 1624 | 2 | NetworkOutput::processPacketQueue | +0 <-- exception catch point (below is guessed and could be incorrect!)
20:41:DEBUG:>> 1624 | 3 | NetworkOutput::sendPacket | +32
20:41:DEBUG:id='5', pri='2', packet '5' of '50' to send, length '130' of '20000'
20:41:DEBUG:__ thread (1624) __ | # | _____ function _____________ | ticks passed from previous function start ______
20:41:DEBUG:>> 1624 | 0 | NetworkThread::tick | +0
20:41:DEBUG:>> 1624 | 1 | NetworkOutput::processOutput | +15
20:41:DEBUG:>> 1624 | 2 | NetworkOutput::processPacketQueue | +0 <-- exception catch point (below is guessed and could be incorrect!)
20:41:DEBUG:>> 1624 | 3 | NetworkOutput::sendPacket | +47
20:41:DEBUG:id='5', pri='2', packet '6' of '50' to send, length '156' of '20000'
20:41:DEBUG:__ thread (1624) __ | # | _____ function _____________ | ticks passed from previous function start ______
20:41:DEBUG:>> 1624 | 0 | NetworkThread::tick | +0
20:41:DEBUG:>> 1624 | 1 | NetworkOutput::processOutput | +15
20:41:DEBUG:>> 1624 | 2 | NetworkOutput::processPacketQueue | +0 <-- exception catch point (below is guessed and could be incorrect!)
20:41:DEBUG:>> 1624 | 3 | NetworkOutput::sendPacket | +47
20:41:DEBUG:id='5', pri='2', packet '7' of '50' to send, length '182' of '20000'
20:41:DEBUG:__ thread (2280) __ | # | _____ function _____________ | ticks passed from previous function start ______
20:41:DEBUG:>> 2280 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)
20:41:DEBUG:__ thread (1624) __ | # | _____ function _____________ | ticks passed from previous function start ______
20:41:DEBUG:>> 1624 | 0 | NetworkThread::tick | +0
20:41:DEBUG:>> 1624 | 1 | NetworkOutput::processOutput | +15
20:41:DEBUG:>> 1624 | 2 | NetworkOutput::processPacketQueue | +0 <-- exception catch point (below is guessed and could be incorrect!)
20:41:DEBUG:>> 1624 | 3 | NetworkOutput::sendPacket | +63
20:41:DEBUG:id='5', pri='2', packet '8' of '50' to send, length '208' of '20000'
20:41:DEBUG:__ thread (1624) __ | # | _____ function _____________ | ticks passed from previous function start ______
20:41:DEBUG:>> 1624 | 0 | NetworkThread::tick | +0
20:41:DEBUG:>> 1624 | 1 | NetworkOutput::processOutput | +15
20:41:DEBUG:>> 1624 | 2 | NetworkOutput::processPacketQueue | +0 <-- exception catch point (below is guessed and could be incorrect!)
20:41:DEBUG:>> 1624 | 3 | NetworkOutput::sendPacket | +78
20:41:DEBUG:id='5', pri='2', packet '9' of '50' to send, length '234' of '20000'

And so on, for a nice 11 MB log (server crashed in the night, i killed it in the morning).


In addition, i think those sectors are a problem also when nobody is there.
A guy was working in another town, with very few dynamic items inside, and the server crashed.
Here's the log:

23:39:5:'y' commands 'nudgeup 10'=1
23:39:'y' commands uid=040003d1d (paper lantern) to 'nudgeup 10'=1
23:39:DEBUG:__ thread (2352) __ | # | _____ function _____________ | ticks passed from previous function start ______
23:39:DEBUG:>> 2352 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)
23:39:CRITICAL:"Access Violation" (0xfffcf153), in NetworkManager::Tick() 0000001 "cleaning queues"
23:39:DEBUG:__ thread (2352) __ | # | _____ function _____________ | ticks passed from previous function start ______
23:39:DEBUG:>> 2352 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)
23:39:DEBUG:__ thread (2352) __ | # | _____ function _____________ | ticks passed from previous function start ______
23:39:DEBUG:>> 2352 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)
23:39:DEBUG:__ thread (2352) __ | # | _____ function _____________ | ticks passed from previous function start ______
23:39:DEBUG:>> 2352 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)
23:39:DEBUG:__ thread (2352) __ | # | _____ function _____________ | ticks passed from previous function start ______
23:39:DEBUG:>> 2352 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)
23:39:DEBUG:__ thread (2352) __ | # | _____ function _____________ | ticks passed from previous function start ______
23:39:DEBUG:>> 2352 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)
23:39:DEBUG:__ thread (2352) __ | # | _____ function _____________ | ticks passed from previous function start ______
23:39:DEBUG:>> 2352 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)
23:39:DEBUG:__ thread (2352) __ | # | _____ function _____________ | ticks passed from previous function start ______
23:39:DEBUG:>> 2352 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)
23:39:DEBUG:__ thread (2352) __ | # | _____ function _____________ | ticks passed from previous function start ______
23:39:DEBUG:>> 2352 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)
23:39:DEBUG:__ thread (2352) __ | # | _____ function _____________ | ticks passed from previous function start ______
23:39:DEBUG:>> 2352 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)
23:39:DEBUG:__ thread (2352) __ | # | _____ function _____________ | ticks passed from previous function start ______
23:39:DEBUG:>> 2352 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)
23:39:DEBUG:__ thread (2352) __ | # | _____ function _____________ | ticks passed from previous function start ______
23:39:DEBUG:>> 2352 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)
23:39:DEBUG:__ thread (2352) __ | # | _____ function _____________ | ticks passed from previous function start ______
23:39:DEBUG:>> 2352 | 0 | NetworkManager::tick | +0 <-- exception catch point (below is guessed and could be incorrect!)

Then the server closed itself, so the error part of the log is that, i didn't cut anything.

We all are using the Enhanced Client and FeatureSA = 0, but i don't think that's relevant.
I also tried to increase in the ini MaxPacketsPerTick from 25 to 35 and 50, MaxQueueSize from 50 to 30, 40 and 60, MaxSizePerTick from 12000 to 13000 and 15000, but it didn't prevent server from crashing, instead in some cases it increased the lag (in the areas with 1200+ items).
In the ini i also have UseAsyncNetwork=2, UsePacketPriority=1, UseExtraBuffer=1.
Additional InformationOptionFlags=01|08|080|0100|0200
Experimental=04|08|040|080|0400|0800|0200|080000|020000

I wrote them here because the list below isn't up to date.
TagsNo tags attached.
Nightly VersionAutomated (specify build number)
Experimental FlagsNone
Option FlagsNone
Internal Build Number
Attached Files

- Relationships

-  Notes
(0002744)
Shamino (reporter)
22-03-15 01:43

 Uhmmm, maybe try with UseAsyncNetwork=0 and test.
(0002745)
Coruja (developer)
23-03-15 01:32

 try using UseAsyncNetwork=0, this feature can make the server unstable because it have some unknown errors

but are you using the latest nightly? yesterday build got some important changes which directly affect enhanced clients (the "open gump" packet) and also it fixes some "network" console errors too
(0002746)
XuN (developer)
23-03-15 15:41

 As it's been said in the other replies, switch to 0 AsyncNetwork setting.

Increasing values in this settings will also help you:

// Maximum number of packets to send per tick
MaxPacketsPerTick=25 // try increasing this to 300

// Maximum number of packets before lowering packet priorities (0 for no limit)
MaxQueueSize=50 // same here

// Maximum number of bytes to send per tick (also governs maximum size of outgoing packets)
MaxSizePerTick=12000 // increase this to 1MB: 1000000

These values may be reduced by your network capacities, but Sphere will perform better anyway... however having such amount of items is impossible to handle in a live server no matter the emulator/game you play... I would recomend to put them into statics (if you were not planning to).
(0002750)
nolok (reporter)
23-03-15 16:41

 Yup, i'm planning to freeze them. Are there other tools than multool to do this?
Anyway, when I'll have a couple of minutes (a couple of days i think) I'll test your solutions, but why do I get errors if I am not in that sector?
(0002752)
XuN (developer)
24-03-15 08:20

 This errors are related to this setting, which never worked as it should, I disabled it in the source in my last commit until it's fixed to avoid any problem more.
(0002753)
nolok (reporter)
24-03-15 22:01

 Tested with last nightly (with AsyncNetwork=0): no crash, even in sectors with 1500 items, that's nice :) Thanks guys
(0002754)
XuN (developer)
25-03-15 16:24

 anytime

- Issue History
Date Modified Username Field Change
21-03-15 12:38 nolok New Issue
22-03-15 01:43 Shamino Note Added: 0002744
23-03-15 01:32 Coruja Note Added: 0002745
23-03-15 15:41 XuN Note Added: 0002746
23-03-15 16:41 nolok Note Added: 0002750
24-03-15 08:20 XuN Note Added: 0002752
24-03-15 22:01 nolok Note Added: 0002753
25-03-15 16:24 XuN Note Added: 0002754
25-03-15 16:24 XuN Status new => resolved
25-03-15 16:24 XuN Fixed in Version => 0.56c Nightly
25-03-15 16:24 XuN Resolution open => fixed
25-03-15 16:24 XuN Assigned To => XuN

Copyright © 2000 - 2010 MantisBT Group
Powered by Mantis Bugtracker