0001914: Segmentation fault - SphereServer BugTracker

(0000915)
trinax (reporter)
14-02-11 11:11
edited on: 14-02-11 23:56

Another one, I have turned on debug script.

11:01:DEBUG:SCRIPT: strsub(0,4,'online') -> 'onli'
11:01:DEBUG:SCRIPT: strsub(0,5,'online') -> 'onlin'
11:01:DEBUG:SCRIPT: strsub(0,6,'online') -> 'online'
11:01:DEBUG:SCRIPT: strsub(0,6,'online') -> 'online'
11:01:DEBUG:SCRIPT: strsub(0,4,'online') -> 'onli'
11:01:DEBUG:SCRIPT: strsub(0,5,'online') -> 'onlin'
11:06:FATAL:(sphere_serv_triggers.scp,292)Server Unstable: Aborted

This time without lots of segmentation faults but only aborted.

FYI, I do A LOT of strsub (possibly including trying to extract a longer string from a short string, like strsub 0 5 on "ab" if that may cause a problem) and strcmpi under f_oncommand for both players and staffs. These errors doesn't look like related to the player number though, so f_oncommand may be irrelevant but...

sphere_server_triggers.scp

[FUNCTION f_onserver_save]
B @1949,4 Kayit Aliniyor,Lutfen Bekleyiniz..
LOG Save Stage Started. Forced: <dARGN1>. Stage#: <dARGN2>
IF (<ARGN2> != -1) && (<ARGN2> != 2147483647)
    LOG Background save detected. Ensuring background saving is disabled. <<-- LINE 292!
    SAVEBACKGROUND = 1
    SAVEBACKGROUND = 0
    //IF !(<ARGN1>)
    //LOG Ensuring foreground save...
    //ARGN1 = 1
    //ENDIF
ENDIF
RETURN 0

(0000916)
Mirravin (reporter)
14-02-11 14:56

Have you tested last Sphere build(#1419)?

(0000917)
trinax (reporter)
14-02-11 15:00
edited on: 15-02-11 00:04

No, these are 1414. If I remember right I had these even 2 years ago before I closed my server (these are actually one of the reasons why I closed), with the pre-release 56b. They are about some scripts I assume but I can't find it because the error doesn't take me anywhere.

(0000918)
Mirravin (reporter)
14-02-11 15:06

Do you use TIMERF CLEAR function?

(0000919)
trinax (reporter)
14-02-11 15:09
edited on: 14-02-11 15:13

No, no timerf clear or timerf -1 in my scripts. I use lots of timerfs-timerds though.

(0000923)
trinax (reporter)
15-02-11 12:16

Decreased the f_oncommand checks a little and server has been running fine for 24 hours now, the problem actually may be about them.

(0000927)
trinax (reporter)
16-02-11 00:10
edited on: 16-02-11 01:03

Another one:

00:14:DEBUG:SCRIPT: strsub(0,6,'s nerdesÄ±n a.q') -> 's nerd'
00:14:DEBUG:SCRIPT: strsub(0,5,'s nerdesÄ±n a.q') -> 's ner'
00:14:DEBUG:SCRIPT: strsub(0,4,'s nerdesÄ±n a.q') -> 's ne'
00:14:d5:'sdfgh' commands 's nerdesÄ±n a.q'=1
00:14:ERROR:fc:Bad Login 0 (Account does not exist)
00:14:ERROR:fd:Bad Login 0 (Account does not exist)
00:14:ERROR:fe:Bad Login 0 (Account does not exist)
00:14:FATAL:Segmentation fault
00:14:FATAL:Segmentation fault
*
00:15:FATAL:Segmentation fault
00:15:CRITICAL:'Main' thread hang, restarting...
00:15:FATAL:(tamamlanansistemler.scp,663)Server Unstable: Aborted

Line 663 is "serv.account update"

I am trying to log every character trigger to find, since sphere doesn't give enough information which is nonsense, but I doubt it will be able to write to file before the crash.

I am using client 2.0.3 for my server btw.

Sphere really should give exactly what was done before any errors, with a flag or something, if it is saying there is a fault, why is not there a fault code for you to find the cause or info for us to find the possible bad scripting? That is a BIG missing feature for debugging. This is ruining the server.

(0000928)
trinax (reporter)
16-02-11 10:48

Again:

03:22:DEBUG:SCRIPT: strsub(0,6,'bandaj') -> 'bandaj'
03:22:DEBUG:SCRIPT: strsub(0,5,'bandaj') -> 'banda'
03:22:DEBUG:SCRIPT: strsub(0,4,'bandaj') -> 'band'
03:22:FATAL:Segmentation fault
03:22:FATAL:Segmentation fault
03:22:FATAL:Segmentation fault
*
03:22:FATAL:Segmentation fault
03:22:CRITICAL:'Main' thread hang, restarting...
03:22:FATAL:Server Unstable: Aborted

(0000929)
trinax (reporter)
16-02-11 15:29

I have removed those strsub checks from players completely, to see if it helps to determine if that is the problem.

In the mean time, are you considering to update the sphere to give information about that fault?

(0000934)
trinax (reporter)
17-02-11 13:46

Another:

13:51:DEBUG:SCRIPT: strsub(0,4,'invis') -> 'invi'
13:51:7d:'handsome' commands 'invis'=1
13:51:P'Maverick' was killed by P'Frank GoztepeLee'.
13:51:FATAL:Segmentation fault
13:51:FATAL:Segmentation fault
13:51:FATAL:Segmentation fault

(0000935)
MrSugarCube (administrator)
17-02-11 14:00

The strsub messages are purely informational messages and don't mean anything. What other debug flags have you got enabled?

There isn't really any information to go by, but you could try disabling NetworkOutThread to see if that makes a difference.

(0000936)
trinax (reporter)
17-02-11 14:03
edited on: 17-02-11 14:07

I only have DEBUGF_SCRIPTS enabled. Which ones should I enable to get some useful information? Is there a flag to log which function,trigger is being executed.

I am disabling networkoutthread now.

(0000937)
trinax (reporter)
17-02-11 20:36
edited on: 17-02-11 23:47

Again:

18:44:P'Naraxyn ImmortaL' was killed by P'Destroy'.
18:44:P'Liana Heizenberg' was killed by N'Associate of Herman'.
18:44:FATAL:Segmentation fault
18:44:FATAL:Segmentation fault
18:44:FATAL:Segmentation fault
18:44:FATAL:Segmentation fault
18:44:FATAL:Segmentation fault

I am re-enabling networkout.

Can't you update sphere to give information before fatals?

Edit:
I think sphere keeps running while those faults are thrown and stops running when it says main thread freeze.

(0000939)
MrSugarCube (administrator)
19-02-11 15:09
edited on: 19-02-11 23:26

If Sphere were able to handle the error nicely then it wouldn't be 'fatal'. :)

I've enabled the same debug information that gets shown in Windows (was previously not working on Linux builds), so if you can reproduce this again on tonight's build (#1425+) we might hopefully get something a bit more meaningful from the error.

(0000940)
trinax (reporter)
20-02-11 11:39

Can't try because 1425 uses too much CPU.

Btw, do I need to keep debugf_Script enabled to see the new messages?

(0000941)
MrSugarCube (administrator)
20-02-11 14:14
edited on: 20-02-11 14:38

Nope, it's automatic.

The high CPU usage is caused by the extra debugging, is the server usable enough to at least get the fault to occur?

(0000942)
trinax (reporter)
20-02-11 17:02
edited on: 24-02-11 00:16

There is many players during the weekend. I can give it a try tomorrow.

The fault is random, happened in between 5 mins - 55 hours so far.

Have been running fine for 30 hours now. -.-"

60 hours..

(0000947)
trinax (reporter)
24-02-11 11:59

It didn't crash for too long, losing players cause of lag.
Can this debugging be an option in sphere.ini?

(0000948)
MrSugarCube (administrator)
24-02-11 12:25
edited on: 24-02-11 12:26

It's not possible sorry, it has to be always on or always off.

If you are still using build 1425 then updating to 1426+ might reduce CPU slightly (compared to 1425, my CPU usage dropped from 80-90% to 60-70%).

(0000949)
Mirravin (reporter)
24-02-11 14:28
edited on: 24-02-11 18:02

Can you add this debugs to the separate builds(normal build; DEBUG build)?
Because I don't have problems with the stability of Sphere, this debugs has essentially reduced the performance for me, but I can't get last build without it...

P.s. Returned to build: #1419 now.

(0000950)
trinax (reporter)
25-02-11 00:09
edited on: 25-02-11 00:09

Tried r1428 but CPU usage didn't drop much. I will purchase a more powerful machine to find the bug because player number drops therefore I can't experience the crash.

I have downloaded and stored r1425 and r1428, you can turn the debug option off for following builds. I will post information when I can run the debug build.

(0000956)
trinax (reporter)
12-03-11 12:12

Finally I got it to crash with debug build.

13:03:ERROR:14:Bad Login 9 (The account details entered are invalid (username or password is too short, too long or contains invalid characters). This can sometimes be caused by incorrect/missing encryption keys)
13:06:FATAL:Segmentation fault
13:06:FATAL:Segmentation fault
.
.
13:07:FATAL:Segmentation fault
13:07:WARNING:Forcing thread 'Main' to terminate...
13:07:CRITICAL:'Main' thread hang, restarting...
13:07:FATAL:Server Unstable: Aborted

Still not much of an information :(

(0000960)
MrSugarCube (administrator)
13-03-11 05:15

Can you check the internal build number when Sphere starts? There are no debug messages.. :(

(0000962)
trinax (reporter)
13-03-11 10:07
edited on: 13-03-11 14:57

12:56:Welcome to Sphere XXXXXXX www.xxxxxxx.com , compiled at Feb 23 2011 (01:02:02), internal build #1428
12:56:WARNING: --- WARNING ---

This is a nightly build of SphereServer. This build is to be used
for testing and/or bug reporting ONLY. DO NOT run this build on a
live shard unless you know what you are doing!
Nightly builds are automatically made every night from source and
might contain errors, might be unstable or even destroy your
shard as they are mostly untested!
---------------------------------

12:56:Signal handlers installed.
12:56:Expansion maps supported: T2A, ML
12:56:Allocating map sectors: 0=7168 1=7168
12:56:Indexing 265 scripts...

(0000970)
trinax (reporter)
17-03-11 15:51
edited on: 17-03-11 23:11

The one I was using is debug build I think :P The crash is still happening, I wrote a bash script to restart the server if hang though so we are holding up at the moment.

Is there a way that I make a trigger that affects all items? There are events that are associated with all npcs|players but not for items unfortunately. [typedef t_normal] on=@timer or something will work? or [typedef 255] maybe?

Asking this to log whats going on.

P.S. Will "typedef trigger" trigger before the one in the itemdef? If not, there should be an option to do it so, and we can hopefully find the bug :P

(0000980)
trinax (reporter)
24-03-11 20:59
edited on: 26-03-11 13:07

Well it crashed :P That script isn't the only place findtype is called though. It is also called every @spellcast under player event.

Additionally, if I remember right, the RAM usage of VPS suddenly hit max when this fault happened 3 years ago (with the pre-release). Now I will try but I don't think I can observe the RAM usage in the current VPS.

Edit:

Ram usage stays the same, cpu usage drops from 30% to 20%.

(0001023)
trinax (reporter)
12-04-11 13:18
edited on: 13-04-11 01:05

Still no output about the crash in #1440

Can't you add a flag-enabled feature, something like script profiler that it will write/append the last function/trigger call before it is called (with script/line/rtime etc. maybe) to a file?

Looks like logging each call before it is done is the simplest and maybe the only way to find the bug, which I assume to be user related and happening in some specific script lines and therefore will be in usermade functions/triggers.

P.S. Isn't OS supposed to output information about the error? Is the signal handled by sphere manually, trying to avoid crash?

(0001027)
MrSugarCube (administrator)
15-04-11 03:48

If secure mode is enabled (SECURE in sphere.ini, or 's' in console) then Sphere tries to handles a number of signals. You can try disabling this if you think the OS will give you any information, but when I do that here I just get a generic error message and the process dies.

I've made a temporary tweak to the segmentation fault handling so that printing the debug information is the first thing Sphere will do after displaying the Segumentation Fault message. If you could try to reproduce on tomorrow's automated build (version #1443) we might have better luck with getting the information out (you'll need to keep secure mode enabled though :)).

(0001037)
Mirravin (reporter)
18-04-11 01:26

That is interesting, error about save exception while closing console in build #1443 has dissappeared at all, instead of extra debugging.
http://tracker.sphere.torfo.org/bugs/view.php?id=1880 [^]

(0001039)
trinax (reporter)
20-04-11 17:20

I still couldn't get it to crash.

(0001045)
trinax (reporter)
26-04-11 22:30
edited on: 26-04-11 22:52

Finally, after 200 hours of runtime :D Funny the actual time we would see some information about the bug was the one with the longest uptime.

22:41:FATAL:Segmentation fault
22:41:DEBUG:__ thread (3056810896) __ | # | _____ function _____________ | ticks passed from previous function start ______
22:41:DEBUG:>> 3056810896 | 0 | CChar::OnTick | +0 <-- exception catch point (below is guessed and could be incorrect!)
22:41:DEBUG:>> 3056810896 | 1 | CObjBase::OnTickStatusUpdate | +1
22:41:FATAL:Segmentation fault
22:41:DEBUG:__ thread (3056810896) __ | # | _____ function _____________ | ticks passed from previous function start ______
22:41:DEBUG:>> 3056810896 | 0 | CChar::OnTick | +0 <-- exception catch point (below is guessed and could be incorrect!)
22:41:DEBUG:>> 3056810896 | 1 | CObjBase::OnTickStatusUpdate | +1
22:41:FATAL:Segmentation fault
.
.
.
.
22:41:FATAL:Segmentation fault
22:41:DEBUG:__ thread (3056810896) __ | # | _____ function _____________ | ticks passed from previous function start ______
22:41:DEBUG:>> 3056810896 | 0 | CChar::OnTick | +0 <-- exception catch point (below is guessed and could be incorrect!)
22:41:DEBUG:>> 3056810896 | 1 | CObjBase::OnTickStatusUpdate | +1
22:41:FATAL:Segmentation fault
22:41:DEBUG:__ thread (3056810896) __ | # | _____ function _____________ | ticks passed from previous function start ______
22:41:DEBUG:>> 3056810896 | 0 | CChar::OnTick | +0 <-- exception catch point (below is guessed and could be incorrect!)
22:41:DEBUG:>> 3056810896 | 1 | CObjBase::OnTickStatusUpdate | +1
22:41:FATAL:Segmentation fault
22:41:DEBUG:__ thread (3056810896) __ | # | _____ function _____________ | ticks passed from previous function start ______
22:41:DEBUG:>> 3056810896 | 0 | CChar::OnTick | +0 <-- exception catch point (below is guessed and could be incorrect!)
22:41:DEBUG:>> 3056810896 | 1 | CObjBase::OnTickStatusUpdate | +1
22:41:WARNING:Forcing thread 'Main' to terminate...
22:41:FATAL:(fix.scp,398)Server Unstable: Aborted

There are lots of them and all seem to be same.

(0001056)
trinax (reporter)
04-05-11 13:18
edited on: 04-05-11 13:19

(0001064)
trinax (reporter)
12-05-11 18:03

(0001066)
MrSugarCube (administrator)
13-05-11 02:32

Which automated build did you use to get that last one?

(0001069)
trinax (reporter)
13-05-11 14:05

#1443

(0001071)
MrSugarCube (administrator)
13-05-11 21:53

Could you try to reproduce again with the latest? There's a [small] chance that some of the changes made in builds 1446, 1451, 1453 and 1454 may have made some kind of difference.

(0001074)
trinax (reporter)
14-05-11 09:53
edited on: 14-05-11 10:02

(0001076)
trinax (reporter)
15-05-11 16:58
edited on: 15-05-11 16:59

(0001090)
trinax (reporter)
20-05-11 20:45
edited on: 21-05-11 12:39

with r1466:

16:00:(x.scp,49)TEST CRASH CONT
16:00:FATAL:Segmentation fault
16:00:DEBUG:__ thread (3057044368) __ | # | _____ function _____________ | ticks passed from previous function start ______
16:00:DEBUG:>> 3057044368 | 0 | CChar::OnTick | +0 <-- exception catch point (below is guessed and could be incorrect!)
16:00:DEBUG:>> 3057044368 | 1 | CObjBase::OnTickStatusUpdate | +2
16:00:FATAL:Segmentation fault
16:00:DEBUG:__ thread (3057044368) __ | # | _____ function _____________ | ticks passed from previous function start ______
16:00:DEBUG:>> 3057044368 | 0 | CChar::OnTick | +0 <-- exception catch point (below is guessed and could be incorrect!)
16:00:DEBUG:>> 3057044368 | 1 | CObjBase::OnTickStatusUpdate | +2
16:00:FATAL:Segmentation fault
16:00:DEBUG:__ thread (3057044368) __ | # | _____ function _____________ | ticks passed from previous function start ______
16:00:DEBUG:>> 3057044368 | 0 | CChar::OnTick | +0 <-- exception catch point (below is guessed and could be incorrect!)
16:00:DEBUG:>> 3057044368 | 1 | CObjBase::OnTickStatusUpdate | +2
16:00:FATAL:Segmentation fault
16:00:DEBUG:__ thread (3057044368) __ | # | _____ function _____________ | ticks passed from previous function start ______
16:00:DEBUG:>> 3057044368 | 0 | CChar::OnTick | +0 <-- exception catch point (below is guessed and could be incorrect!)
16:00:DEBUG:>> 3057044368 | 1 | CObjBase::OnTickStatusUpdate | +2
16:00:FATAL:Segmentation fault

EDIT: Another one:
01:08:(asd.scp,5173)TEST CRASH CONTF
01:08:(asd.scp,5173)TEST CRASH CONTF
01:08:(asd.scp,5173)TEST CRASH CONTF
01:08:FATAL:Segmentation fault
01:08:DEBUG:__ thread (3056597904) __ | # | _____ function _____________ | ticks passed from previous function start ______
01:08:DEBUG:>> 3056597904 | 0 | CChar::OnTick | +0 <-- exception catch point (below is guessed and could be incorrect!)
01:08:DEBUG:>> 3056597904 | 1 | CObjBase::OnTickStatusUpdate | +3
01:08:FATAL:Segmentation fault
01:08:DEBUG:__ thread (3056597904) __ | # | _____ function _____________ | ticks passed from previous function start ______
01:08:DEBUG:>> 3056597904 | 0 | CChar::OnTick | +0 <-- exception catch point (below is guessed and could be incorrect!)
01:08:DEBUG:>> 3056597904 | 1 | CObjBase::OnTickStatusUpdate | +3
01:08:FATAL:Segmentation fault
01:08:DEBUG:__ thread (3056597904) __ | # | _____ function _____________ | ticks passed from previous function start ______
01:08:DEBUG:>> 3056597904 | 0 | CChar::OnTick | +0 <-- exception catch point (below is guessed and could be incorrect!)
01:08:DEBUG:>> 3056597904 | 1 | CObjBase::OnTickStatusUpdate | +3
01:08:FATAL:Segmentation fault

What is statusupdate? Can I look somewhere specific for that?
I am logging every single function/trigger with serv.log, sphere reads scripts linearly right? It does not go another function/trigger until current one is finished? So if something I did caused the crash, last serv.log should belong to it?

(0001921)
Ben (manager)
18-01-14 18:45

Closing this as it is more then likely a script issue.

Issue History
Date Modified	Username	Field	Change
13-02-11 22:57	trinax	New Issue
14-02-11 11:11	trinax	Note Added: 0000915
14-02-11 11:12	trinax	Note Edited: 0000915	View Revisions
14-02-11 14:56	Mirravin	Note Added: 0000916
14-02-11 15:00	trinax	Note Added: 0000917
14-02-11 15:01	trinax	Note Edited: 0000917	View Revisions
14-02-11 15:06	Mirravin	Note Added: 0000918
14-02-11 15:09	trinax	Note Added: 0000919
14-02-11 15:13	trinax	Note Edited: 0000919	View Revisions
14-02-11 15:13	trinax	Note Edited: 0000919	View Revisions
14-02-11 22:34	trinax	Note Edited: 0000915	View Revisions
14-02-11 22:34	trinax	Note Edited: 0000915	View Revisions
14-02-11 23:56	trinax	Note Edited: 0000915	View Revisions
15-02-11 00:04	trinax	Note Edited: 0000917	View Revisions
15-02-11 00:04	trinax	Note Edited: 0000917	View Revisions
15-02-11 12:16	trinax	Note Added: 0000923
15-02-11 13:01	trinax	Tag Attached: crash
16-02-11 00:10	trinax	Note Added: 0000927
16-02-11 00:10	trinax	Note Edited: 0000927	View Revisions
16-02-11 00:21	trinax	Note Edited: 0000927	View Revisions
16-02-11 00:22	trinax	Note Edited: 0000927	View Revisions
16-02-11 00:43	trinax	Note Edited: 0000927	View Revisions
16-02-11 00:43	trinax	Note Edited: 0000927	View Revisions
16-02-11 00:45	trinax	Note Edited: 0000927	View Revisions
16-02-11 00:51	trinax	Note Edited: 0000927	View Revisions
16-02-11 00:52	trinax	Note Edited: 0000927	View Revisions
16-02-11 00:53	trinax	Note Edited: 0000927	View Revisions
16-02-11 01:03	trinax	Note Edited: 0000927	View Revisions
16-02-11 10:48	trinax	Note Added: 0000928
16-02-11 15:29	trinax	Note Added: 0000929
17-02-11 13:46	trinax	Note Added: 0000934
17-02-11 14:00	MrSugarCube	Note Added: 0000935
17-02-11 14:03	trinax	Note Added: 0000936
17-02-11 14:07	trinax	Note Edited: 0000936	View Revisions
17-02-11 20:36	trinax	Note Added: 0000937
17-02-11 20:38	trinax	Note Edited: 0000937	View Revisions
17-02-11 20:39	trinax	Note Edited: 0000937	View Revisions
17-02-11 20:39	trinax	Note Edited: 0000937	View Revisions
17-02-11 23:47	trinax	Note Edited: 0000937	View Revisions
19-02-11 15:09	MrSugarCube	Note Added: 0000939
19-02-11 23:26	MrSugarCube	Note Edited: 0000939	View Revisions
20-02-11 11:39	trinax	Note Added: 0000940
20-02-11 14:14	MrSugarCube	Note Added: 0000941
20-02-11 14:38	MrSugarCube	Note Edited: 0000941	View Revisions
20-02-11 17:02	trinax	Note Added: 0000942
20-02-11 23:14	trinax	Note Edited: 0000942	View Revisions
22-02-11 13:24	trinax	Note Edited: 0000942	View Revisions
24-02-11 00:16	trinax	Note Edited: 0000942	View Revisions
24-02-11 11:59	trinax	Note Added: 0000947
24-02-11 12:25	MrSugarCube	Note Added: 0000948
24-02-11 12:26	MrSugarCube	Note Edited: 0000948	View Revisions
24-02-11 14:28	Mirravin	Note Added: 0000949
24-02-11 18:02	Mirravin	Note Edited: 0000949	View Revisions
25-02-11 00:09	trinax	Note Added: 0000950
25-02-11 00:09	trinax	Note Edited: 0000950	View Revisions
12-03-11 12:12	trinax	Note Added: 0000956
13-03-11 05:15	MrSugarCube	Note Added: 0000960
13-03-11 10:07	trinax	Note Added: 0000962
13-03-11 10:09	trinax	Note Edited: 0000962	View Revisions
13-03-11 10:10	trinax	Note Edited: 0000962	View Revisions
13-03-11 10:11	trinax	Note Edited: 0000962	View Revisions
13-03-11 10:11	trinax	Note Edited: 0000962	View Revisions
13-03-11 10:12	trinax	Note Edited: 0000962	View Revisions
13-03-11 14:57	trinax	Note Edited: 0000962	View Revisions
17-03-11 15:51	trinax	Note Added: 0000970
17-03-11 15:52	trinax	Note Edited: 0000970	View Revisions
17-03-11 15:56	trinax	Note Edited: 0000970	View Revisions
17-03-11 15:59	trinax	Note Edited: 0000970	View Revisions
17-03-11 16:00	trinax	Note Edited: 0000970	View Revisions
17-03-11 18:15	trinax	Note Edited: 0000970	View Revisions
17-03-11 23:11	trinax	Note Edited: 0000970	View Revisions
24-03-11 20:59	trinax	Note Added: 0000980
25-03-11 13:32	trinax	Note Edited: 0000980	View Revisions
25-03-11 21:38	trinax	Note Edited: 0000980	View Revisions
26-03-11 00:36	trinax	Note Edited: 0000980	View Revisions
26-03-11 13:07	trinax	Note Edited: 0000980	View Revisions
12-04-11 13:18	trinax	Note Added: 0001023
13-04-11 00:52	trinax	Note Edited: 0001023	View Revisions
13-04-11 00:52	trinax	Note Edited: 0001023	View Revisions
13-04-11 00:53	trinax	Note Edited: 0001023	View Revisions
13-04-11 01:01	trinax	Note Edited: 0001023	View Revisions
13-04-11 01:03	trinax	Issue Monitored: trinax
13-04-11 01:05	trinax	Note Edited: 0001023	View Revisions
15-04-11 03:48	MrSugarCube	Note Added: 0001027
18-04-11 01:26	Mirravin	Note Added: 0001037
20-04-11 17:20	trinax	Note Added: 0001039
26-04-11 22:30	trinax	Note Added: 0001045
26-04-11 22:31	trinax	Note Edited: 0001045	View Revisions
26-04-11 22:52	trinax	Note Edited: 0001045	View Revisions
04-05-11 13:18	trinax	Note Added: 0001056
04-05-11 13:19	trinax	Note Edited: 0001056	View Revisions
12-05-11 18:03	trinax	Note Added: 0001064
13-05-11 02:32	MrSugarCube	Note Added: 0001066
13-05-11 14:05	trinax	Note Added: 0001069
13-05-11 21:53	MrSugarCube	Note Added: 0001071
14-05-11 09:53	trinax	Note Added: 0001074
14-05-11 10:02	trinax	Note Edited: 0001074	View Revisions
15-05-11 16:58	trinax	Note Added: 0001076
15-05-11 16:59	trinax	Note Edited: 0001076	View Revisions
20-05-11 20:45	trinax	Note Added: 0001090
20-05-11 20:55	trinax	Note Edited: 0001090	View Revisions
20-05-11 20:56	trinax	Note Edited: 0001090	View Revisions
20-05-11 20:56	trinax	Note Edited: 0001090	View Revisions
20-05-11 21:11	trinax	Note Edited: 0001090	View Revisions
21-05-11 12:39	trinax	Note Edited: 0001090	View Revisions
18-01-14 18:45	Ben	Note Added: 0001921
18-01-14 18:45	Ben	Status	new => closed
18-01-14 18:45	Ben	Assigned To	=> Ben
18-01-14 18:45	Ben	Resolution	open => unable to reproduce

Relationships