After about an hour or so "Internal exception: java.net.SocketTimeoutException: Read timed out"

Discussion in 'Bukkit Help' started by Adambean, Aug 21, 2011.

Thread Status:
Not open for further replies.
  1. Offline

    Adambean

    Hello there,

    I've had an issue with our server for some time now, but not come across any solutions in searches and threads here.

    Between an hour and an hour and 50 minutes of server runtime all players on our server will notice the server has stopped responding, and get disconnected with an error message "Internal exception: java.net.SocketTimeoutException: Read timed out".

    There doesn't appear to be any trigger patten. It happens when I'm playing there alone, and I'm never doing the same thing or in the same place when it happens. Players do not have to be present for the issue to happen at all. If the server has been running for say 2 hours and someone tries to join, they immediately get stuck at "downloading terrain".

    The server still appears to be running normally if I use an external server query tool. It will report the number of player slots, world names, and the players that were connected are still on it. If players attempt to reconnect they will be stuck on the "downloading terrain" prompt. Using a 3rd party mod to show my co-ordinates at the top left (replaces version string), I see that I'm falling until it notices the server isn't sending data and gets "read timed out".

    I'm finding this hard to troubleshoot as the logs do not show any clue as to why this happens. It just looks normal. As it's a rented server I can't see a console window.

    Solution
    It turned out that iConomy 5.0/5.0.1 was the direct cause of this issue. Details

    Server information

    Hardware: Four Intel quad-core Xeon's with many GB of RAM (1.5 GB allocated to our Minecraft server process), stored on an SSD. (shared server with insane hardware)

    OS: FreeBSD 8.2 Release P2 amd64.

    Java: OpenJDK based on JDK 1.6, probably also amd64.

    Wrappers: None.

    Server build: CraftBukkit build 1060 on Minecraft 1.7.3, though this issue happened with builds 1000, 953, and any other that were released since Minecraft 1.6

    Launch command: Unsure, as it's rented and started/stopped/monitored from a web page (not on the same box as the Minecraft server).

    Plugins:
    Most of these were present long before we had this issue.
    • AdminCmd 5.7.1
    • AppleTree 0.5
    • AutoSave 2.0.1 (installed after the issue, just to ensure we don't lose any data)
    • ChestShop 3.01
    • iChat 2.3.1 (replaced for mChat due to new permissions base)
    • iConomy 5.01
    • iConomyChestShop 2.75 (replaced for ChestShop 3.01)
    • LWC 3.31
    • mChat 1.7.10_5
    • MCDocs 10
    • nChat 0.5 was removed as we thought this may be causing the problem (player chat was no longer getting formatted around the same time the issue started), replaced it with iChat
    • NoCheat 1.11
    • Permissions 3.0a (replaced for PermissionsBukkit and SuperpermsBridge)
    • PermissionsBukkit 1.1
    • PermissionsPlus 2.1.1 (replaced for PermissionsBukkit and SuperpermsBridge)
    • Remote Bukkit 0.4
    • SuperpermsBridge 1.2
    • Stackable 0.5.1
    • StarGate 0.5.5
    • Towny 0.74
    • Warpz0r 1.3.1
    • WorldEdit 4.6
    • WorldGuard 5.2.2
    Error log: Nothing at all shows any indication around the time the mass time out occurs. All that happens is a few minutes after it happens and the players are shown the "read timed out", a few generic connection timeout messages appear for players. That would be normal as the players are no longer accepting data from the server after they've left.


    Resolution attempts: First it was suggested our world could be too large or have "errors", so I used MCEdit to trim out chunks nobody used (like when someone just wanders off for miles for no reason). That didn't help, so I changed to use a fresh world -- no change.

    It was also suggested that the originally allocated 600MB of RAM was not sufficient for 8 players with the plug-ins we had, so I used the MemMonitor plug-in to verify that. With 1 person connected and doing stuff the average usage was 280 MB out of 587 MB allocated. Even with 5 of us there it didn't reach 500 MB. We were upgraded to 1.5 GB to make sure it wasn't a memory usage problem.

    Later I noticed that iConomy was posting "[INFO] [iConomy] Could not create connection: java.sql.SQLException: Login timeout" every minute or so in the server log even though we use H2SQL. This was confirmed to be a bug in iConomy 5.0, and updated to 5.01 fixed this.

    As we have FTP access to the Minecraft server folder I downloaded it all and ran it locally. I joined it and left myself connected for a few hours while I went to the pub for lunch. Came back 3 hours later, it was running just fine with me still standing there. I am running Sun's JDK 1.7 x64 on my machines though. Oddly with just me connected it used 160 MB less RAM than alone on our rented server, only averaging 120 MB usage.

    Finally, as the server host has many servers we were moved to a different machine -- same issue.

    Resolution suggestions: I have not tried running vanilla yet, mainly because switching to it erases our plug-in configuration and data files -- and there is a lot of it. The same with removing plug-ins individually. I will try this if no other suggestion is available, but I'm hoping that one of the plug-ins we use may be known to cause problems.


    Thank you for your time reading all of that. I will be grateful for any suggestions, help, or information you may have. I really can't see where the issue is.
     
  2. Offline

    Adambean

    I have since removed Permissions 3 and PermissionsPlus, switched to PermissionsBukkit with SuperpermsBridge -- same issue. Also replaced iChat for mChat, and iConomyChestShop for ChestShop.

    I've also noticed that when people join right after the server starts up they get in fine, though an hour or so at runtime if someone joins the server stops talking to the new and already present players.

    Just found out WorldGuard can dump a configuration report, so here it is:
    Code:
    WorldGuard Configuration Report
    Generated 2011-08-28 14:21 +0000
    
    Version: 5.2.2
    
    ------------------
    Server Information
    ------------------
    
    Java                : Sun Microsystems Inc. 1.6.0 (http://java.sun.com/)
    Operating system    : FreeBSD 8.2-RELEASE-p2 (amd64)
    Available processors: 16
    Free memory         : 183 MB
    Max memory          : 1523 MB
    Total memory        : 416 MB
    Server ID           : unnamed
    Server name         : Multiplay UK :: MineeeeCraft!
    Implementation      : git-Bukkit-0.0.0-980-g4ed23b1-b1060jnks (MC: 1.7.3)
    Player count        : 1/8
    
    -------
    Plugins
    -------
    
    Warpz0r          : 1.3.1
    MCDocs           : 10
    Questioner       : 0.4
    AutoSave         : 2.0.1
    iConomy          : 5.01
    Permissions      : 2.7.7
    ChestShop        : 3.01
    Stackable        : 0.5.1
    AppleTree        : 0.5.1
    Towny            : 0.74.0
    RemoteBukkit     : 0.4
    WorldGuard       : 5.2.2
    WorldEdit        : 4.6
    PermissionsBukkit: 1.1
    Stargate         : 0.5.5
    LWC              : 3.41
    AdminCmd         : 5.7.1
    mChat            : 1.7.10_5
    NoCheat          : 1.11
    
    ------
    Worlds
    ------
    
    World (0)       :
        Information:
            ID                : 4017554565098729919
            Environment       : NORMAL
            Player count      : 1
            Entity count      : 135
            Loaded chunk count: 1606
            Spawn location    : Location{world=CraftWorld{name=World}x=-27.0y=56.0z=65.0pitch=0.0yaw=0.0}
            Raw time          : 211375888
        Entities   :
            CraftPig            : 4 [0.002491]
            CraftSquid          : 7 [0.004359]
            CraftBoat           : 8 [0.004981]
            CraftPlayer         : 1 [0.000623]
            CraftSkeleton       : 7 [0.004359]
            CraftChicken        : 14 [0.008717]
            CraftPoweredMinecart: 1 [0.000623]
            CraftCreeper        : 5 [0.003113]
            CraftMinecart       : 63 [0.039228]
            CraftCow            : 5 [0.003113]
            CraftPainting       : 6 [0.003736]
            CraftSpider         : 1 [0.000623]
            CraftSheep          : 4 [0.002491]
            CraftWolf           : 1 [0.000623]
            CraftZombie         : 8 [0.004981]
    World_nether (1):
        Information:
            ID                : -2268927006438942112
            Environment       : NETHER
            Player count      : 0
            Entity count      : 131
            Loaded chunk count: 626
            Spawn location    : Location{world=CraftWorld{name=World_nether}x=0.0y=64.0z=0.0pitch=0.0yaw=0.0}
            Raw time          : 138751208
        Entities   :
            CraftArrow    : 1 [0.001597]
            CraftGhast    : 2 [0.003195]
            CraftPigZombie: 128 [0.204473]
    
    --------------------
    Global Configuration
    --------------------
    
    Configuration:
        suppressTickSyncWarnings: false
        useRegionsScheduler     : true
        activityHaltToggle      : false
        autoGodMode             : false
    
    --------------------
    World Configurations
    --------------------
    
    World (0)       :
        Information   :
            Configuration file: /themcserverfolder/plugins/WorldGuard/worlds/World/config.yml
            Blacklist file    : /themcserverfolder/plugins/WorldGuard/worlds/World/blacklist.txt
            Regions file      : /themcserverfolder/plugins/WorldGuard/worlds/World/regions.yml
        Configuration :
            CONFIG_HEADER                 : #
    # WorldGuard's world configuration file
    #
    # This is a world configuration file. Anything placed into here will only
    # affect this world. If you don't put anything in this file, then the
    # settings will be inherited from the main configuration file.
    #
    # If you see {} below, that means that there are NO entries in this file.
    # Remove the {} and add your own entries.
    #
    
            opPermissions                 : true
            fireSpreadDisableToggle       : false
            enforceOneSession             : true
            itemDurability                : true
            classicWater                  : false
            simulateSponge                : true
            spongeRadius                  : 2
            pumpkinScuba                  : false
            redstoneSponges               : false
            noPhysicsGravel               : false
            noPhysicsSand                 : false
            allowPortalAnywhere           : false
            preventWaterDamage            : []
            blockTNT                      : false
            blockLighter                  : false
            disableFireSpread             : false
            disableFireSpreadBlocks       : []
            preventLavaFire               : false
            allowedLavaSpreadOver         : []
            blockCreeperExplosions        : false
            blockCreeperBlockDamage       : true
            loginProtection               : 3
            spawnProtection               : 0
            kickOnDeath                   : false
            exactRespawn                  : true
            teleportToHome                : false
            disableContactDamage          : false
            disableFallDamage             : false
            disableLavaDamage             : false
            disableFireDamage             : false
            disableLightningDamage        : false
            disableDrowningDamage         : false
            disableSuffocationDamage      : false
            teleportOnSuffocation         : false
            disableVoidDamage             : false
            teleportOnVoid                : false
            disableExplosionDamage        : false
            disableMobDamage              : false
            useRegions                    : true
            highFreqFlags                 : false
            regionWand                    : 287
            blockCreatureSpawn            : []
            useiConomy                    : true
            buyOnClaim                    : true
            buyOnClaimPrice               : 2.0
            maxClaimVolume                : 317500
            claimOnlyInsideExistingRegions: false
            maxRegionCountPerPlayer       : 26
            antiWolfDumbness              : false
            signChestProtection           : false
            removeInfiniteStacks          : false
            disableCreatureCropTrampling  : false
            disablePlayerCropTrampling    : false
            preventLightningFire          : false
            disallowedLightningBlocks     : []
            disableThunder                : false
            disableWeather                : false
            alwaysRaining                 : false
            alwaysThundering              : false
            disablePigZap                 : false
            disableCreeperPower           : false
            disableHealthRegain           : false
            disableMushroomSpread         : false
            disableIceMelting             : false
            disableSnowMelting            : false
            disableSnowFormation          : false
            disableIceFormation           : false
            disableLeafDecay              : false
        Blacklist     :
            State: DISABLED
        Region manager:
            Type             : com.sk89q.worldguard.protection.managers.FlatRegionManager
            Number of regions: 4
            Global region    :
                Status: UNDEFINED
    World_nether (0):
        Information   :
            Configuration file: /themcserverfolder/plugins/WorldGuard/worlds/World_nether/config.yml
            Blacklist file    : /themcserverfolder/plugins/WorldGuard/worlds/World_nether/blacklist.txt
            Regions file      : /themcserverfolder/plugins/WorldGuard/worlds/World_nether/regions.yml
        Configuration :
            CONFIG_HEADER                 : #
    # WorldGuard's world configuration file
    #
    # This is a world configuration file. Anything placed into here will only
    # affect this world. If you don't put anything in this file, then the
    # settings will be inherited from the main configuration file.
    #
    # If you see {} below, that means that there are NO entries in this file.
    # Remove the {} and add your own entries.
    #
    
            opPermissions                 : true
            fireSpreadDisableToggle       : false
            enforceOneSession             : true
            itemDurability                : true
            classicWater                  : false
            simulateSponge                : true
            spongeRadius                  : 2
            pumpkinScuba                  : false
            redstoneSponges               : false
            noPhysicsGravel               : false
            noPhysicsSand                 : false
            allowPortalAnywhere           : false
            preventWaterDamage            : []
            blockTNT                      : false
            blockLighter                  : false
            disableFireSpread             : false
            disableFireSpreadBlocks       : []
            preventLavaFire               : true
            allowedLavaSpreadOver         : []
            blockCreeperExplosions        : false
            blockCreeperBlockDamage       : false
            loginProtection               : 3
            spawnProtection               : 0
            kickOnDeath                   : false
            exactRespawn                  : false
            teleportToHome                : false
            disableContactDamage          : false
            disableFallDamage             : false
            disableLavaDamage             : false
            disableFireDamage             : false
            disableLightningDamage        : false
            disableDrowningDamage         : false
            disableSuffocationDamage      : false
            teleportOnSuffocation         : false
            disableVoidDamage             : false
            teleportOnVoid                : false
            disableExplosionDamage        : false
            disableMobDamage              : false
            useRegions                    : true
            highFreqFlags                 : false
            regionWand                    : 287
            blockCreatureSpawn            : []
            useiConomy                    : false
            buyOnClaim                    : false
            buyOnClaimPrice               : 1.0
            maxClaimVolume                : 30000
            claimOnlyInsideExistingRegions: false
            maxRegionCountPerPlayer       : 7
            antiWolfDumbness              : false
            signChestProtection           : false
            removeInfiniteStacks          : false
            disableCreatureCropTrampling  : false
            disablePlayerCropTrampling    : false
            preventLightningFire          : false
            disallowedLightningBlocks     : []
            disableThunder                : false
            disableWeather                : false
            alwaysRaining                 : false
            alwaysThundering              : false
            disablePigZap                 : false
            disableCreeperPower           : false
            disableHealthRegain           : false
            disableMushroomSpread         : false
            disableIceMelting             : false
            disableSnowMelting            : false
            disableSnowFormation          : false
            disableIceFormation           : false
            disableLeafDecay              : false
        Blacklist     :
            State: DISABLED
        Region manager:
            Type             : com.sk89q.worldguard.protection.managers.FlatRegionManager
            Number of regions: 2
            Global region    :
                Status: UNDEFINED
    
    -------------
    END OF REPORT
    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 18, 2016
  3. Offline

    Mapatti

    Might be mobs spawning but not despawning. Try use plugin that limits mobspawns.
     
  4. Offline

    Adambean

    Now that you mention it there are a lot of mobs in non-towny claims areas... way more than when CanaryMod controlled it back in the day.
     
  5. Offline

    Mapatti

    I can't see any other reason why would it crash after hour or few.
     
  6. Offline

    Adambean

    Looks like the plug-in list is having some issues. Would you be able to name a plug-in which can limit the mob spawn rate? Not looking for anything complicated, something similar to hMod/CanaryMod's built-in mob rate limiter would be enough.
     
  7. Offline

    Mapatti

    Damn can't find any at the moment.
     
  8. Offline

    Adambean

    I'm not sure it's mobs. First time in a while, but I've been connected for 2 and a half hours. Everything was still fine. Just out of curiosity I disconnected and reconnected, and the issue happened. Couldn't get back in due to read timed out.

    It's like the server doesn't like talking to new clients after it's been running for so long, and stops talking to everyone when a new client arrives after that time.
     
  9. Offline

    Spl1tz

    I also got this issue.
    Since you don't have a mob controlling plugin yet, try to disable mobs to test.
     
  10. Offline

    Adambean

    Turned off mobs completely, restarted, joined. Left myself in for an hour then reconnected, read timed out.
    Appears mobs isn't the cause.
     
  11. Offline

    erdrickk

    I have this exact problem for 2 days now. I have tried every single combo of my plugins with bukkit 1060, 1064 etc.... I have no idea why this is happening.

    I will give the mob limit thing a try
     
  12. Offline

    Adambean

    I remembered the new "view-distance" setting since 1.6 when this issue started. I set it at 15 right away. I will try this at the default 10 when I get home from work.
     
  13. Offline

    erdrickk

    I tried 7, 10, 15 etc.... still getting the time out errors

    someone mentioned that McMMO + Citizens causes weird crashes sometimes - I might remove Citizens and see what happens but I use iQuest which depends on citizens....so er.... my users are going to hate me.. :p

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 18, 2016
  14. Offline

    Adambean

    Yep same, default 10 didn't fix anything.
     
  15. Offline

    jasvecht

    Try turning off Towny, that worked for me.

    Don't tell the author if it works though :p
     
  16. Offline

    erdrickk

    If I turned off Towny my users would revolt :p

    I am going to tough it out until 1.8 I did remove War v1.6 and that seemed to help alot. My theory is that since 1060 there has been some sort of Chunk Memory leak thing going on and only happening to certain chunks. Well plug-ins like War resets chunks each time a battle is done and could be causing the server crashes for me if there is some memory leakage going on. I am going to go Vanilla for a week or two with a new map then once bukkit is ready to go and the plugins I use are updated I am going to load up the old map and plugins and allow my users to move their shit over to the new map and restore the stats / money etc..
     
  17. Offline

    jasvecht

    I know, but if you try it for say 6 hours you'll know if it's where the issue lies. To be frank, that's the way to go. Turn off random plugins for short times until you identify the bugger :)


    And War simply isn't done, the Author hasn't been able on making a production version of it and using it isn't safe at all.
     
  18. Offline

    erdrickk

    I know but there is no plugin like War....it's just sooooo much fun :p
     
  19. Offline

    Adambean

    It's also hard to discover which plug-in is the cause when I can run a clone of the MCserver locally and have no issues at all :(
     
  20. Offline

    erdrickk

    Funny thing is I didn't change a thing and now my server has been stable for 14 hours or so where as before it would have Timed out 10 times by now.

    /frustrated
     
  21. Offline

    Adambean

    I see a handful of you also getting this. Would you all mind dumping your /plugins response?
     
  22. Offline

    erdrickk

    Sure! here yah go. Also I realize I run alot of large plugins but I have tons of memory and this has been running stable up until 1060

    Also - the permissions is version 2.7.4

    plugins
    08:34:42 [INFO] Plugins: FlamboyantFlock, ModTRS, SimpleCensor, ClearInventory, FalseBookCore, FalseBookIC, iQuest, mcbans, Stargate, FalseBookBlock, ecoCreature, AutoSave, LogBlockQuestioner, MyHome, WorldEdit, FalseBookExtra, ChestShop, bAntiCaps, War, Runecraft, Towny, Lottery, HelpCenter, LoginMessage, WorldGuard, Questioner, Minequery, Tombstone, AssignPermissions, iConomy, iChat, Permissions, LWC, Citizens, MobArena, LogBlock, AdminCmd, mcMMO
     
  23. Offline

    vapid2323

    Its going to be more help to get a stacktrace to find the issue, instead of guessing what plugins create the issue a stacktrace will point them out for you.

    This is the third thread I have tried asking people to do this, not sure if its to technical or something but I have yet to have anyone try it lol, let see if it works this time around.

    Run the following command as root where 3935 is the process ID

    jstack -l 3935 > stackdump20.txt

    You will then find the log in your /root folder, after that post it up on pastebin and we can see what everyone is getting.

    Just to clarify, you need to do this WHEN you get the errors, not when your server is working correctly.

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 18, 2016
  24. Offline

    Adambean

    A lot of us on rented servers don't have console access let alone root access. This won't be possible for all of us.
     
  25. Offline

    vapid2323

    That sucks! I cant even imagin not haveing SSH lol I would go nuts
     
  26. Offline

    Wahrheit

    It's his iConomy setup somehow, I did the console check for him.
     
  27. Offline

    Adambean

    It's configured to use H2SQL and appears to work fine in game. No transaction has failed. yet the console is spammed with SQL login timeouts.

    Edit: Updated to iConomy 6.0 using Flatfile instead, we'll see how this goes :)
     
  28. Offline

    vapid2323

    Can you post the link to iConomy 6 ??
     
  29. Offline

    Adambean

    http://mirror.nexua.org/iConomy/6.0/
    Had to turn off iConomy completely as other plug-ins weren't compatible with it. I'll try the server without iConomy to see if it still has this issue.

    Edit: Good news -- usually I get to 1 hour 50 minutes maximum before the server stops talking to players, but I reached the auto-restart at 2 hours. I even reconnected a few minutes before the auto-restart and got in instantly.

    So it turns out iConomy has been causing the problems so far, though I will turn off the auto-restart timer completely, leave the server running overnight, and try and join tomorrow morning. If that goes well I will be posting a bug report to iConomy.

    Edit2: Even better news -- after 12 hours of runtime I connected to it, got in instantly with no issues at all.

    Before I mention this to iConomy developers, could those of you also getting this issue please try disabling/removing iConomy and do the same test I have done? I would like my results to be as concrete as possible.
     
Thread Status:
Not open for further replies.

Share This Page