My server just locked up again and I ran a java stack trace (jstack) on it. Here's the results: http://pastebin.com/NWvgfWPm Here's a shortened version (just Bukkit threads): "Thread-73806" prio=10 tid=0x000000001924a800 nid=0x6e4f waiting on condition [0x000000004c4a9000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at net.minecraft.server.NetworkMasterThread.run(SourceFile:226) "Thread-10" prio=10 tid=0x0000000018a3f000 nid=0x15e6 waiting on condition [0x000000004548d000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at de.diddiz.util.MySQLConnectionPool$ConnectionReaper.run(MySQLConnectionPool.java:99) "Timer-0" daemon prio=10 tid=0x0000000018bda000 nid=0x15e5 runnable [0x000000004508c000] java.lang.Thread.State: RUNNABLE at java.util.HashMap.put(HashMap.java:401) at java.util.HashSet.add(HashSet.java:217) at com.nijiko.permissions.PermissionCache.cacheResult(PermissionCache.java:22) at com.nijiko.permissions.Entry.cache(Entry.java:255) at com.nijiko.permissions.Entry.has(Entry.java:243) at com.nijiko.permissions.Entry.has(Entry.java:229) at com.nijiko.permissions.Entry.has(Entry.java:229) at com.nijiko.permissions.Entry.hasPermission(Entry.java:195) at com.nijiko.permissions.ModularControl.permission(ModularControl.java:269) at com.nijiko.permissions.ModularControl.permission(ModularControl.java:256) at com.nijiko.permissions.ModularControl.has(ModularControl.java:249) at com.gmail.nossr50.mcPermissions.permission(mcPermissions.java:64) at com.gmail.nossr50.mcPermissions.unarmedAbility(mcPermissions.java:108) at com.gmail.nossr50.skills.Skills.monitorSkills(Skills.java:209) at com.gmail.nossr50.mcTimer.run(mcTimer.java:36) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) "Ebean-iConomyChestShop.0" daemon prio=10 tid=0x0000000019547000 nid=0x15e3 in Object.wait() [0x000000004073e000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x000000077b1ba190> (a java.lang.Object) at java.lang.Object.wait(Object.java:503) at com.avaje.ebeaninternal.server.lib.thread.PooledThread.run(PooledThread.java:75) - locked <0x000000077b1ba190> (a java.lang.Object) at java.lang.Thread.run(Thread.java:722) "EbeanBackgroundThread" daemon prio=10 tid=0x0000000018fce000 nid=0x15e2 waiting on condition [0x000000004488a000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at com.avaje.ebeaninternal.server.lib.BackgroundThread$Runner.run(BackgroundThread.java:170) at java.lang.Thread.run(Thread.java:722) "Thread-5" prio=10 tid=0x0000000018ab2000 nid=0x15e1 in Object.wait() [0x0000000044489000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.bukkit.craftbukkit.scheduler.CraftScheduler.run(CraftScheduler.java:97) - locked <0x000000077c7714b0> (a java.util.TreeMap) at java.lang.Thread.run(Thread.java:722) Locked ownable synchronizers: - None "Listen thread" prio=10 tid=0x0000000018b4e000 nid=0x15e0 runnable [0x0000000040cf5000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:396) at java.net.ServerSocket.implAccept(ServerSocket.java:522) at java.net.ServerSocket.accept(ServerSocket.java:490) at net.minecraft.server.NetworkAcceptThread.run(SourceFile:40) "Thread-4" daemon prio=10 tid=0x00000000189a4800 nid=0x15df runnable [0x0000000044088000] java.lang.Thread.State: RUNNABLE at java.io.FileInputStream.read(Native Method) at jline.Terminal.readCharacter(Terminal.java:99) at jline.UnixTerminal.readVirtualKey(UnixTerminal.java:128) at jline.ConsoleReader.readVirtualKey(ConsoleReader.java:1453) at jline.ConsoleReader.readBinding(ConsoleReader.java:654) at jline.ConsoleReader.readLine(ConsoleReader.java:494) at net.minecraft.server.ThreadCommandReader.run(ThreadCommandReader.java:23) "Server thread" prio=10 tid=0x00002aaab80e6800 nid=0x15de runnable [0x0000000043c87000] java.lang.Thread.State: RUNNABLE at java.util.HashMap.put(HashMap.java:401) at java.util.HashSet.add(HashSet.java:217) at com.nijiko.permissions.PermissionCache.cacheResult(PermissionCache.java:22) at com.nijiko.permissions.Entry.cache(Entry.java:255) at com.nijiko.permissions.Entry.has(Entry.java:243) at com.nijiko.permissions.Entry.has(Entry.java:229) at com.nijiko.permissions.Entry.hasPermission(Entry.java:195) at com.nijiko.permissions.ModularControl.permission(ModularControl.java:269) at com.nijiko.permissions.ModularControl.permission(ModularControl.java:256) at com.nijiko.permissions.ModularControl.has(ModularControl.java:249) at com.ChaseHQ.Statistician.StatisticianPlugin.permissionToRecordStat(StatisticianPlugin.java:156) at com.ChaseHQ.Statistician.EventDataHandlers.EDHPlayer.PlayerMove(EDHPlayer.java:106) at com.ChaseHQ.Statistician.Listeners.StatisticianPlayerListener.onPlayerMove(StatisticianPlayerListener.java:48) at org.bukkit.plugin.java.JavaPluginLoader$7.execute(JavaPluginLoader.java:286) at org.bukkit.plugin.RegisteredListener.callEvent(RegisteredListener.java:58) at org.bukkit.plugin.SimplePluginManager.callEvent(SimplePluginManager.java:332) - locked <0x000000077c7efc40> (a org.bukkit.plugin.SimplePluginManager) at net.minecraft.server.NetServerHandler.a(NetServerHandler.java:164) at net.minecraft.server.Packet10Flying.a(SourceFile:126) at net.minecraft.server.NetworkManager.b(NetworkManager.java:226) at net.minecraft.server.NetServerHandler.a(NetServerHandler.java:85) at net.minecraft.server.NetworkListenThread.a(SourceFile:105) at net.minecraft.server.MinecraftServer.h(MinecraftServer.java:451) at net.minecraft.server.MinecraftServer.run(MinecraftServer.java:361) at net.minecraft.server.ThreadServerApplication.run(SourceFile:422) "Thread-1" daemon prio=10 tid=0x00002aaab80d0000 nid=0x15d6 waiting on condition [0x000000004386c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at net.minecraft.server.ThreadSleepForever.run(SourceFile:73) Does anyone know how to read this, or see any glaring problems?
isn't it something with minecraft.net ? It happened again, and I can't log in to server "21:56:48 [INFO] Read timed out" same time connection to minecraft.net isn't working.... just a guess, really desperate now
hmm I could not get to game at all this time, downloaded the latest build 1024, checking what it will do, at least I could log in... :-/
This has nothing to do with minecraft.net, as it is happening when there is no problem with minecraft.net. This has been occuring very often (within an hour) on my bukkit, the latest rb 1000. This wasn't happening previously. Something has to give here, I am surprised there has been no clear definition of what is causing it.
Yes scratch the minecraft.net theory, might have been just coincident... version git-Bukkit-0.0.0-947-g9dc3fea-b1024jnks is holding for now, I'm going to sleep and hopes that it will be up when I wake up
"Server thread" prio=10 tid=0x00002aaab80e6800 nid=0x15de runnable [0x0000000043c87000] java.lang.Thread.State: RUNNABLE at java.util.HashMap.put(HashMap.java:401) at java.util.HashSet.add(HashSet.java:217) at com.nijiko.permissions.PermissionCache.cacheResult(PermissionCache.java:22) at com.nijiko.permissions.Entry.cache(Entry.java:255) at com.nijiko.permissions.Entry.has(Entry.java:243) at com.nijiko.permissions.Entry.has(Entry.java:229) at com.nijiko.permissions.Entry.hasPermission(Entry.java:195) at com.nijiko.permissions.ModularControl.permission(ModularControl.java:269) at com.nijiko.permissions.ModularControl.permission(ModularControl.java:256) at com.nijiko.permissions.ModularControl.has(ModularControl.java:249) at com.ChaseHQ.Statistician.StatisticianPlugin.permissionToRecordStat(StatisticianPlugin.java:156) at com.ChaseHQ.Statistician.EventDataHandlers.EDHPlayer.PlayerMove(EDHPlayer.java:106) at com.ChaseHQ.Statistician.Listeners.StatisticianPlayerListener.onPlayerMove(StatisticianPlayerListener.java:48) at org.bukkit.plugin.java.JavaPluginLoader$7.execute(JavaPluginLoader.java:286) at org.bukkit.plugin.RegisteredListener.callEvent(RegisteredListener.java:58) at org.bukkit.plugin.SimplePluginManager.callEvent(SimplePluginManager.java:332) - locked <0x000000077c7efc40> (a org.bukkit.plugin.SimplePluginManager) at net.minecraft.server.NetServerHandler.a(NetServerHandler.java:164) at net.minecraft.server.Packet10Flying.a(SourceFile:126) at net.minecraft.server.NetworkManager.b(NetworkManager.java:226) at net.minecraft.server.NetServerHandler.a(NetServerHandler.java:85) at net.minecraft.server.NetworkListenThread.a(SourceFile:105) at net.minecraft.server.MinecraftServer.h(MinecraftServer.java:451) at net.minecraft.server.MinecraftServer.run(MinecraftServer.java:361) at net.minecraft.server.ThreadServerApplication.run(SourceFile:422) Sounds like A permissions issue.
One thing I have found, and others have 'confirmed' on #bukkit to also seem to have worked: Reduce the amount of GCs you run! The more heavy you GC, the more likely you get time outs. Do not ask me why that is - I went from 4-5 Read Time Outs with some very very heavy GC setting to no timeouts with only a Concmark Sweep. No guarantees - but something you can test!
I ran two more stack traces and one of the few similarities was the Permissions caching. I've posted on the Permissions thread and I'm waiting for @rcjrrjcr to respond. @GmK To clarify I'm running cb1000 and tried the concmark sweep.
well 1024 just just timed out I'm starting to give up... I will try to run normal minecraft server and new map just to see if it is beacause of server ( it wasn't doing it before ) or because of craftbukkit.. (
@Kainzo Because you asked - mine doesnt seem to relate to Permissions: Code: Java stack information for the threads listed above: =================================================== "Timer-5": at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000006e0373b48> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290) at org.bukkit.craftbukkit.scheduler.CraftScheduler.cancelTasks(CraftScheduler.java:294) at net.erbros.PvPTime.PvPTime.checkTimeClock(PvPTime.java:146) at net.erbros.PvPTime.PlayerEventListener.onPlayerCommandPreprocess(PlayerEventListener.java:15) at org.bukkit.plugin.java.JavaPluginLoader$5.execute(JavaPluginLoader.java:272) at org.bukkit.plugin.RegisteredListener.callEvent(RegisteredListener.java:58) at org.bukkit.plugin.SimplePluginManager.callEvent(SimplePluginManager.java:338) - locked <0x00000006e029bfd8> (a org.bukkit.plugin.SimplePluginManager) at net.minecraft.server.NetServerHandler.handleCommand(NetServerHandler.java:706) at net.minecraft.server.NetServerHandler.chat(NetServerHandler.java:677) at org.bukkit.craftbukkit.entity.CraftPlayer.chat(CraftPlayer.java:148) at com.beecub.bCoolDown.bWarmUpTimer.run(bWarmUpTimer.java:29) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) "Server thread": at org.bukkit.plugin.SimplePluginManager.callEvent(SimplePluginManager.java:333) - waiting to lock <0x00000006e029bfd8> (a org.bukkit.plugin.SimplePluginManager) at net.minecraft.server.World.k(World.java:445) at net.minecraft.server.World.applyPhysics(World.java:432) at net.minecraft.server.BlockRedstoneWire.g(BlockRedstoneWire.java:49) at net.minecraft.server.BlockRedstoneWire.doPhysics(BlockRedstoneWire.java:286) at net.minecraft.server.World.k(World.java:453) at net.minecraft.server.World.applyPhysics(World.java:428) at net.minecraft.server.BlockRedstoneWire.g(BlockRedstoneWire.java:49) at net.minecraft.server.BlockRedstoneWire.doPhysics(BlockRedstoneWire.java:286) at net.minecraft.server.World.k(World.java:453) at net.minecraft.server.World.applyPhysics(World.java:433) at net.minecraft.server.BlockRedstoneWire.g(BlockRedstoneWire.java:49) at net.minecraft.server.BlockRedstoneWire.doPhysics(BlockRedstoneWire.java:286) at net.minecraft.server.World.k(World.java:453) at net.minecraft.server.World.applyPhysics(World.java:433) at net.minecraft.server.BlockRedstoneWire.g(BlockRedstoneWire.java:49) at net.minecraft.server.BlockRedstoneWire.doPhysics(BlockRedstoneWire.java:286) at net.minecraft.server.World.k(World.java:453) at net.minecraft.server.World.applyPhysics(World.java:428) at net.minecraft.server.BlockRedstoneWire.g(BlockRedstoneWire.java:49) at net.minecraft.server.BlockRedstoneWire.doPhysics(BlockRedstoneWire.java:286) at net.minecraft.server.World.k(World.java:453) at net.minecraft.server.World.applyPhysics(World.java:428) at net.minecraft.server.BlockRedstoneWire.g(BlockRedstoneWire.java:49) at net.minecraft.server.BlockRedstoneWire.doPhysics(BlockRedstoneWire.java:286) at net.minecraft.server.World.k(World.java:453) at net.minecraft.server.World.applyPhysics(World.java:428) at net.minecraft.server.BlockRedstoneWire.g(BlockRedstoneWire.java:49) at net.minecraft.server.BlockRedstoneWire.doPhysics(BlockRedstoneWire.java:286) at net.minecraft.server.World.k(World.java:453) at net.minecraft.server.World.applyPhysics(World.java:428) at net.minecraft.server.BlockRedstoneWire.g(BlockRedstoneWire.java:49) at net.minecraft.server.BlockRedstoneWire.doPhysics(BlockRedstoneWire.java:286) at net.minecraft.server.World.k(World.java:453) at net.minecraft.server.World.applyPhysics(World.java:433) at net.minecraft.server.BlockRedstoneWire.g(BlockRedstoneWire.java:49) at net.minecraft.server.BlockRedstoneWire.doPhysics(BlockRedstoneWire.java:286) at net.minecraft.server.World.k(World.java:453) at net.minecraft.server.World.applyPhysics(World.java:433) at net.minecraft.server.World.update(World.java:401) at net.minecraft.server.World.setData(World.java:343) at org.bukkit.craftbukkit.block.CraftBlock.setData(CraftBlock.java:62) at com.sk89q.craftbook.ic.ICUtil.setState(ICUtil.java:52) at com.sk89q.craftbook.ic.families.FamilySISO$ChipStateSISO.set(FamilySISO.java:82) at com.sk89q.craftbook.ic.families.FamilySISO$ChipStateSISO.setOutput(FamilySISO.java:120) at com.sk89q.craftbook.gates.logic.Clock.think(Clock.java:74) at com.sk89q.craftbook.ic.SelfTriggeredICMechanic.think(SelfTriggeredICMechanic.java:48) at com.sk89q.craftbook.MechanicManager.think(MechanicManager.java:446) at com.sk89q.craftbook.bukkit.MechanicClock.run(MechanicClock.java:33) at org.bukkit.craftbukkit.scheduler.CraftScheduler.mainThreadHeartbeat(CraftScheduler.java:137) at net.minecraft.server.MinecraftServer.h(MinecraftServer.java:438) at net.minecraft.server.MinecraftServer.run(MinecraftServer.java:361) at net.minecraft.server.ThreadServerApplication.run(SourceFile:422) Found 1 deadlock. Whole Pastebin: http://pastebin.com/RqkhSnXM
Looks like the scheduler, some kind of IC or PVPtime toggles - causing that.. what plugin for IC are you using?
For ICs we use Craftbook, for the PvP we use PvPTime - I for now disabled PvPTime and the bcooldown thing,both from the first part of the stack, and if that doesnt work will disable CraftBook. Let's see....
We were using bcooldown - it caused a few time outs so I yanked it. Unsure if that could be it, but its probably not running threadsafe.
@Kainzo I -FINALLY- fixed the "Read Timed Out" issues. It was the Permissions 3.x plugin. I rebuilt it with 'fair mode' and it's been running all night at around 1GB with no memory leaks. Here's a link to the thread post with more details and a download link (if you all want to try): http://forums.bukkit.org/threads/ad...in-of-tomorrow-935.18430/page-190#post-496631 Note: I haven't had -any- permission problems or "Read Timed Out" issues since updating.
I've got to agree, for my situation at least. I switched to PermissionsEx two weeks ago, and haven't had the error since (I used to get it every other day, at the longest interval). PermissionsEx is working out really well, and it's got some nice built-in commands and storage backends. It was also very easy to convert to it's format. It seems to be working for me, at the least.
Hey guys, I am having a very similar issue. For those of you that are suggesting that it is Permissions, I switched away from Permissions 3.x in favor Bukkits built in permissions. Any one having issues with those permissions?
For me was the solution to turn of falsebook IC and minecart mania... Server is now running 3 days without error using this plugins: DispenserButton, CreeperHeal, DreamLand, Essentials, BukkitSpeak, Jail, Permissions, DeathTpPlus, FalseBookBlock, WaterproofRedstone, SupernaturalPlayers, CraftBukkitUpToDate, WorldGuard, FalseBookExtra, Tombstone, War, Landmarks, DropClear, FalseBookCore, IgnoreLogger, iChat, WorldEdit, EssentialsSpawn, creaturebox, dynmap, LogBlock, WormholeXTreme
ehm... ^^ toght cool maybe a permissions fix i tried it the server spammed errors then the WHOLE server crashed (a vps) and now if i watch in my admin terminal to reboot it says unknown hardware error so maintenance... ehm... XD i tested it a while now (btw hoster had an ram dmg but i dont think its from my server on a virtual machine XD) and it realy solves the read time out... but it spamms java errors and freezes and it generaly freezes now on restart...???? so i have to choose now between freeze and crash XD
I'm getting this, but it seems to happen even when I run the vanilla server software from Minecraft.net.
I found that DeathTPPlus starts off fine and after a few minutes its CPU usage skyrockets. I removed this and my server feels like new.