[Simple Benchmark] CoreProtect vs LogBlock

Discussion in 'Bukkit Discussion' started by TheBeast808, Jan 5, 2013.

?

Which block logger do you think best supports a lag -free server?

LogBlock 34 vote(s) 54.0%
CoreProtect 20 vote(s) 31.7%
Hawkeye 9 vote(s) 14.3%
  1. Offline

    TheBeast808

    dev.bukkit.org profile:
    CFUSERNAME
    My Plugins (CFCOUNT)
    Minecraft account:
    MCUSERNAME
    EDIT: I did some proper profiling using Java VisualVM. Results are below and confirm what I found in my first post.


    I'm always looking for ways to improve my server's performance, and I heard CoreProtect was supposed to be the best. I am a long time user of LogBlock and was reluctant to switch, but I still wanted to check what performance differences there were.

    Methodology: To test the two loggers, I setup a stripped down test server. The two loggers would not run at the same time to avoid possible interference between the two plugins. Both would be setup to use MySQL, but other than that, their configuration files would be left completely default. Between the tests, the plugin files would be swapped out, mysql would be cleaned, and the server would got through a complete restart. To actually test the loggers, I made a plugin that would generate 50,000 BlockBreakEvents as fast as it could. My reason for choosing the BlockBreakEvent was that it was the most common on my public server and it was the easiest to generate with the stress tester. The BlockBreakEvents would all be using my player, so I would have to be logged in at the time of the test. As for the target block, it would be randomly selected from within -300,256,-300 and 300,0,300 for each event. After each logger had time to process the 50,000 blocks, I would use a /timings merged to get the results.

    Results:
    CoreProtect: Trial 1, Trial 2, and Trial 3
    LogBlock: Trial 1, Trial 2, and Trial 3

    Interpretation: The benchmark was a very simple one, and although it's not conclusive, it raises doubts in my mind towards CoreProtect being "fast, efficient, data logging and anti-griefing tool." The benchmarks say nothing about how much work was done in total by each logger. My guess is that LogBlock's event times are so low because the majority of its work is done in another thread after it is removed from queue. I have not read any of CoreProtect's source code and have no idea what system it uses to process block events and store them, so I cannot speak as to what it is doing to make its timings so high. Regardless, I think I speak for most admins when I say that we want to keep as much work off of the main server thread as possible. My interpretation of the logs is that LogBlock is the best in this regard, as it does not do very much work in the main thread.


    The benchmarks I used were very rudimentary. Perhaps somebody out there with more experience profiling plugins in detail can do better benchmarks and tell us once and for all which logger is the best.

    Edit: here is the most up-to-date benchmark data I have(Feb 28):
    [IMG]

    This post has been edited 2 times. It was last edited by TheBeast808 Mar 1, 2013.
    Rmarmorstein, zipfe and jorisk322 like this.
  2. Offline

    np98765

    dev.bukkit.org profile:
    CFUSERNAME
    My Plugins (CFCOUNT)
    Minecraft account:
    MCUSERNAME
    I've always wondered how much more "efficient" CoreProtect is than LogBlock. I'd be interested in viewing others' benchmarks, too, if anyone else is willing to.
  3. Offline

    TheBeast808

    dev.bukkit.org profile:
    CFUSERNAME
    My Plugins (CFCOUNT)
    Minecraft account:
    MCUSERNAME
    I'll be testing flatfile CoreProtect vs LogBlock in the morning to see if it makes a difference on the main thread.
  4. Offline

    TheBeast808

    dev.bukkit.org profile:
    CFUSERNAME
    My Plugins (CFCOUNT)
    Minecraft account:
    MCUSERNAME
    Just got done with the CoreProtect flatfile tests. Actually seems like it got slower using flatfile over mysql. Anyways, here are the results: trial 1, trial 2, trial 3, trial 4
  5. Offline

    mbaxter ʇıʞʞnq ɐ sɐɥ ı Administrator

    dev.bukkit.org profile:
    CFUSERNAME
    My Plugins (CFCOUNT)
    Minecraft account:
    MCUSERNAME
    Coreprotect has some seriously inefficient handling of events. Last I checked it created a new thread for some events, each time it fired, which is horribly inefficient.

    Additionally, it makes asynchronous calls to methods that should never be called asynchronously, risking a server crash.
    Comphenix likes this.
  6. Offline

    TheBeast808

    dev.bukkit.org profile:
    CFUSERNAME
    My Plugins (CFCOUNT)
    Minecraft account:
    MCUSERNAME
    Just read the source to the BlockBreakEvent handler. Some of the worst code ever. I've heard that CoreProtect is slightly obfuscated, and I sure hope that was the obfuscator. I could eat alphabet soup and shit an event handler better than that. There were a ton of things that were just completely unnecessary and that likely took up a significant amount of resources.
  7. Offline

    asofold

    dev.bukkit.org profile:
    CFUSERNAME
    My Plugins (CFCOUNT)
    Minecraft account:
    MCUSERNAME
    I think one might need use a million events for a reliable statement, using java :p
  8. Offline

    Milkywayz

    dev.bukkit.org profile:
    CFUSERNAME
    My Plugins (CFCOUNT)
    Minecraft account:
    MCUSERNAME
    I myself use logblock would not just switch over to coreprotect just because they advertise it to have performance in mind.

    I am like you, I try to improve the performance anyway I safely can, without drastically changing the server. However I would not switch simply because the logblock commands are more familiar (and to me, a lot easier).
  9. Online

    evilmidget38 BukkitDev Staff BukkitDev Staff

    dev.bukkit.org profile:
    CFUSERNAME
    My Plugins (CFCOUNT)
    Minecraft account:
    MCUSERNAME
    Always nice to see a plugin comparison that uses actual tests rather than, "XX is better than YY because (some irrelevant and incorrect "fact")."
    MaliciousMan and np98765 like this.
  10. Offline

    TheBeast808

    dev.bukkit.org profile:
    CFUSERNAME
    My Plugins (CFCOUNT)
    Minecraft account:
    MCUSERNAME
    Finally decided to do some proper testing. I'll be honest, I was a bit too lazy to set each up and isolate them. I loaded both in the same server instance to do the testing, so interference is possible but not probable.

    Just like before, 50,000 block break events were generated. But this time, instead of using Bukkit's built in plugin profiler, I used an external profiler called Java VisualVM.
    [IMG]

    Once again, I found that coreprotect took many magnitudes more CPU time than LogBlock. The picture above shows methods after I used a regex to filter out all non-LogBlock and non-CoreProtect methods from the profiler. Unfiltered, CoreProtect's onBlockBreak() still dwarfs all other methods run at the time. It is 1000% more intensive than the next leading method -- sun.net.www.http.KeepAliveCache.run(). How does this compare to LogBlock? CoreProtect's block break handler used 14575% more time than LogBlock's.

    Here are the downloads to .csv files of the results of the profile, if you want to look at the full results:
    Unfiltered
    Filtered

    Now I just need to find somebody to explain to me why in the hell CoreProtect has a reputation for being fast!
    keelar and -_Husky_- like this.
  11. Offline

    midnightfang22

    dev.bukkit.org profile:
    CFUSERNAME
    My Plugins (CFCOUNT)
    Minecraft account:
    MCUSERNAME
    Rumors based on hopes and dreams?
    MaliciousMan and blorp6161 like this.
  12. Offline

    Intelli

    dev.bukkit.org profile:
    CFUSERNAME
    My Plugins (CFCOUNT)
    Minecraft account:
    MCUSERNAME
    Things where users actually see the end results (ex: performing rollbacks) are definitely faster. I never claimed stuff like actual logging was faster.

    As well, unlike some "other" plugins, CoreProtect logs blocks (ex: torches) that fall off other blocks. This means extra checks are done in the block break event. You can disable this by disabling "natural-break: true" in the configuration file.

    As well, as mentioned, CoreProtect does launch a new thread every time the event runs. We're slowly discontinuing this method of logging in favour of an alternating consumer (which it already uses for liquid flow logging)
    johnnywoof likes this.
  13. Offline

    TheBeast808

    dev.bukkit.org profile:
    CFUSERNAME
    My Plugins (CFCOUNT)
    Minecraft account:
    MCUSERNAME
    Actually, you said it was a fast logger.
    "CoreProtect is a fast, efficient, data logging and anti-griefing tool. Rollback and restore any amount of damage. Designed with large servers in mind, CoreProtect will record and manage data without impacting your server performance."
    "Fast efficient data logging."
    "Multi-threaded to ensure server performance is never impacted"

    But that's besides what I want to say. What do you think would be a fair test for rollbacks? First to complete a rollback? Most efficient rollback?
    Just ran the tests with natural-break set to false. It would appear that having this off gives a marginal boost with flatfiles, but a significant boost when using SQL. No natural-break and SQL got the main onBlockBreak down to around 25,000ms.
    Natural-breal: false, flatfile
    Natural-breal: false, SQL
  14. Offline

    OPJewelz

    dev.bukkit.org profile:
    CFUSERNAME
    My Plugins (CFCOUNT)
    Minecraft account:
    MCUSERNAME
    What I've heard is that LogBlock is faster and more lag free as it uses MySQL. It stores the information in that database instead of files. In MySQL it always keeps the file open "ish". If the config files where open in coredata it would be fast too. Also you don't end up with 5GBs of files when using LogBlock. I would really recommend using LogBlock. I love it
  15. Offline

    Mortalsinn

    dev.bukkit.org profile:
    CFUSERNAME
    My Plugins (CFCOUNT)
    Minecraft account:
    MCUSERNAME
    Core Protect uses MySQL as well.....
  16. Offline

    OPJewelz

    dev.bukkit.org profile:
    CFUSERNAME
    My Plugins (CFCOUNT)
    Minecraft account:
    MCUSERNAME
    Oh sorry, I thought about CoreData -.-
  17. Offline

    Mortalsinn

    dev.bukkit.org profile:
    CFUSERNAME
    My Plugins (CFCOUNT)
    Minecraft account:
    MCUSERNAME
    The thing I dislike about CoreProtect is it really has no safety features. With logblock you can limit the amount that can be rolled back at once. With CP, a pissed off moderator / admin could roll back the entire map if he wanted to.
  18. Offline

    Intelli

    dev.bukkit.org profile:
    CFUSERNAME
    My Plugins (CFCOUNT)
    Minecraft account:
    MCUSERNAME
    If flatfile logging versus MySQL logging is making a difference in your benchmarks, that means they're invalid. All logging is done in secondary threads and does not impact the performance of the main server thread whatsoever.
  19. Offline

    chiisana

    dev.bukkit.org profile:
    CFUSERNAME
    My Plugins (CFCOUNT)
    Minecraft account:
    MCUSERNAME
    Not sure how your result differ so substantially, however, I'm going to go on a whim and say there are probably other factors causing errors in the result. The natural-break thing are all handled in the main `onBlockBreak` event handler, prior to entering the logging part of the code (`Functions.log_break`). So the affect it should have on the `onBlockBreak` event handler should be the same between logging methods. If you are seeing 48kms and 25kms in the same function doing the same thing, then there are probably other factors slowing everything down (or, speeding everything up) on the test environment.

    This post has been edited 1 time. It was last edited by chiisana Jan 15, 2013.
  20. Offline

    TheBeast808

    dev.bukkit.org profile:
    CFUSERNAME
    My Plugins (CFCOUNT)
    Minecraft account:
    MCUSERNAME
    Alright, I'm going to get a larger sample size to lower the uncertainty.
  21. Offline

    asofold

    dev.bukkit.org profile:
    CFUSERNAME
    My Plugins (CFCOUNT)
    Minecraft account:
    MCUSERNAME
    One millisecond per blockbreak event would be a very high value, together with high level enchantments or skill use it might add up to something noticable. Indeed the number of events is not so high, so a larger sample size could be good.

    Should be important to have the same testing conditions, though.

    One question is if to disable the feature in CoreProtect which logs indirectly broken blocks like torches, just for the case that LogBlock does not log those.
    Then the block breaking should have the same conditions (environment).
    And the database setup has to be comparable.
  22. Offline

    Intelli

    dev.bukkit.org profile:
    CFUSERNAME
    My Plugins (CFCOUNT)
    Minecraft account:
    MCUSERNAME
    Two things. Are you ensuring that the chunks the blocks are in are loaded first before passing the events to the plugins? Secondly, are you ensuring that you're not passing air blocks to the plugins to be logged as being broken?
  23. Offline

    Intelli

    dev.bukkit.org profile:
    CFUSERNAME
    My Plugins (CFCOUNT)
    Minecraft account:
    MCUSERNAME
    Did my own tests, and CoreProtect is using around 8000ms over 50,000 iterations, versus 2000ms for LogBlock.

    To note, LogBlock splits things over a number of different functions (called within the primary thread).

    I've identified the cause of the problem though. It'll be fixed in an upcoming release of CoreProtect ;)
  24. Offline

    zipfe

    dev.bukkit.org profile:
    CFUSERNAME
    My Plugins (CFCOUNT)
    Minecraft account:
    MCUSERNAME
    Interesting thread, and it even produced a result.
  25. Offline

    tanveergt5

    dev.bukkit.org profile:
    CFUSERNAME
    My Plugins (CFCOUNT)
    Minecraft account:
    MCUSERNAME
    use coreprotect for a month then laugh at the coreprotect directory which is over 700mb LOL
    I switched back to Hawkeye reloaded, does the job, has what I need and a nice web gui.
  26. Offline

    TheE

    dev.bukkit.org profile:
    CFUSERNAME
    My Plugins (CFCOUNT)
    Minecraft account:
    MCUSERNAME

    This post has been edited 3 times. It was last edited by TheE Jan 20, 2013.
  27. Offline

    darkhelmet

    dev.bukkit.org profile:
    CFUSERNAME
    My Plugins (CFCOUNT)
    Minecraft account:
    MCUSERNAME
    Sounds interesting. I'm never satisfied with our performance but honestly always felt coreprotect was really slow from a visual perspective. We've been doing some unofficial benchmarking with Prism and with an average server (with 512M RAM, 2.7Ghz dual core processor) 6,886 blocks from a tnt explosion restored in 2.2 seconds - 2210 milliseconds from command entry to the last block set in rollback completion. It's fast. I've checked our memory a bit to make sure it's not insane and from early tests it looked good but that needs more examination.

    Query-related stuff in Prism is done entirely by a separate thread and we also use multiple inserts-per-connection to reduce overhead. Essentially, even with 10k blocks exploded from a tnt chain reaction, the recorder will not lag the server. Block changes are all queued and processed by a sync task in batches of 1000 (but so fast the player never knows they're in batches).

    One thing I think needs attention too is database design. It's always difficult to balance a smart, efficient design with a simple-to-manage system that winds up in usually-learning server owner's hands. However, the hawkeye AND coreprotect database designs are seriously in need of attention. Adjusting the crazy column formats/lengths in hawkeye saved us 1GB in data storage for the same contents, and why cp needs four tables and coordinates stored as a single string is beyond me.

    And last I checked, neither of those used indexes on the db either.
    evilmidget38 likes this.
  28. Offline

    darkhelmet

    dev.bukkit.org profile:
    CFUSERNAME
    My Plugins (CFCOUNT)
    Minecraft account:
    MCUSERNAME
    P.S. I'm officially voting for Prism, which isn't on the list ;)
    Milkywayz likes this.
  29. Offline

    darkhelmet

    dev.bukkit.org profile:
    CFUSERNAME
    My Plugins (CFCOUNT)
    Minecraft account:
    MCUSERNAME
    I was working on our sqlite support yesterday and did some more tests on mysql performance. At least on that same dev machine mentioned previously, a 6,307 block rollback (from the moment the command was entered to the moment the last block was in place from the rollback) took 1633 milliseconds (1.6 seconds) to complete.
  30. Offline

    TheBeast808

    dev.bukkit.org profile:
    CFUSERNAME
    My Plugins (CFCOUNT)
    Minecraft account:
    MCUSERNAME
    I'm not dead! but I have been busy.
    I recoded my event generator to incrementally generate the loads of events(currently set to 5000/30 seconds). I test-test(yes, a test to test the testing) that seemed to indicate CO with SQL and no natural logging was about what the last test indicated. But, before I was able to do any tests I'd want to upload I became very busy. I'll try to do the tests soon, and maybe try to benchmark other loggers.
    XanderCrews01 likes this.

Share This Page