Persistence - How to get it work properly

Discussion in 'Plugin Development' started by DjDCH, Apr 17, 2011.

Thread Status:
Not open for further replies.
  1. Offline

    DjDCH

    Hi everybody,

    I have my plugin HiddenDoor that I need to update, but I have some problem with my data storage handling. I currently use a Serializable class to store the data, but I have some issue when recompile the jar and unserialize data generated with the previous jar.

    Few months ago, I have eared about the Persistence plugin and I have thought to using it. But, I have recently eared that Bukkit have now its own persistence handling, called EBeans. Since the Persistence is now inactive, I think it might be a good idea to use the embed persistence handling of Bukkit. However, I didn't find any information or documentation about it.

    Since I need to store Block location distribute in several worlds, I want to know the best way to store this data. If EBeans is the best way, I would like to know how to use it. If some documentation is available, link-me to it.

    Thanks in advance for your help.

    DjDCH
     
  2. Offline

    FrozenBrain

    Hi,

    Sammy made a tutorial about persistance (*click*). If you want, you can have a look at HomeBukkit as an very basic example, too.

    - Frozen
     
  3. Offline

    Mixcoatl

    In my humble opinion, EBeans is the correct way to persist this information. More than likely, in the process of converting to EBeans you will actually reduce your total code size and complexity. It's a nice little bonus. It will require adding some annotations to your persistent classes though.
     
  4. Offline

    Sammy

    @Mixcoatl I have the same opinion, but the lack of info about Ebeans makes some key features hard to understand.

    For example, if you are using sqlite, @OneToOne @OneToMany @ManyToMany annotations don't seem to works...

    But the real problem is adding new columns to a already started table (maybe I want to add a new feature to my plugin without having to make a new db file).
     
  5. Offline

    Mixcoatl

    Things are rarely easy, but we can reach "there" from "here." :)
    Are you sure about this? I have not, myself, worked with the join relationship annotations, but they're JPA-standard. Can you provide more information? Maybe we can get it to work. :)
    No kidding! I have run into this one, myself! In fact, I'm holding off on making a change to Socials! because of this very problem. After playing with the database setup code for several hours, the best I could come up with is the following manual procedure:
    1. Stop your Bukkit server.
    2. Use SQLiteAdmin or an equivalent tool to open the SQLite database file.
    3. Export each table to a .CSV file using the tool's export options.
    4. Close SQLiteAdmin (or equivalent.)
    5. Remove the SQLite database file entirely (right-click and delete in Windows or rm -f <filename> in Linux.)
    6. Start your Bukkit server to re-create the SQLite database file.
    7. Stop your Bukkit server again.
    8. Use SQLiteAdmin (or equivalent) to open the new SQLite database file.
    9. Import each table from the .CSV files you created in step #3 to reload your data. You will have to manually map the column names and discard any column values whose types have changed.
    10. Close SQLiteAdmin (or equivalent) again.
    11. Start your Bukkit server again (again).
    At some point, I plan to document the above procedure with screen captures or a video. Perhaps when I make the next update to Socials! Anyhow, I suspect this is a problem with MySQL, too. It is either a weakness in EBeans' DDL implementation or a weakness in how Bukkit bootstraps the DDL code.
    Also: you can download SQLiteAdmin here.
     
  6. Offline

    Sammy

    Quoting RightLegRed
    Well, I tried and tried other join relationships and just resigned myself to believe the awful truth: :'(
    BUT, @Embedded and @Embeddable works great
    What I don't understand is that Update, call an other slq functions are implemented on Ebean, but Alter Table doesn't seem to be
     
  7. Offline

    Mixcoatl

    Read the bug carefully: "[...] when using the @OneToMany and @ManyToOne annotations on a Set, ebeans fails when using SQLite. [...]"
    Try using a List instead of a Set and this should work.
    Unfortunately, @Embed and @Embeddable are utterly useless for creating join tables or lists of records. These are exclusively for when you want two discreet entities to persist their fields in the same database record.
    Sir, you actually seem to understand the problem quite well! The DDL logic does not modify the database schema when it detects the class has changed. It can certainly tell that the schema differs. My only real question is whether this is the responsibility of the client code (e.g. Bukkit) or the EBeans API to perform. We've had to implement this very logic in a project at work, but we're using ADO.NET in C# at the moment.
     
  8. Offline

    Sammy

    Unfortunately I tried List too and no cake was given to me :confused:
    True, but it was the only useful annotation that worked for me.
    Lets call @RightLegRed , maybe he has some tips for us

    PS: I really liked Socials, awesome idea man... and very very beautifully coded :D
     
  9. Offline

    Mixcoatl

    Do you know that you may need to specify an @Join annotation as well?
    I gave someone else some pointers on getting this working and, supposedly, they were able to do so.
    @Embed and @Embedded are very useful for very specific things! It's not "bad" it's just not what we really want!
    I'll see if I can find the thread where we were discussing persisting lists of data. The plug-in was Lock... something. I forget. I'll see if I can find it again.
    Thanks. Code is my art. :) I just released another update an hour or so ago that fixes the capitalization. Little things like that really bother me, but it wasn't important enough to hold up a release. Now that's resolved and I can move on to more serious stuff!

    So, the thread is here, and you've already weighed in on it. I don't know how I missed the entire tail end of that conversation. I just have "unwatched" it or something silly like that.

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 13, 2016
  10. Offline

    DjDCH

    Thanks to both of you. I will take a look to this.
     
  11. Offline

    Sammy

    @Mixcoatl <-- Does this Javax Annotation work ? looool
    Quoting DThielke:
    Well, if what he says is true, Im going to switch to MySql and try
     
  12. Offline

    Mixcoatl

    Yes. It turns your class into a n00b. :eek:
    There are a lot of good reasons to use MySQL, when you can. For operators who run a local server, or a very small server, it can be a pain in the posterior region. There just aren't any other viable options at the moment.
     
  13. Offline

    Sammy

    Well, all my plugins are for personal use (I'm now making my first public one ), so I think that we'll switch to mySQL
     
  14. Offline

    Mixcoatl

    I had a bunch of plug-ins I did for my own personal edification under hMod. I've made a handful of cute or interesting plug-ins for Bukkit, but nothing before Socials! that I thought would spark enough interest to be worth releasing.
     
  15. Offline

    Sammy

    I putted mine on the WIP forum today, but it didn't caused that much hype... I'm going to release it never the less, just for the sake of having a public plugin ^^
    It was my first Java program, now I'm recoding it for minecraft/bukkit

    The "cooler ones" I prefer to keep under my server only :rolleyes:
     
  16. Offline

    NathanWolf

    Wow, this thread got _long_ before I got a chance to see it.

    @DjDCH- Basically, there is now (from what I understand) a persistence engine built into Bukkit. I'm hoping that I (or someone else) refactors the Persistence plugin to use it- in the meantime, it's probably a better idea to use the built-in ORM rather than start new with Persistence.

    Also, @WinSock recently refactored my CrowdControl plugin from using Persistence to EBeans- this is probably a good example, I haven't gotten a chance to look at it yet.

    See the diffs here.

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 13, 2016
  17. Offline

    phaed

    No need to go through these lengths. Haven't tested using SQLite but with MySQL you can add new properties/columns to your classes/tables quite easily. The key is modifying them on the same place.

    Say I want to add a velocity property to my class.

    First add the new property in the class:
    Code:
    ...
    int height;
    int velocity;
    ...
    
    Then alter the table adding the column in the correct place:
    Code:
    alter table myclass add velocity int(11) after height
    
    Remove the nulls:
    Code:
    update myclass set velocity = 0
    
    Start your server. And that's about it.
     
  18. Offline

    Mixcoatl

    We use quite a bit of dynamic SQL at work, with MSSQL, Oracle, Informix, Firebird, SQLite, and proprietary RDBMSes. We use MySQL internally for infrastructure applications that require a database.

    I just chose not to discuss altering the database schema to keep the procedure simple. [Plus, SQLite supports a very, very limited ALTER TABLE syntax that I really didn't want to have to troubleshoot when things go haywire.]
     
  19. Offline

    phaed

    Cognitive dissonance and name dropping aside, this was the question posed (subject: changing schema post-fact):
    And this is how you answered (subject: delaying a schema change to social because of inability to do it easily):
    I get it, you get your kicks out of trying to be better than everyone else. It might do you good to take a tip evey once in a while.

    And while we're on the topic of name dropping, we use quite a lot of dynamic SQL, with MSSQL, and Oracle at SapientNitro where I work as a senior developer and part of team which builds all the sweepstakes sites for MARS candy (Snickers, M&M, Twix, etc, etc) dealing daily with databases that get hit with numbers that you wouldn't believe. And I choose to discuss altering the database schema to keep the procedure simple.
     
  20. Offline

    Mixcoatl

    Seriously? I mean... seriously? :confused:
    That's correct, and it remains correct in spite of your insistence otherwise.

    The exact nature of this problem, which I considered inappropriate for this thread, comes in several pieces which I'll now describe so we're on the same page.
    1. SQLite's ALTER TABLE syntax is extremely limited. It supports adding columns but does not support removing or altering their types (or certain constraints.) This inability to alter the type of a column is where it really breaks down for me. [Of course, in MySQL this would not be an issue.]
    2. Bukkit's Avaje eBeans integration is missing hooks to re-build your database schema on the fly based upon differences that it detects between the entity model and the database schema. There is sufficient information here to "fix" the schema, but it requires stepping outside of the framework to do it [and SQLite still does not support parts of what I would need; even with MySQL instead of SQLite, the appropriate hooks are still missing in Bukkit to correctly fix the database schema.]
    So, to recap: the problem I am trying to solve is not that I can't modify my own columns in my own development environment, it's that I can find no way -- within the existing frameworks -- to modify the end user's schema without manual intervention.

    I could step outside of these existing frameworks and shoehorn it in if i wanted, but this is ugly and, quite frankly, I don't like it at all.

    Your procedure works beautifully for server operators who are on MySQL and have sufficient SQL background to alter their own tables. I just don't consider this the lowest common denominator that I must support.
    Now you're actually kinda funny.:D
    Including tips completely unrelated to specific issues I'm trying to resolve, that I've researched thoroughly and already discussed to death?

    Look, if you have a solution that will work, I would be delighted to use it, and delighted to credit you with it. The only recommendation you've given thus far is to tell server operators to switch to MySQL and then tell them to change the schema manually, which is not at all acceptable (in my humble opinion.)
    Well, crap. I only work with universities, marine port authorities, airports, oil rigs, banks, the U.S. Coast Guard, the U.S. Navy, FEMA, the FAA, the DOD, and other government agencies. :( I guess I really do suck. :(
    Then it stands to reason that your target audience is much different than mine. Simple for me is not necessarily simple for my target audience. I cannot consider it acceptable to require server operators to enter SQL commands to fix something the plug-in should be aware of.
     
  21. Offline

    phaed

    Fair enough.
     
  22. How do I limit my results to let's say the first five.
    I want to show a toplist of the 5 players with the highest level on my server and normally I would do a SQL Query with "OrderBy" Select "name" "level" and then "LIMIT" but I could find anything like that in persistance.

    Is it possible in any way? Or do I have to switch back to Standard MySQL and do a Query?

    Thanks in advance
     
  23. Offline

    phaed

    Code:
    PagingList<Player> pages = plugin.getDatabase().find(Player.class).orderBy("level").findPagingList(5);
    
    if (pages.getTotalPageCount() > 0)
    {
        List<Player> players = pages.getPage(0).getList(); //top 5
    }
    Source: http://www.avaje.org/doc/ebean-userguide.pdf (Page 30)
     
  24. Offline

    Acrobot

    Okay, so my question:

    I have a field "buyPrice", and is it possible to generate average from, for example:
    plugin.getDatabase().find(Transaction.class).where().eq("itemID", itemID).eq("buy", 1);
    ?

    I currently do it in a loop, but I feel it is inefficient.
     
  25. Offline

    phaed

    Read: http://www.avaje.org/doc/ebean-userguide.pdf (Page 11)
     
  26. Offline

    Acrobot

    @phaed
    Yeah, I have already read that, but I don't completely understand. Should I make something like

    Code:
    @OneToOne
    Transaction transaction;
    Double averagePrice;
    
    And then, somehow, use RawSql?
    If you could help me with RawSQL, I would be very, very happy :)

    I have Transaction class that looks like that:
    https://github.com/Acrobot/iConomyC...com/Acrobot/iConomyChestShop/Transaction.java

    My price is stored in "float price", and my amount of items bought/sold is stored in "int amount".

    How can I, using RawSQL, get average price per item from every transaction?
    I currently do it in forEach loop, where "t" is Transaction:
    t.getPrice()/t.getAmount();

    So, can I do this using RawSQL?

    Something like avg(price/amount)
     
  27. Offline

    phaed

    Ahh. I wouldn't do this with ebeans, add a getAverage method to your Transaction object:

    Code:
        public double getAverage() {
            return this.price/this.amount;
        }
    
    Pull your transactions, then do the calculation:
    Code:
    List<Transaction> ts = plugin.getDatabase().find(Transaction.class).where().eq("itemID", itemID).eq("buy", 1).findList();
    
    for (Transaction t : ts)
    {
        double ave = t.getAverage();
    }
    
    You should probably also remove all your @NotNull annotations from all primitive data types (int, long, boolean) as these cannot be null in the first place.
     
  28. Offline

    Acrobot

    @phaed
    Thanks :)
    I'll keep that in mind!
     
  29. Offline

    axefan

    I've been using the persistence engine built into Bukkit for a new plugin and everything was working fine until I tried to save a List of objects with more than 10,000 items.

    The problem is that no exception is raised and the return value of EbeanServer.save(Collection) indicates that all objects were written. But when querying the table or browsing the db, not all objects are present! So there are missing records with no indication that the records were not written.

    I've created a simple plugin for testing here:
    https://github.com/axefan/DatabaseErrorDemo

    Also, I started a thread to discuss this issue here:
    http://forums.bukkit.org/threads/bug-in-ebeanserver-save-collection.17692/

    This built-in EbeanServer object is handy and seems to be fast, but can we trust it? Is anyone else seeing what I'm seeing?
     
Thread Status:
Not open for further replies.

Share This Page