I would like to request there be a plugin-specific file, located in a plugin's directory, that would be readable/writeable in the same manner as SystemNotes. This file would serve as the central repository for all data the plugin generates.

Example:
    vendetta/plugins/myplugin/plugindata.txt

Currently, generated plugin data is stored using the SystemNotes system, which is character-specific. There is a way to share SystemNotes between characters, but it's still not ideal, nor widely used. This requires players to manually sync SystemNotes files with data across their multiple characters.

Disconnecting generated data from the character would simplify the entire system.

Since it would utilize the same principles as SystemNotes, only two new API functions would need to be created for interacting with the single, specific file.

There should also be no negative effects from this, since the same thing is already happening with SystemNotes. In fact, this should have a positive effect on the plugin environment, since it wouldn't have to replicate data or have the player potentially working from two different datasets because they forgot to sync them.

I think this continues to be a source for a lot of extra haggling for plugin developers and should definitely be done.. it wouldn't hurt to have it also use an industry standard interchange format like XML or JSON.. it would seem that there are XML function already built into the API which makes that potentially low hanging fruit.

Besides the interchange format.. it is just really needed to have a file like this anyway.. it opens up a lot of functionality across characters with the plugins and just really makes a lot of things easier to do. I also can see no negative effects of this, only improves the tools the community has to help develop cooler stuff for the game!

+1

+1 anything to make the learning process for us new plugin dev or makes it easier for the vet devs to explain it to us, +1 again!

currently we only have system notes to store data in.. these are text files found in your vendetta folder and they are character specific.. meaning that any data you store there ...like notes about a system or asteroid locations stored by targetless or other.. these are not available to your other characters on the same account without some really shady hacking on the part of a plugin developer.. the kind of hacking we really shouldn't have to do, and that you don't want plugin devs doing..

having a file that is specific to the plugin itself instead of to the character will solve these issues. Then a plugin developer can decide if the data he storing is character specific (ie.. put it in system notes) or if it is data intended to be shared on the account (put it in plugin notes). It is of immeasurable value since enabling OS functions directly is a big no no.

example:
A guild specific plugin should store it's data in a place that is uniquely accessible by a single character on an account, since another character may be in a different guild.. Likewise a plugin that stores things like asteroid data or notes about whats available at the station would want to store that information where all characters on the account can benefit.

What protections would be in place to prevent a plugin from overwriting or reading another plugin's data? Are you expecting plugins to work together? In a similar vein, what about concurrent access where two plugins try to write or read at the same time?

Seems like you'd need some sort of JSON/XML DAO that will force separate objects per plugin. Or possibly it would be safer to allow plugins to write to a data file located in the plugin directory, maybe enforcing the file to have the same name as the directory or something to prevent spamming writing to the disk.

What protections are there with the existing SystemNotes system? None.

We've had this insecure data storage system since SystemNotes were implemented. While it would be nice to have it be protected in some way, I really don't see that being a blocking issue, considering where we're coming from.

One possible way to complicate things that might provide some security would be to implement an API function that could only be used at load time. This function would return two functions for reading and writing to the data file. This could work because at load time the plugin loader knows precisely which directory it's reading from. It would then be able to write the anonymous functions that would give access to only that file, and could only be obtained by another plugin if the plugin author stored it publicly or purposely shared it.

As for concurrent writes, that can't happen, since the VO Lua sandbox doesn't allow multiple blocks of code to run simultaneously.

For those that aren't familiar, here are the existing SystemNotes API function calls, and here are some proposed API function calls for the plugin data file.

LoadSystemNotes(<system id>)
SaveSystemNotes(<string>, <system id>)

LoadPluginData(<plugin dir>)
SavePluginData(<string>, <plugin dir>)

+1 we would be able to make more fun plugins & that's always a welcome addition

it is possible for devs to lock access to files based on some criteria. For instance, access to the files could be restricted by the underpinnings of the provided API functions. This suggestion does not include accessing a file with multiple plugins, but rather having a file that is accessible by an individual plugin specifically. The API could just assign a file with an arbitrary name to each plugin and it won't really matter where it is stored.

That being said, the whole system could be enhanced by even more by creating a "scoping" system of files. Such that there are certain files that are accessible in different ways.. files that are "global" or "plugin specific" or "character specific" or "account specific".. this is probably next level restricted access that gives more freedom to the plugin developer while continuing to protect the OS of users.

Furthermore, since access to files is restricted to API calls already.. and this suggestion is the same.. the API calls can have timers that restrict access frequency... say.. you can only call the associated read function every so many seconds, and perhaps you cannot hold the file open, and you can only write too it every so often.

As for concurrent access possibilities (though this is out of scope from the OP).. this is generally done on an OS using "file locking" where anything that wants access to a file must wait in line until the other accessor has the file unlocked. This is how it's done in operating system development, and the functionality is built into OS in general.

However, out of the questions about this that are listed by others in the thread so far.. all of them become somewhat moot once you are putting an API between the programmer and the OS function of reading and writing files. The implementer of the function (incarnate/raybondo) is in complete control of the rules for how/when it can happen. The plugin developer cannot use the LUA OS library to access files directly, that has not been built in to VO's LUA API

(just got home from a trip, i'll flesh out my post later)

...
Incarnate had some concerns about file access in modern designs when I suggested something somewhat relative, which can be seen here.

+1, give draugath whatever he needs within reason

JSON preferred over XML

I think also being able to save JSON structures server side would be very handy, then configurations could be portable.

bump

I'm hoping to hear some thoughts from inc on this..

I appreciate Incarnate's concerns, voiced in the thread Luxen linked. I also wonder, how much of the work could be done in series? If a suitable API could be formalized that will support the end goal, then implementing the most basic existing functionality would be a positive step forward. Additional restrictions could be added in later as time allows and need dictates.

Considering the voiced concerns about preventing abuse, perhaps the idea of a function only usable at load-time to get the relevant plugin-specific functions would be prudent.

I'm hoping to hear some thoughts from inc on this..

I'm not really sure what you're looking for me to add, that I didn't already say in Luxen's thread from last year.

I mentioned there was some "We're looking at doing some maintenance on the Lua API, before long." I was going to have us do that this past winter, but shifted that priority due to some other client issues.

But, it's still on the TODO, grouped with a bunch of other related changes.

If you want some kind of specific notes on what a storage API would look like, I don't have an answer right now. We'll kick it around internally and do whatever makes the most sense (for us) at the time. (I will try to take a look at this thread, again, before we do).

I am not currently promising anything definitive, other than "I understand your request".

I you want an ETA, I definitely don't have that. We have to get through the current mess of "unplanned server BS" before I can go back to my regularly-scheduled inaccurate ETAs.

I think also being able to save JSON structures server side would be very handy, then configurations could be portable.

If such a thing were to ever happen, it would be very limited in size (reasons previously stated). At the moment, this is just not a priority.

Late to this but want to add some things:

I thought everybody just symlinked their settings folders between all characters. That is possible on all platforms. My only wish would be for there to not be character-specific folders at all. There are very few things that I want to be character specific, and even the built-in system notes are not among them. Plugins that need character-specific data already prefix their settings with charid because of all of the above, anyway. And the game can store character-specific settings in config.ini already, so there really is no need for character-specific settings folders at all.

Furthermore, as a plugin developer, I would not want data to be stored in the plugin's directory itself, because that would make it harder or impossible for other plugins to access it (e.g. my Prospector plugin can import from and export to targetless - if all I had was were targetless' own functions I would impact the operation of that plugin and if it didn't have a global table at all it would be impossible).

The only problem there is for me - and that is not plugin specific - is lack of built-in device-sync, but the implications for a system allowing storing arbitrary amounts of data are obvious so I'm not even asking for that. Besides, it can be handled with other software that you use to sync data between devices.

Snib, keep in mind that there were two different levels of access for the file being discussed here. One was the same level of access that System Notes gives us already. The other was a protected mode for that plugin only, though potentially with the ability to share access with other plugins.

If the access is the same as the System Notes system, then you wouldn't lose any degree of access compared to what you have now. The major benefit, either way, is storing the data in an obvious location that can be easily backed-up or transferred to another installation.

> The major benefit, either way, is storing the data in an obvious location that can be easily backed-up or transferred to another installation.

It's much easier to back up just the single Settings directory we already have as opposed to backing up individual files in a lot of subdirectories.

Furthermore, putting it into the plugin's directory itself is just too problematic from an end-user point of view, you'll end up with users accidentally deleting or sharing their data. Separating code and data is usually a good idea.

That said, I'm not opposed to a separate <install_location>/plugin_data/<pluginname>/ structure. That would indeed be superior to the current system insofar as we would not have to worry about file name collisions. I'm still opposed to just letting us save a single file in there though, that's super inefficient for plugins that work with a lot of data (basically targetless vs Prospector approach).

Since it's been awhile, and I've had my own recent experience with using multiple files for breaking up large quantities of data, I just wanted to bring to the fore this sentence from Snib.

Snib said:
I'm still opposed to just letting us save a single file in there though, that's super inefficient for plugins that work with a lot of data (basically targetless vs Prospector approach).

I definitely agree that supporting multiple files per plugin, in some way, would be beneficial.

Off the top of my head, I might be more in favor of virtualizing the concept of file access from the plugin sandbox. I'm not sure that individual "new" file-creation will always be a possibility by the client, particularly as different platforms become more and more restrictive.

A single compressed archive that the client writes to, however, and allocates "chunks" to plugins via some virtualized API methodology, might be better long-term.

But, I don't know, we'd have to dig into the feasibility.

One thing to keep in mind, which is only alluded to: plugins that save large quantities of data (specifically asteroid DBs) have lately taken steps to distribute the data across multiple system notes files to reduce the cost of a write command while flying around.

I'm concerned that such a single compressed archive might be more restrictive from an I/O standpoint than what we have now.