Anyone here know how to create a Database and a Parser?

I know.. random and out of the blue, but I want to know if anyone has the know-how on this.

What kind (MS Access, MySQL, etc.)? What language (php, perl, ruby, etc.)? For what purpose? To be integrated how?

dnar and myself are both IT people. I have done databasing before. What exactly are you looking for?

I want to create a simple, yet powerful, program to take a Spy Report (for starters) and store the information until called upon by a Query which would then display any relevant information.

Lets say you take

Resources at Alyxan [2:228:6] (Players 'Tiyami') on 06-21 18:04:41

Metal: 21.768 Crystal: 12.252
Deuterium: 965 Energy: 1.718

Activity
Your espionage does not show abnormalities in the atmosphere of the planet. There appears to have been no activity on the planet within the last hour.
fleets
Defense
Building
Metal Mine 16 Crystal Mine 15
Deuterium Synthesizer 5 Solar Plant 17
Robotics Factory 5 Shipyard 2
Metal Storage 4 Crystal Storage 2
Deuterium Tank 2
Research
Espionage Technology 4 Computer Technology 2
Weapons Technology 5 Shielding Technology 6
Armor Technology 7 Energy Technology 6
Combustion Drive 6 Impulse Drive 5
Laser Technology 6 Ion Technology 2
Astrophysics 3
Chance of counter-espionage: 0 %

Store that info then query for metal > 20,000 and select for no fleets or something.

I want to have a link to the original report come up from that.

I don't know where to begin. I assume I would first have to create a program to 'read' the pasted information first.... then create something to display the results... then actually hand create the variables... but I want to learn how jus need some ideas and pointers first.

While not too hard, it does require some idea as to methodology. Do you want this to be an .exe on windows? Then java and access should work well enough. Browser based? Then javascript/php/mysql will do the trick.

I guess my suggestion would be to look at how esp reports are always laid out. Check this as plain text, not as the in game message shows (copy/paste to word). You need to then look for the bits of data you want to store: date/time, planet location, ships, guns, res ... It would be best if each has its own column (data group) in your database. You may want to derive other bits of data (a boolean for no fleets/defense if no ships/guns are listed). Be sure to note that not every esp report contains all the information.

Once you're able to pull out the data, inserting it into a database is easy (usually one line of code). The hard part is retrieval. I'd recommend a GUI that uses drop down lists of possible values (metal, <, 24, false, etc.). When search is indicated (button press usually), your code converts that into a query readable by the database. The database can then find matching rows (single esp reports) that match the criteria. Since all the data for an esp report is in the database, it can reconstruct it from the info it has stored. If one is found, it should display the entire esp report. If multiple, I list the planet location, with a date/time, sorted by either location or date/time. Link the reports to the location (another query to the database on a specific row, don't store all reports on the same page).

Once you create the database, you shouldn't need to alter it ('hand create the variables'). Your variable list isn't that hard to figure out (ignore activity as it isn't relevant past date/time):
planet location (name too?)
date/time
player name (for linking to other tables/databases later)
metal amount
crystal amount
deut amount
energy amount (does this matter?)
fleet (true/false/unknown)(is a solar sat part of a fleet?)
# of battleships
# of cruisers
# of [ship type]
defense (true/false/unknown)
# of rocket launchers
# of light lasers
# of [defense type]
lvl of metal mine
lvl of crystal mine
lvl of [building]
lvl of esp tech
lvl of comp tech
lvl of [research type]
percent chance of counter esp (does this matter?)

You'll probably want a unique index number for each esp report as well (makes the multi-results part easier).

I guess it comes down to what technologies you want to use, how you want to interface with it, who/what can interface with it, and time you want to dedicate to it. The database setup is probably the easiest part of the whole thing.

Questions/comments?

Tacticus has hit a lot of areas, so as not to be redundant, I will just ask what tools you have available? e.g. Do you own MS Office/Access? MS Access is one of the simplier DB programs to learn if you have no DB experience. I haven't used it for quite awhile, but there is probably a tutorial built in...if not, I'm sure there are plenty available on the web.

The other thing to look for is if someone may have already done something simliar to what you want to do. Clout posted a link to a program that reformats Combat reports. Whoever wrote that may have done what you want or could probably whip it out in a hurry. I haven't gone to that link, but you might go there and see if you can track down the developer.

Well, as of now the only program I have available (well.. programs I guess) is a .txt editor (duh) and OpenOffice.Org Database manager thingy. Been Playing around with it a bit.

I've not used the open office one. My guess is it's based on some version of sql. Are you running linux? Getting sql on there is easy.

nope, Win7 x64

and... making the tables is the easy part, I agree.. but how do I get the information ONTO the tables.. thats the part I dont get >_<

Might come down to data entry, but looks to see if the db has an import function...usually the data can be seperated by commas or tabs...then getting the data file in a comma deliminated format is either done through manual editing or writing a program...if you haven't done any programming it may be manual editing. I would still pursue the guy that wrote the combat report formatter though.

Thats what I was thinking... cus I need something that can read the esp report (the pasted version or later via using a Firefox Addon) interpret the data, insert the data into the appropriate fields of the tables, then with the menu selection Tacticus mentioned recall whatever relevant info I want...

Idealy I want this usable so that you can do 100s of eps reports at once, then just copy the entire page of all the reports in one sitting , paste all the info into the parser, and then be able to narrow down what your looking for. Like fleets but no defense, or resources with no defenses, or what have you.

If I can get even the simplest form of this to work, I'll expand on it and make it usable for combat reports, esp reports, galaxy list info, stats, etc.... jus for shits and giggles

The hump to get over is finding a program that can read and write to an OpenOffice.org database <_<

I too have OO on Win 7 x64. I have never used base before so did some investigating. It uses a modified version of SQL called HSQL/HSQLDB. You should be able to create a program (java) that takes plain text esp reports and spits out HSQL statements. You can have these run automatically, or you can copy/paste the lot directly into base (tools->sql). Once there, I'd not bother with another program for displaying purposes. Base is a lot like access, so it has built in forms/queries.

I created a database and put in some fake values. I was then able to get all the info I needed based on search parameters. Once you get everything set up, you can save the query, and then produce a form based on that query. So when you update the table, you only have to open the form, which will run the query, and display it as you want. You can have many forms and queries saved, making it quick and dirty to get exactly what you want.

I think all you really need to do is write a converter program. The easiest thing would be to copy/paste the form into the program. It then reads the report and converts it into an HSQL statement. It should add all new reports to the ones done (in this instance), so that when you have all of them converted, you just copy/paste all the HSQL statements to base in one motion. You are then off and running.

If you ask really nice and I have the time, I could write a draft program for you (probably in php) so you can see what needs to be done, syntax, and so on.

yes, that is exactly what I am looking for thanks for looking into the database for OOo

How hard would it be for me to make such said program from scratch? If you know of any websites that could help me out, please show me the links.

It wouldn't be that difficult if you use a language with native string processing. Java and php are good examples. Looking at some esp reports, what I would do (without giving it much thought) is to read the entire report into a variable. Then split/break that on all newline characters ('\n'). This will give an array of paragraphs. You can ignore any that start with 'activity', 'your', '[number]', 'from', 'to', 'subject', or 'date' (you can pull the date from another one).

That should get rid of all the unimportant info. All remaining ones are useful. You then split those on spaces and tabs. You'll note that resources, research, buildings, defenses, and fleets all come in name/value pairs. Thus you should run the first word through a switch statement, to determine which column the value will go into. You then pair the second word, the value, to that column.

The harder to get info is planet name, location, date, time, player name, and percent counter esp. The hard part is that player names and planet names can be multiple words. So you need to look for constructs. For instance, the planet name starts after the word 'at' and ends before the word starting with '[' (this will be the planet location). The location is immediately after the name. The player name starts 2 words after the location and ends with the word ending in '')', that's single quote followed by close parentheses. The next word is ignored. The one after is the date. The one after is the time. This is the case for all esp reports. Counter esp is done in a similar manner (second to last word in the second to last paragraph).

To set boolean flags for defenses and fleets, check to see if there is a paragraph with one word 'fleets' or 'defense'. If present, the esp report got that info. Next, check to see if the following paragraph has the first word being a ship or gun. If so, you have a fleet and/or defense. Otherwise not. If the paragraph with the one word 'fleets' or 'defense' was not present, then there is no info and you should leave the boolean flag unset.

And that's really all there is to it. I'd start with an esp report that has all the info available and use it as your test case. Remember to make no field/value required. Cause once you get the full one working, try various partial reports and make sure it works.

I can't really provide links since you still haven't mentioned what language you plan on using, or other programs you will have available to use. Php requires a web server (or apache) to run, so that might not be the easiest thing. Javascript could probably do it, but be a bit messier. Java can do it, Eclipse would be of value.

As I said earlier, I dont have any software.... so I need help finding what would work best. I don't want to have to run a web-server... so the php won't work out very well.

I've never worked on Java or Javascript before. I've done php but was very limited to just some basic alterations of a pre-constructed hosting website. I'm great with understanding code and finding what parts are for which function... I'm not so good at starting from scratch as I don't know what to use to get the results I need.

Thus, any tutorials that you KNOW are good and include ALL of the command structures and explain in detail what each command does... I'd appreciate it. I'm looking around the web for something, but its not easy to look for what I don't know yet

what is Eclipse and is it free?

Eclipse is probably the best known program for creating programs, namely java programs. It has support, via plugins, for other languages. It is a free program available at eclipse.org. Today they release indigo, their newest version. It is free.

It is quite useful and user friendly. It offers command completion, debugging, and many many other features. I use it all the time.

Tutorials....I can't say I've ever used a java tutorial or book. I just start typing and if I can't figure something out, I google it. One of the more helpful sites is the java documentation site: http://download.oracle.com/javase/6/docs/api/overview-summary.html

PS: I learned java after c++. We were given c++ programs and told to convert them to java. Nothing else, no help, no book, nothing. So we learned java or failed the course. Eclipse has lots of built in help. For instance, if you hover the mouse over a function, it will pop up a box telling you parameters, usage, returns, and links to more info.

If we can find a time where we are both on skype, I can walk you through creating a simple java program via ecplise. I can send you code on various projects so you can see how to read/write files, guis, user input, command line arguments, and so on. Your best friend will probably be google though. I still refer to it constantly.

cool, that sounds like a PLAN. hopefully I can find time to get started on learning this Eclipse by fiddling with it tonight... well.. no not tonight cus I work till midnight then work at noon tomorrow so need sleep. Tomorrow night! Huzzah!

I found a good deal of video tutorials on Eclipse and how to do various things with it. Currently on the Standford U. Videos on programming.. VERY helpful.

We'll see if I can get started on it this month, or maybe more study will be better lol