Anyone know of any good ones?
Anyone know of any good ones?
lucidbot for IRC
Disclaimer
Take everything I post with a grain a salt. It tastes better that way.
Yeah, try to get a programmer who knows a language that has some server-side support (PHP, Java, Perl, C#, Python beings the one that readily come to mind) with a modicum amount of free time on board.Originally Posted by Palem
From there, convince him to code a basic intel repository (at first, should simply gather info from CBs, SOMs, SOSs, Surveys, Infiltrates and KD pages, store them in the DB and display it in a KD overview).
After that, he can add on the basic features using an incremental design.
If you start small and then slowly add to it, the task of coding your own Pimp won't be as garantuan-looking as it would otherwise appear.
If it helps, here is the path I took:
1) Support raw enemy intel and display in a KD overview (don't bother with editable columns at this point, just show the more vital info... you can add more columns later on if some info your KD judges is important is not displayed)
2) Add support for raw self-intel.
3) Add support for Papers and the ability to generate activity reports on attacks.
4) Add the ability to generate dragon slaying reports.
5) Add detailed province views (mirrors the province view-ingame, but shows a bunch of additional stats that are not shown in-game)
6) Support duration ops.
7) Support all ops.
oc, there were a bunch of intermediate steps (and details) I did not list here, but this is the big picture.
Ah, make sure you encapsulate all game mechanics and the parsing logic in a separate place in the code. This will make implementing age changes A LOT easier.
Make sure you use regexes for parsing and don't be afraid to "simply" the intel string by removing some stuff (like linefeeds) before you stick it in the regex (often will make the regex a lot simpler). Using anything less than regexes is pure madness.
Also, in retrospect, you probably want to implement 5 before 3-4.
btw, go easy on SpoiledTechie.
Honestly, I don't think he's as skilled a programmer as Molesquirrel, but his heart is in the right place and he tries very hard even if he often falls short.
I guess a flawed intel repository is better than no repository.
Last edited by Magn; 15-11-2010 at 18:26.
Writing a new pimp is a pain. As in the last war pimp totally broke I decided to try and write a little something like pimp. I would have taken SpoiledTechie his offer to look at his code but my knowledge of .net is not exactly great.
My "pimp" is able to understand kd pages, sots and soms, thus is nowhere near complete. It does know to take info from the most recent source where it's available (nw from kd page, acres from sot/kd page), understands that gnomes supply perfect intel (you assign the members of your group a level of trust. max level = gnome). I'll make the utopia page parsing and the intel sorting/encoding open source so that people can contribute http://bitbucket.org/JRD/upoopu-common (all but the webapp will be open source, but the webapp is boring, it only sends stored data to the browser, actually sorting out the mess of kd pages, sots and soms is done by the browser in javascript)
What I definitely need to fix before making it public is: basic op tracking (first only ms, more or less to proof it works), and workable account creation/management. I also need to make the webapp look a bit more pretty.
The utopia page parsers are written in python, and I truly think they're a thing of beauty (anyone cares to write one for news, buildings, science, and spells? ;))
PHP Code:class SoMParser(DataParser):
__responds_to__ = ("som",)
def __init__(self, data):
super(SoMParser, self).__init__(data)
self.pb = pb.IntelSoM
self.header = "SoM on %(province)s (%(kd)s:%(isl)s)"
@memoized
def parse(self):
province, kd, isl = self.get("([0-9] |^)[province] of [kingdom] [location]", 1, -2, -1)
if not kd or not isl:
# a real SoT, not self intel
province, kd, isl = self.get("military elders of [province] [location]", 0, -2, -1)
military_pct = self.get("approximately [pct] of our maximum population")
wages = self.get("wage rate is [pct]")
military_efficiency = self.get("military is functioning at [pct] efficiency")
off_me = self.get("offensive military effectiveness [pct]")
def_me = self.get("defensive military effectiveness [pct]")
offense = self.get("net offensive points at home [number]")
defense = self.get("net defensive points at home [number]")
_out = 0
army_home = {}; army_out_1 = {}; army_out_2 = {}; army_out_3 = {}; army_out_4 = {}
_temp = self.get("Army #1 \([number] days left")
if _temp:
_out = 1
army_out_1["away"] = _temp
_temp = self.get("Army #2 \([number] days left")
if _temp:
_out = 2
army_out_2["away"] = _temp
_temp = self.get("Army #3 \([number] days left")
if _temp:
_out = 3
army_out_3["away"] = _temp
_temp = self.get("Army #4 \([number] days left")
if _temp:
_out = 4
army_out_4["away"] = _temp
self.catcher("generals", _out + 1, "generals %s" % " ".join(["[number]"] * (_out + 1)), army_home, army_out_1, army_out_2, army_out_3, army_out_4)
self.catcher("soldiers", _out + 1, "soldiers %s" % " ".join(["[number]"] * (_out + 1)), army_home, army_out_1, army_out_2, army_out_3, army_out_4)
self.catcher("ospecs", _out + 1, "[ospec] %s" % " ".join(["[number]"] * (_out + 1)), army_home, army_out_1, army_out_2, army_out_3, army_out_4)
self.catcher("dspecs", _out + 1, "[dspec] %s" % " ".join(["[number]"] * (_out + 1)), army_home, army_out_1, army_out_2, army_out_3, army_out_4)
self.catcher("elites", _out + 1, "[elite] %s" % " ".join(["[number]"] * (_out + 1)), army_home, army_out_1, army_out_2, army_out_3, army_out_4)
self.catcher("horses", _out + 1, "war horses %s" % " ".join(["[number]"] * (_out + 1)), army_home, army_out_1, army_out_2, army_out_3, army_out_4)
self.catcher("land", _out + 1, "captured land %s" % " ".join(["[number]"] * (_out + 1)), army_home, army_out_1, army_out_2, army_out_3, army_out_4)
training_ospecs = self.get("[ospec](?P<training>( [number]){0,24})", "training", split=" ")
training_dspecs = self.get("[dspec](?P<training>( [number]){0,24})", "training", split=" ")
_elite_name = self.get("[elite]")
race = ""
for e, r in [(r[5], r[0]) for r in c.RACES]:
if e.lower() == _elite_name.lower():
race = r
break
training_elites = self.get("(?P<training>( [number]){0,24})", "training", split=" ")
training_thieves = self.get("thieves(?P<training>( [number]){0,24})", "training", split=" ")
data = {}
for k, v in locals().items():
if k != "self" and k[0] != "_":
# print "%s = %s (%s)" % (k, v, type(v))
data[k] = v
return data
Last edited by Yadda9To5; 16-11-2010 at 00:01. Reason: Showing off my parser
Yeah, I was in a similar situation.
After encountering 2-3 situations where Pimp went down during hostilities and I had to generate an overview by hand in Excel using raw intel, either I coded my own Pimp or stepped down from any position of leadership permanently.
Well, its pretty good.
The SOM parser was probably the worse so you got that over with.
For basic intel, you only have survey, SoS and Infiltrate left (and frankly, the last 2 are a joke to code).
Would that mean that you store the info related to each piece of intel in the DB?
Wouldn't it have been simpler to store province rows in the DB with columns for all the info you could possibly want on a province and update the relevant columns on a province row shortly after parsing a piece of intel?
Anyways, that was my approach to it.
I think this is something the original Pimp did very well and when it ain't broken, you probably don't want to fix it.
Basically, associate a poster to the most recent intel of each type gathered on a province (along with a datetime for when the intel was gathered) and let the viewer make up his own mind about the reliability of the source.
Not sure how efficient that is for the bandwidth.
You may want to limit what is sent to the relevant rows (for example, sending only the info about the KD the player is currently viewing).
That being said, doing the number crunching on the client-side seems like a good idea (well, for non-security stuff anyways) for efficiency.
If it helps, here's how I did it (the duration ops tracking):
Get the time when the player submit the op, increase the time by the number of ticks the op will last, truncate the minutes (you want to op to expire at the beginning of a tick) and store it in the DB. This is the op's expiration date.
At display time, calculate the difference between the current time and the expiration date to get how long the op has left and display it to the user in a neatly formatted string.
Also, each time a user view the ops, make sure you delete expired ops (if you are worried about efficiency, you can have a deamon delete expired ops at the beginning of each tick).
Many aspects of your app are already better-looking than mine.
I think that clarity and user-friendliness are the main consideration here. After all, it isn't a game, its an intel repository :P.
Last edited by Magn; 16-11-2010 at 07:09.
that's truly a neat parser allrightWriting a new pimp is a pain. As in the last war pimp totally broke I decided to try and write a little something like pimp. I would have taken SpoiledTechie his offer to look at his code but my knowledge of .net is not exactly great.
My "pimp" is able to understand kd pages, sots and soms, thus is nowhere near complete. It does know to take info from the most recent source where it's available (nw from kd page, acres from sot/kd page), understands that gnomes supply perfect intel (you assign the members of your group a level of trust. max level = gnome). I'll make the utopia page parsing and the intel sorting/encoding open source so that people can contribute http://bitbucket.org/JRD/upoopu-common (all but the webapp will be open source, but the webapp is boring, it only sends stored data to the browser, actually sorting out the mess of kd pages, sots and soms is done by the browser in javascript)
What I definitely need to fix before making it public is: basic op tracking (first only ms, more or less to proof it works), and workable account creation/management. I also need to make the webapp look a bit more pretty.
The utopia page parsers are written in python, and I truly think they're a thing of beauty (anyone cares to write one for news, buildings, science, and spells? ;))
PHP Code:class SoMParser(DataParser):
__responds_to__ = ("som",)
def __init__(self, data):
super(SoMParser, self).__init__(data)
self.pb = pb.IntelSoM
self.header = "SoM on %(province)s (%(kd)s:%(isl)s)"
@memoized
def parse(self):
province, kd, isl = self.get("([0-9] |^)[province] of [kingdom] [location]", 1, -2, -1)
if not kd or not isl:
# a real SoT, not self intel
province, kd, isl = self.get("military elders of [province] [location]", 0, -2, -1)
military_pct = self.get("approximately [pct] of our maximum population")
wages = self.get("wage rate is [pct]")
military_efficiency = self.get("military is functioning at [pct] efficiency")
off_me = self.get("offensive military effectiveness [pct]")
def_me = self.get("defensive military effectiveness [pct]")
offense = self.get("net offensive points at home [number]")
defense = self.get("net defensive points at home [number]")
_out = 0
army_home = {}; army_out_1 = {}; army_out_2 = {}; army_out_3 = {}; army_out_4 = {}
_temp = self.get("Army #1 \([number] days left")
if _temp:
_out = 1
army_out_1["away"] = _temp
_temp = self.get("Army #2 \([number] days left")
if _temp:
_out = 2
army_out_2["away"] = _temp
_temp = self.get("Army #3 \([number] days left")
if _temp:
_out = 3
army_out_3["away"] = _temp
_temp = self.get("Army #4 \([number] days left")
if _temp:
_out = 4
army_out_4["away"] = _temp
self.catcher("generals", _out + 1, "generals %s" % " ".join(["[number]"] * (_out + 1)), army_home, army_out_1, army_out_2, army_out_3, army_out_4)
self.catcher("soldiers", _out + 1, "soldiers %s" % " ".join(["[number]"] * (_out + 1)), army_home, army_out_1, army_out_2, army_out_3, army_out_4)
self.catcher("ospecs", _out + 1, "[ospec] %s" % " ".join(["[number]"] * (_out + 1)), army_home, army_out_1, army_out_2, army_out_3, army_out_4)
self.catcher("dspecs", _out + 1, "[dspec] %s" % " ".join(["[number]"] * (_out + 1)), army_home, army_out_1, army_out_2, army_out_3, army_out_4)
self.catcher("elites", _out + 1, "[elite] %s" % " ".join(["[number]"] * (_out + 1)), army_home, army_out_1, army_out_2, army_out_3, army_out_4)
self.catcher("horses", _out + 1, "war horses %s" % " ".join(["[number]"] * (_out + 1)), army_home, army_out_1, army_out_2, army_out_3, army_out_4)
self.catcher("land", _out + 1, "captured land %s" % " ".join(["[number]"] * (_out + 1)), army_home, army_out_1, army_out_2, army_out_3, army_out_4)
training_ospecs = self.get("[ospec](?P<training>( [number]){0,24})", "training", split=" ")
training_dspecs = self.get("[dspec](?P<training>( [number]){0,24})", "training", split=" ")
_elite_name = self.get("[elite]")
race = ""
for e, r in [(r[5], r[0]) for r in c.RACES]:
if e.lower() == _elite_name.lower():
race = r
break
training_elites = self.get("(?P<training>( [number]){0,24})", "training", split=" ")
training_thieves = self.get("thieves(?P<training>( [number]){0,24})", "training", split=" ")
data = {}
for k, v in locals().items():
if k != "self" and k[0] != "_":
# print "%s = %s (%s)" % (k, v, type(v))
data[k] = v
return data
Corruption is a serious impediment to civil liberties.
I dunno. On one side, things are all neatly structured into an OO design with the work split between various member functions and I like that.
On the other, I have a hard time understanding what is being done by casually reading your code and I don't like that (some of the variable names could definitely be clearer).
Overall, it seems like you split your SOM into several smaller regexes.
I'm not enough of a regex connaisseur to tell you whether its definitely good or bad, but I'd be enclined to think that you should make it one large regex (nothing prevents you from spliting it into smaller strings for clarity before joining it into one large string and compiling it), because that way, all the info is fetched in its proper context within the intel string.
Afterwards, the above allows you to do thing like fetching a CB+SOM (formatted back-to-back on the same string) without having to worry whether the info tagged as being from the SOM was really taken from the CB and vice-versa and generally prevents players from doing things like posting half a SOM (which is probably an error from the user that you want to catch).
Efficiency-wise, its hard to put a nail on it, but I'd be enclined to say that parsing the entire thing once with 1 regex is more efficient than making several passes at it with smaller regexes.
Last edited by Magn; 16-11-2010 at 20:31.
Yes, each type of intel is stored in the database separately. (Actually binary, encoded as protocolbuffer, so it isn't even possible to filter it in the database, of course some meta data, like timestamp, kd, isl, prov is stored in separate fields)
Updating fields in the database as you go might have been easier, but has two problems. Firstly not all users have access to the same data (user a might be part of group 1 and group 2, user b of group 2 and 3), so i would have needed top store intel per user per group. Secondly I want people to be able to access all old, original, data that was added to the site, so they can decide whether the derived numbers make sense. (Also how could i possibly invalidate intel? If an admin wants to remove a fake SoM, then how to know which values in the database are correct, and which ones aren't as they are derived from the SoM. Plus i'd have needed an insane number of fields (think buildings coming in, troops coming in))
As all sorting is done client side I saw no benefits in storing separate data.
Of course, only limited sets (data associate with a single kingdom, or only the most recent data) are send to the browser. Parsing, however, is incredibly smooth. As quite a few browsers now support data storage it is, in theory for now, even possible to store all data client side and only incrementally update it with new data.
As i keep all data expiration is simple. I just need to check if the current time (which is synced to server time) is smaller than data + expiration time.
Indeed, user-friendliness is the most important aspect. My app will sadly fail in one aspect, no support for angel. Though i think the utools forum-agent sends raw data, so that one should work.
The variable names are the names of the data fragments/fields, and encoded as such in the protocolbuffers used to encode the data. (less space efficient than angels custom encoding, but much more understandable, plus making the protocolbuffer formats available should allow everyone to use the parsed data in a bot or something)
Neither am i much of a regex guru, but yes, i use multiple regexes. This way its (at least to me) understandable what data comes from where. One big regex would give me headaches ;). The get method "eats" the data, all data that is matched is removed. That way i can always match from the start of a string. And like you said, data is normalized first. All white space sequences are converted to a single space.
You could use Angels forum agent to log data.
Support email: utopiasupport@utopia-game.com <- please use this and don't just PM me| Account Deleted/Inactive | Utopia Facebook Page | #tactics <-- click to join IRC|PM DavidC for test server access
K, so you're using GAE? In that case, I'll trust in your expertize as I have no experience with it.
I'm aware that they store server-side info differently and that may have some bearing on the design.
Well, if you store all the lowest-level info that the intel gives you, the number of fields is large, but its also fixed (unless the developpers suddenly decide to add new information to intel which seldom happens).
In tradditional Pimp design, the info posted by a player belongs to 1 group (his KD) and the info posted by all players is assumed to be trusted.
The main source of mistrust comes from how old the intel his, whether the player is non-gnome and sent few thieves and whether the player didn't copy-paste the intel properly, but the player's intent is assumed to be genuine.
To reinforce this, you need to provide the monarch with a way to add or remove players from his Pimp KD (after that, its really up to the monarch).
However, what the monarch will not want is to have to sort through a ton of old dated intel (if he'd wanted to do that, he'd do intel sharing in the forums).
If you store each piece of intel separately, you'll either have to force the monarch to manage it by manually deleting old intel (very bad, don't do it) or you'll have to provide an automated mechanism to flush old intel at some point. Either way, you should use the latest intel provided by default (kinda the expected behavior) to make it transparent for the monarchs who won't want to manage those details.
Thats easy. You don't store derived info in the DB. You store the basic info given to you by the intel.
You infer the derived info at display time.
That works too.
Well, I'm not very familiar with the parsing facilities of Javascript.
I am with those in Python.
Since the server will have to do some parsing in Python anyways, storing the relevant info in groups ensures that I'll have to do the parsing only once (call me lazy) and its not much of an overhead for the server (it will have to do the hard work of parsing the string either way).
Angel is overated. I'm pretty close to the point where all relevant Angel facilities will be supported directly in my Pimp.
Coding 1 big regex for each piece of intel is in many ways similar to coding one big program.
You find patterns in the regexes that repeat themselves and you store them in string variables.
From there, you create your regex strings dynamically (when the parsing module first loads), making use of those string variables and you compile the resulting strings to get your regexes (1 regex for KD pages, 1 regex for SoTs, 1 regex for Papers, etc).
Last edited by Magn; 17-11-2010 at 17:38.
Saving all info in a single table of the database seems like a really bad idea, if that's what you're suggesting. The big issue with the current Pimp is performance is it not? Having a single table is a perfect way to ensure low performance.
Also, as far as I'm aware, Pimp never used to save old intel. Add something new and it overwrote what was there before. Flushing out old intel (if you even need to do that) shouldn't be especially hard (unless you use the one table approach). All your intel should have timestamps, and so you can easily just delete everything older than x hours or days or whatever. Only thing my bot flushes is the armies out from soms. It flushes them before responding to a request to print a SoM, to be exact.
Also, as far as supporting angel goes. If you've got everything else in place, it shouldn't take too much work to add support for angel unless you've coded everything else very badly. At least with angel you know that different browsers aren't gonna cause you headaches by formatting stuff differently. Also, having people be able to auto send intel via forum agent (or uTools) is very convenient. I don't see how it's user friendly to force people to copy intel (with angel turned off, if they have it), go to the webpage and paste it in, when they could instead just copy it and have angel take care of the rest. Pimp is for storage, so unless you're gonna build in more functionality in there, I don't see it as a place where people are gonna spend a lot of time.
Also, why are people encouraging others to code their own solutions? Why don't you just share your own instead. Not like this game couldn't use the help...
Last edited by Luc; 18-11-2010 at 10:33.
Nope, no appengine, though the web part is playframework+scala so it should work on appengine (i'd need to adjust my data model a bit). I just let the browser sort through the data instead of the server.
My my descrription was a bit vague. Below is how I store intel:
PHP Code:@Entity
class Intel(
@Index(name = "idx_name")
var name: String,
@Index(name = "idx_hash")
var hash: String,
@Index(name = "idx_major_location")
var major_location: Int,
@Index(name = "idx_minor_location")
var minor_location: Int,
@Index(name = "idx_intel_type")
var intel_type: String,
@Index(name = "idx_intel_sub_type")
var intel_sub_type: String,
@Lob
var data: Array[Byte]
) extends Model {
def unique_key = this.hash + ":" + this.intel_type + ":" + this.intel_sub_type
}
I so agree, i don't use it, utools does basic math just find and for finer details there is excel. But its ability to upload data is pretty convenient.
Sadly it appears utools also sends angel formatted data :/ Conceptually parsing angel data is simple. However, as you wrote a bot you probably know to program. Thus you also know that "shouldn't take too much work" doesn't exactly exist. Things always take longer than expected, especially simple things.
Good thing my code is opensource ;) http://www.upoopu.com/source/
http://lucidessence.com/lucidbot/
his bot is opensource too
Support email: utopiasupport@utopia-game.com <- please use this and don't just PM me| Account Deleted/Inactive | Utopia Facebook Page | #tactics <-- click to join IRC|PM DavidC for test server access
Yes, but we're talking user friendliness here. The average player doesn't use excel if he has access to angel, and that's for no other reason than user friendliness and ease of use.
It sends angel formatted data because forum agent has an established protocol for intel sending that everyone can use and angel has been the de facto tool for formatting for many years. What would be the point in trading that for something application specific? Having the formatted text in your database makes it really easy to display intel in a way that people know how to read (or just save the export line if you wanna use that API). For someone who knows PHP (and I don't), creating a site that receives intel from forum agent or utools takes almost no time at all (considering PHP examples are offered at utopiatemple). Even I managed that.Sadly it appears utools also sends angel formatted data :/ Conceptually parsing angel data is simple. However, as you wrote a bot you probably know to program. Thus you also know that "shouldn't take too much work" doesn't exactly exist. Things always take longer than expected, especially simple things.
Well, angel formatted intel includes the same info, so if all your storage functionality is smart enough, all you need is more regexes. Of course it takes some time, but it's rather easy work, especially since no browsers with unique formatting of copied text are involved. Angel formatting is really strange in some ways too though. For example, the province name is included in all self intel except surveys. WTH?! Makes about as much sense as not including prov name in spells/ops like greed ingame.
Yeah, that's good, kudos to you. My reaction was mainly aimed toward Magn, who basically just said "do it yourself", which is a rather bad attitude imo.Good thing my code is opensource ;) http://www.upoopu.com/source/
Last edited by Luc; 18-11-2010 at 18:33.
There are more than 1 table.
I have at least 6-10 tables (can't be bothered to look the exact number atm), as it wouldn't make sense to store some stuff in a fixed sized table.
However, most of the fixed size info about a province will always fetched together.
Put it into too many tables and you get stuck doing useless joints on info that really should be displayed together.
It is more powerful than what I do, but in a way that is useless for a tool that will always show only the freshest intel.
My way of storing it is simply more efficient, because I don't need to delete old intel. Old intel simply gets overwritten when new intel goes in.
I always hated Angel (piece of malware if you ask me) and how some KD mates put pressure on me to provide them with export lines when I post intel even though I don't use Angel.
Can't wait until my tool makes Angel 100% obsolete in my KD.
As you can see, there is a big conflict of interest with me supporting Angel with my tool.
No headache whatsoever. Cross-browser compatibility across the major browsers (IE, Firefox, Opera, Chrome, Safari) is something I've had no choice to take into account for my main project and it permeates everything else I code for the web.
Well, you got me here. We'll simply agree to disagree.
Call me a fool, but I believe that a big part of the beauty of web apps is that they don't force you to download any separate executable whatsoever to run.
Depends on the amount of functionality you provide. At this point, mine is more than just a storage place (it started as such, but now it does a fair amount of processing and analysis on the data).
I guess because I'm simply not as charitable with my time as Spoiled Techie which is why I think you guys should go easy on him (in my defense, I do tend to be more generous with my money, but I have little of it atm).
I coded the tool to be used for my KD and spreading it around would just be giving up a competitive edge ( in part because its more than just an intel repository, a chunk of my in-game analysis went in there).
Also, making the tool available to a large number of people involves some overheads and pressures that I have little incentive to undertake given my current situation (I don't exactly roll in the money right now, I'm working on something else that will actually generate income and my time is very precious). I'm a programmer foremost and an admin second.
Last edited by Magn; 18-11-2010 at 19:14.
There are currently 1 users browsing this thread. (0 members and 1 guests)
Posting Permissions
