```yaml
---
slug: transactions-part-1
published: 2006-04-13
description: Gaia's original "transaction manager" explained
excerpt: ~
changelog:
  - on: 2020-02-23
    note: This was rescued from the archives of the old [felocity.org](https://web.archive.org/web/20050507121849/http://www.felocity.org/journal). As a result, many links in this piece were left pointing to archive.org destinations.
---
```

Everything is on track for SXSWi this year, and excited isn't even a good word for how I feel right now. It's 4 am [Ed. Now it is noon the following day…] and I'm downright giddy. Last year I blew my first chance to go because I didn't move fast enough. This year, I jumped on travel planning much quicker and managed to grab a room at the Hilton Austin. I'm in on the 10th and out on the 14th, and if I didn't have some larger projects, I'd be staying much longer. To those going, I'll see you there (just look for the kid who looks like an excited lost puppy). However, if you notice the title, this isn't about SXSWi entirely, but instead is focused much more on some meaty PHP things: transactions. For the new bank code at Gaia, there needed to be a way to move items and gold about the site through a standardized (and [5 nines" reliability) system. Add to that full logging in the event of failure, and the ability to "transact" just about any two things on the site, and you get the foundation of the transaction system. This is part one of a multi-part article which will touch on the surface and theories used in the transactions framework. (Part 2 will focus more or architecture.)

* **Part 1: Theory**
* [Part 2, Transaction Framework](/thunked/transactions-part-2)

When the project first began, transactions were an odd concept. Not like database transactions, but a sequence of events that were atomically linked and reversible. Examples included purchasing from the Gaia Store (give the system gold, the system gives an item), making a trade (place an item into an "escrow" neutral zone), auctioning off an item (again, an escrow style neutral zone), purchasing non-pixels such as a Trading Pass or a Guild Purchase where the result is a database row but no items or gold, and many other things that went beyond the basic "remove gold, add item" mentality. This meant it was time for some research into complex data analysis models. Enter Martin Fowler, whose books I had read back in college, but like most college students made a point of forgetting. Thankfully, John had Analysis Patterns which was all we needed to get underway. I'm going to digress for a moment to say Fowler is a very intelligent man, but I remember why I couldn't remember what he wrote- Fowler has a hard time communicating why a pattern works the way it does in a way that connected with me. If you're someone who needs to know the "why", you might have to suck it up like I had to. After going cover to cover, it appeared there was a very basic system underneath even the most complex banking systems. To abridge the text, we started referring to it as the Conceptual Account Thingy (or CAT for short).

## Accounts, Entries, and All That Jazz
Keeping with our simplified version of things, an Account was described as nothing more than a container. This container could hold things, and those things belong to some entity. As an added bonus, [Gaia's "DAO"](http://72squared.com/index.php/module/Articles/action/Display/id/37) information was also stored with the Account in order to provide a means to get out and save data when needed. Accounts were then abstracted and an Account class was created for the "things" on Gaia that were owned by "someone". The initial list included:

* Gold (owned by user)
* Inventory (owned by user)
* Trades (owned by the trade row, with two users participating)

The third one is very interesting because a Trade became what was called a "Summary Account" in that it contained two Accounts inside of it, a left half and a right half. The left half was owned by the trade row and by the left user, while the right half was owned by the trade row and the right user. Because of this, the only entity that owns the entire trade is the trade row itself. When retrieving Accounts, they are always fetched by their owner which is often a primary key. This ensures speed of performance during the Account loading process. With a means to load and save the data, the next step is to manipulate it.

According to Fowler, the Entry class is nothing more than an entry that will be logged to an Account. Think of recording a check in the checkbook's ledger. This process is formally called Posting an entry, but for practical purposes, entries are applied to an Account, and then the changed Account is saved back to the database. To soften the load on PHP, the entries became lightweight containers of metadata which provided little more than a pretty interface for method calls. A simplified version of an Entry looked like: (note, "Container" is nothing more than an abstracted hash table in order to provide useful namespacing functionality. In this case, the `datatypecontainer` enforces typecasting on items going in and coming out of it.)

```php
<?PHP
/**
* Required Include Files
*/
require_once(DIR_CLASSES . 'container/datatypecontainer.php');

class TransactionEntry
{
    function TransactionEntry()
    {
        $this->Container = new DataTypeContainer();
    }
    function importData($data) {}

    function exportData($preserveId = true) {}

    function setEntryId($v) {}
    function getEntryId() {}

    function setTransactionId($v) {}
    function getTransactionId() {}

    function setEntityId($v) {}
    function getEntityId() {}

    function setAccountType($v) {}
    function getAccountType() {}

    function setObjectId($v) {}
    function getObjectId() {}

    function setAttribute($v) {}
    function getAttribute() {}

    function setAsAdd() {}
    function setAsSubtract() {}
    function setPolarity($v) {}
    function getPolarity() {}

    function setTimestamp($v) {}
    function getTimestamp() {}

    function setStatus($v) {}
    function getStatus() {}

    function cloneInverse()
    {
        $obj = new TransactionEntry();
        $data = $this-&gt;exportData(false);
        $data["polarity"] = $data["polarity"] * -1;
        $obj-&gt;importData($data);
        return $obj;
    }
}

?>
```

I left the `cloneInverse` statement because it was of particular note. From above, it was said that everything done to an Account needs to be able to be undone. The secret to this is being able to "reverse" all of the entries that have been applied to a transaction, and then reapply them. The net result of this will always be the Account's original state. In extensions of this Entry object, decoration aliases are used such as `setUserId()` in order to simplify the application programmers' work. With entries, and things to apply those entries to, the next step is to set up the rules.

## Posting Rules
When the transaction manager hands an entry to an Account, it is the Account's responsibility to take that entry and apply it. For different Accounts, applying an entry means different things. Inventory Accounts (owned by a user) are by far the hardest with their need to add/remove binary strings. An Account can always look at the class of an Entry object if that Account supports multiple types of entries. At present, since the inventories on Gaia (housing, main, game inventory, etc) are separated, there is no need to develop sophisticated entry handling. Fowler gets into extreme detail about Posting Rules and covers more than is needed by this system. However, the basics of how to apply a rule and what happens when applying an entry to an Account need to still be established. It was agreed upon that should a posting rule fail, it will fail all future objects in the posting rule change, and triggering a rollback of the transaction by taking the `cloneInverse()` of each Entry and reapplying them.

![image](https://user-images.githubusercontent.com/1795/146686016-cfbac26c-f5a8-4af8-99c9-f263450b608b.png)

Every `Account` then has a public `postEntry()` method, which takes an `Entry` object as an argument. Using the generic accessor functions from the base `TransactionEntry` object, the `Account` can apply the Object ID, Attribute, and Polarity to their entry. Each call to `postEntry()` returns `TRUE` on rule posting success. All that remains is to tackle the fact that the web (and the Accounts) are constantly in use. Collisions can occur when a second transaction manager is started while the first one is still in process. Since the first transaction hasn't saved, the second process has dirty stale old data.

Fowler presents two methods of avoiding collision, both which are worth touching on briefly. Mileage varies with either one.

## Posting Rules With "Catchup"
Of the two methods Fowler discussed, journaling is by far the most widely employed. To draw a real-life analogy, it functions much like a checkbook register. At any given point, a quick glance through the register can tell someone what the "balance" is, including all transactions which haven't cleared. Much like processing checks out of order, an Account can be measured by taking its present state, and then applying all deduction Entry objects which have not cleared yet. This gives the transaction manager the net low for the account, making it possible for the system to check for insufficient funds, missing items, and incomplete database rows. In a set of linear steps in PHP/MySQL, you might get something like the following:

1. Process 1 writes what it wants to do (status uncleared) to the DB
2. Process 1 gets an insert ID as a result of this
3. Process 1 loads the Account
4. Process 1 loads all deducting transactions up to its insert ID on top of the Account
5. Process 2 can start, and at any point after #<span/>1, its Net Account will include all of Process 1's unresolved deductions.

While this is a very reliable system, the need to constantly bring an Account to current isn't a very cheap process query wise. Also, every system needs to implement this transaction system so that a blind write to someone's inventory isn't destroyed in the process. Since the Vending system, Shopping system, Item Granting System (used in games) all would not start equipped with this transaction manager, it was not meant to be. If you're starting from scratch, this is certainly the way to go. If not, well, you can always improvise.

## Improvising with innoDB Locking
The problem was, as stated above, the asynchronous nature of Account objects. Lacking the ability to journal the transactions, the other solution to an Asynchronous problem is to break it. By redefining the Accounts to be as small a data set as possible and introducing a simple retry. The most likely collision on the site is the Gold Account, which would require a user to earn gold on the site within the same timeframe as say, completing a Marketplace Transaction. Marketplace can see the lock, wait a split second, and try again. The resulting failing collision rate is so low that it's considered acceptable to have a user retry in the event of a complete lockout.

I've made mention a few times to locking, and this is done with two tools: innoDB and the MySQL `SELECT...FOR UPDATE` syntax. We've made liberal use of the innoDB storage engine for it's transaction support, and according to MySQL, using the `SELECT...FOR UPDATE` syntax places a good lock on the innoDB row, letting the process finish what it was doing. If during the select phase some database row it needs is locked, it reseeds at a small random interval and performs its work on a retry. When complete, the transaction is committed and the changes recorded.

For spanning multiple databases with a single transaction, John developed a DB spanning object for the DAO he developed. It chains commits to fire off in sequence, effectively minimizing the failing window gap. Half commits are still a problem should for example, all the servers magically lose power and they are mid-commit. Unfortunately, this is a failing of the improvisation that can only be solved with a true multi-phase commit. In production testing, it's failure rate is wholly dependent on the hardware's reliability. While this has a low failing rate, it is substantial enough to warrant exploring extensions to the DAO's DB-spanning system.

## Conclusions

The system is currently in testing with a very good success rate. There are a few tweaks needed to optimize the system of Account and Entry objects in order to reduce weight when including them for processing. I'd also like to better explore the DAO's integration into the Account object- the current method alters the conceptual DAO behavior in order to solve a problem where more than one Account object may be tied to the same database row. More on that next time though.

This was rescued from the archives of the old [felocity.org](https://web.archive.org/web/20050507121849/http://www.felocity.org/journal). As a result, many links in this piece were left pointing to archive.org destinations.

Document

Gaia's original "transaction manager" explained

Link

code

<?PHP
/**
* Required Include Files
*/
require_once(DIR_CLASSES . 'container/datatypecontainer.php');

class TransactionEntry
{
    function TransactionEntry()
    {
        $this->Container = new DataTypeContainer();
    }
    function importData($data) {}

    function exportData($preserveId = true) {}

    function setEntryId($v) {}
    function getEntryId() {}

    function setTransactionId($v) {}
    function getTransactionId() {}

    function setEntityId($v) {}
    function getEntityId() {}

    function setAccountType($v) {}
    function getAccountType() {}

    function setObjectId($v) {}
    function getObjectId() {}

    function setAttribute($v) {}
    function getAttribute() {}

    function setAsAdd() {}
    function setAsSubtract() {}
    function setPolarity($v) {}
    function getPolarity() {}

    function setTimestamp($v) {}
    function getTimestamp() {}

    function setStatus($v) {}
    function getStatus() {}

    function cloneInverse()
    {
        $obj = new TransactionEntry();
        $data = $this-&gt;exportData(false);
        $data["polarity"] = $data["polarity"] * -1;
        $obj-&gt;importData($data);
        return $obj;
    }
}

?>


Fence

Transacting Items on Gaia Online

```yaml
---
slug: transactions-part-2
published: 2006-10-02
description: More on Gaia Online's "Bank Transaction Manager"
changelog:
  - on: 2020-02-23
    note: This was rescued from the archives of the old [felocity.org](https://web.archive.org/web/20050507121849/http://www.felocity.org/journal). As a result, many links in this piece were left pointing to archive.org destinations.
---
```

It's been a while since I've touched on the topic, mostly because I've been busy with work for Gaia. However, this incubation period has let the transaction system mature, and it continues to hold out well with over 29 million completed transactions in the last 7 months. Since its inception, it's been adjusted to support item granting, and is currently being extended to support transactions in the Gaia Marketplace / Auction House. In the first article, I covered the surface of the transaction manager, and some of Fowler's theories which were behind it. In order to avoid recoding large parts of the site, a lock-first mentality was chosen for how to handle keeping an account up to date. This time, we're going to focus on all the players / classes of the transaction manager and how they work together.

* [Part 1: Theory](/thunked/transactions-part-1)
* **Part 2, Transaction Framework**

As in most programming languages, PHP provides support for both public and private methods within a class. In development, one of the primary objectives was to minimize the external facing interface for the end-developer, while still keeping the backend components easy for a developer to maintain. Each component was given specific tasks, responsibilities, and a list of classes that it would couple with in order to provide functionality.

## Bank Transaction Manager
The first component is the first of three end-developer facing classes. The Bank Transaction Manager is responsible for creating Transaction Entries, adding Transaction Entries to its internal stack, and then processing the entry stack while reporting on the success or failure of the process. When a Bank Transaction Manager is initialized, it creates an Account Manager object for tracking the accounts which it needs to load. As entries are created with the `createEntry()` method, they are fed to the end developer and then added back in to the system via `addEntry()`.

A Transaction Entry is nothing more than a shell object, providing a clean interface for setting an object's ID, it's attribute, quantity, and other associated properties. When `addentry()` is called, the Bank Transaction Manager uses these properties in order to decide what Account to load (via the Account Manager). This pairing is then set aside until execution.

During the execution phase, the Bank Transaction Manager tells the Account Manager to initialize all relevant accounts. It then loops through every entry/account pair and passes the entry into the account. In processing terms, these are called posting rules. Every posting rule has a boolean success, and the sum of all functions returning an ultimate success or failure for `execute()`.

If any errors are encountered, the Bank Transaction Error (a simple static class) is written to with information about the failure.

## Account Manager
In a horizontally scaled environment, a user's inventory or a row of trading information might reside on completely separate machines. To encapsulate the loading and saving of these various things is the reason for the Account Manager object. Requested accounts are queued up, and then loaded via a single call to `loadAccounts()`. Beyond simply loading accounts, the account manager should be able to

- handle information if the account has changed
- prevent data overlap by avoiding querying for duplicate data
- provide an interface for the instantiation of new Account objects

During the execution of the Bank Transaction Manager, the Account Manager is called (specifically when `BankTransactionManager::addEntry()` is used). Accounts are queued for loading as entries are added, and are then loaded at once. There are two ways to load the account data, one which uses blocking, and one which does not. Even at 4.5 million registered members, the blocking method has not been an issue, and so that is the method we'll be talking about. In an InnoDB database, a row can be locked during the select phase as part of a transaction by using the following syntax: `SELECT 'columns' FROM 'table' WHERE 'criteria' FOR UPDATE;`

The `FOR UPDATE` places a lock on the row, leaving your PHP script as the only script that has access until the lock expires. This feature is only available on InnoDB however, as the MyISAM engine supports neither row locking nor transactions.

When asked to save, the `AccountManager` calls the `isChanged()` method on every account in order to verify if it has been altered. Accounts which have been altered are then saved back to the database and the locks released.

## Transaction Entry
There's not much to say about the `TransactionEntry` object. As a shell, all it contains is a data payload. It's information consists of: "how many", "of what", "with what serial", and "adding or subtracting". To help simplify the development end, the Entry object is extended to support Game items, Regular Items, Gold, and Gift Credits. During execution, these small instruction sets are passed to the Account object.

## Account
Defined as the authoritative source regarding all data coming in and going out, these objects are modeled in spirit after the VO J2EE pattern. They exist to hold an account's data, and are decorated with methods to support the changing of that data. When the Account Manager calls `loadAccounts()`, every queued account object is asked for its database information and required fields. The sum of this data is aggregated, and passed into the `importData($fieldName, $data)` function. When the Bank Transaction Manager executes, it then takes every Transaction Entry object, and passes it directly into `postEntry($t)`. It is at this point that the heavy lifting is actually done.

Every account maintains a ruleset for how its data is to be altered, known as "posting rules". These rules are very strict, and breaking them should cause the entire transaction to fail for the user and return an error. The most common example of failure is if the posting rule contains an item type that the account does not support, such as posting gold information to an inventory account. With a single public interface, the logic of altering the account information is also internalized, protecting it from the outside world. Upon altering the account, `postEntry()` will return either `TRUE` or `FALSE`, which will be evaluated by the Bank Transaction Manager object.

Adding new accounts or extending existing accounts to add functionality is very easy. The methods exposed publicly are small, specifically `getDaoName()` and `getDaoFields()` for Gaia's database support, `importData()` for loading, `exportAccount()` for saving, `isChanged()` for the Account Manager's save required check, and `postEntry()` for altering the account's data.

## Code in Practice (End Developer Side)

```php
$btm = new BankTransactionManager($transaction);

// create entry (Account, Entry Type)
$e = $btm->createEntry("Gold.Gold");
$e->setUserId($user_id);
$e->addGold($gold_amount);
$btm->addEntry($e);

$e = $btm->createEntry("ServerGold.Gold");
$e->setUserId(ENTITY_SERVER);
$e->subtractGold($gold_amount);
$btm->addEntry($e);

try {
    $btm->execute();
}
catch (Exception $e) {
    throw $e;
}
```

More on Gaia Online's "Bank Transaction Manager"

$btm = new BankTransactionManager($transaction);

// create entry (Account, Entry Type)
$e = $btm->createEntry("Gold.Gold");
$e->setUserId($user_id);
$e->addGold($gold_amount);
$btm->addEntry($e);

$e = $btm->createEntry("ServerGold.Gold");
$e->setUserId(ENTITY_SERVER);
$e->subtractGold($gold_amount);
$btm->addEntry($e);

try {
    $btm->execute();
}
catch (Exception $e) {
    throw $e;
}


Transacting Items on Gaia Online, Part 2

```yaml
---
slug: phpbb-permissioning
published: 2004-12-02T00:00:00-08:00
description: How Gaia's volunteer moderator system outstripped phpbb's capabilities
changelog:
  - on: 2020-02-23
    note: This was rescued from the archives of the old [felocity.org](https://web.archive.org/web/20041207010434/http://felocity.org/) site. Writing on the internet was pretty different back then. As an aside, this was how I ended up working for one of the coolest social gaming companies at the time.
---
```

Currently, I've been doing a lot of work with the way phpbb handles its permissions. The method is a group based Moderator scheme, which then uses a different user attribute for most moderation functions. Many people know about Gaia's two moderator groups, and this is actually more in line with how phpbb intended permissions to be. Give a group power to modify forum X, give another group power to modify forum Y. The problem is, all moderation functions check to see if you are an admin. In phpbb's architecture, there is no admin "group". The only reasoning I could find for it was that it simply wasn't thought of. Many other large scale systems (Xoops comes to mind) use an administrative group just like any other group. Permissions are then applied on the group level.

So then, in centralizing the permissioning model, we see two primary approaches. The first of these is to centralize permissions into groups, and then add users to groups. However, with over 1.3 million users, the group table will grow very large, very quickly. However, you could change permissions for an entire group of people by just adjusting the group. The other way that came to mind was to centralize the permissions on the users themselves. This would allow for "groups of permissions" instead of "groups of users". From a management perspective, this actually isn't the better of the two solutions. However, assuming a static package of permissions relative to a user's power on the site, it makes sense to simply give a user a level / value / package. You could still potentially build a model for a finer granularity of control, but in the case of Gaia, a moderator / admin / etc has a predefined series of tools. The likelihood of adding someone who would mod area A but not area B is actually small (compared to the total user base) and so these individual cases are realistic enough to handle in a separate table.

The other piece of rationale to a user-centric permissioning system is its need to be both forward-compatible with future development, and yet backwards compatible with everything else that has been written. This is where the nightmare truly begins. Several pieces around the site need to be reengineered to the new permission model, including the way our forum tree is built, the piece that says if you can view a topic or not, forum viewing, jumpboxes, etc. However, there are prefabricated templates and expected results. So the new code has expected input, and output, but a completely different internal engine. Essentially then, we are emulating phpbb's authorization functions while everything under the hood changes. Thankfully, all future developments already include the user-centric model, so the biggest overhaul is in the forums.

I've got some 3 pages of notes about ideas of where to change code. It's time now to see if anything out of these notes will pan out.

This was rescued from the archives of the old [felocity.org](https://web.archive.org/web/20041207010434/http://felocity.org/) site. Writing on the internet was pretty different back then. As an aside, this was how I ended up working for one of the coolest social gaming companies at the time.

How Gaia's volunteer moderator system outstripped phpbb's capabilities

phpbb Permissioning

```yaml
---
slug: most-minteresting
published: 2005-10-30T00:00:00-08:00
description: Turning Mint's traffic data into a feed of posts
changelog:
  - on: 2020-02-23
    note: This was rescued from the archives of the old [felocity.org](https://web.archive.org/web/20050507121849/http://www.felocity.org/journal). As a result, many links in this piece were left pointing to archive.org destinations.
---
```

Let's get right to it- I've been blogging a lot about [Mint](http://www.haveamint.com/) lately, and there really is a good reason for it all. As a PHP/MySQL person, I love to tinker. You may have noticed a new thing where the portfolio used to be (if you're not the type for RSS). Thanks to Shaun's Mint and the good plugin folks of [Textpattern](http://www.textpattern.com/), I've written up a little something I like to refer to as the "Most Mintersting Articles". Puns and jokes aside, this is a pretty robust plugin for TXP, and includes its own caching module.

The need was a simple enough one, I wanted more exposure for the more interesting things on the site I've written. Some articles get a lot more reads than others, and some articles get a lot more comments. Since Mint is really good at collecting where on my site people went, it made sense to design a plugin to focus on that aspect. Additionally, Mint rolls its data and deletes old logs so the "freshest" pages are always going to update with time.

## Caching the Minty Joy
If I could walk out of here with a `LIKE %` in my SQL and run it every single time no matter what, I would have. However, that's a bad idea on a small machine with a big database table. With a little bit of caching, we can eliminate the server strain that would otherwise be caused by the plugin. (And if it's really extreme, removing the wild-cards from the resource variable can remove the problem altogether.) The caching concept is actually based on a very old method we used for Gaia Online, put together with a bit more modern piece from the new database code we rolled out. It is based on the idea that if a query should be cached, it's SQL should never change. This means we can base the filename for the cache off of the SQL query. Since we have a very small number (1) of queries in this case, a CRC32 will suffice.

## The Goods
The rest really just needs to be seen to be believed. Of course, you need Mint (link at the top there) to make use of this, but Shaun made some awesome software, so it's well worth the price. I'd like to think of this as another good reason to get fresh and minty!

Update: Version 0.2 now supports the Mint object if Mint is on a different database than TXP.

Download minteresting v 0.2 available as a [gist](https://gist.github.com/Jakobo/f460f8f08cf92379ca47fcea4b3220f0)

Turning Mint's traffic data into a feed of posts

Most Minteresting

A recursive acronym for "PHP: Hypertext Preprocessor"