Categories
Startup World Wide Web

Eukles: Marketing Automation done DIY

For the past couple of years I have been playing around with the idea of a hard divide between outgoing communication (transactional emails, push notifications, newsletter contact list synchronization, marketing automation, slack notifications, etc) and the business logic of my applications.

It never made sense to me that a backend system that manages ticket sales for an should would also implement a mailer queue; but also it doesn’t make sense that the copywriter who wants to change the content of an email would have to talk to a developer to do so.

In 2017 I started working on removing all logic for these kind of actions from my websites and replace them with a single event driven microservice that would handle anything related to outgoing communication for all projects I build.

My goals where to:

  • Be vendor independent: Services like Mailchimp, Moosend, Sendpulse etc have basically already built what I wanted to have, but given my DIY nature I wanted to stay independent from them.
  • Base all actions from a stream of events sent by the business logic.
  • Manage all GDPR related permissions and preferences from a central location.
  • Cut all links between the app domain and this microservice: no accessing data that has not been provided. Push only.
  • Be able to synchronize lists of users (= segments) according to relationships and synchronize these to newsletter services. (ie: automatically make an email list of all registered users who are attending a certain quiz).
  • Provide a management panel where non coders could change the content and the logic of the outgoing notifications.

That is what I started building. I even setup a website to ask feedback from smarter people to make sure I wasn’t reinventing a square wheel, but in the end I kept building nevertheless: Eukles was born.

The Slack notification

I started with our ticking website CatLab Events, and my first goal was pretty modest: send a slack notification to our channel every time someone bought a ticket for one of our quizzes. I setup a MySQL database (probably not the best fit, but as I wanted quick results I stayed with the tools I knew best) and a Laravel/Charon REST API and also created a separate API client package for PHP to easily implement in existing projects. A few days later I registered my first event:

/**
 * @param OrderConfirmed $e
 * @throws \GuzzleHttp\Exception\GuzzleException
 */
public function onOrderConfirmed(OrderConfirmed $e)
{
    $order = $e->order;

    $attributes = [
        'event' => $order->event,
        'order' => $order,
        'ticketCategory' => $order->ticketCategory
    ];

    if ($order->group) {
        $attributes['group'] = $order->group;
        $attributes['subject'] = $order->group;
    } else {
        $attributes['subject'] = $order->user;
    }

    // Track on ze eukles.
    $euklesEvent = \Eukles::createEvent(
        'event.order.confirmed',
        $attributes
    );

    $euklesEvent->unlink($order->user, 'registering', $order->event);

    if ($order->group) {
        $euklesEvent->link($order->group, 'attends', $order->event);
        foreach ($order->group->members as $member) {
            if ($member->user) {
                $euklesEvent->setObject('member', $member->user);
            }
        }
    }

    \Eukles::trackEvent($euklesEvent);
}

To break it down: what I wanted to communicate to Eukles was:

  • An event.order.confirmed event has happened
  • This event has some related properties (event, order, ticketCategory, group and subject (which can be user or group, depending on the event type).
  • Unlink the ‘registering’ relationship between the active user and the event (= the user is not registering anymore, as the registration is already done). The idea is that these relationship actions could be defined either in the business logic or triggers on Eukles itself.
  • Link the ‘attends’ relationships between the group and the event. This would allow us to create a segments of all groups that attend a specific event and synchronize that to Mailchimp later.
  • Add all members of the group as properties of the event (= when a group has 4 members, send an email to every single one of them)

(Note that all models used as properties would have to implement an ‘EuklesModel’ interface that included methods to serialize the content that is send to Eukles.)

The content of the event properties is stored in two places:

  • As part of the event, so that we always know the state of a model at the time of the event creation.
  • (If a unique identifier is provided) the model it is also synchronized with a centralized project-model database that contains the relationships between project-models and always contains the last known properties. Every time an event is sent, the content of these models get updated.

Now I just had to add a SlackService, store authentication details for these services in the database and setup a few Action and Trigger models that would listen for the event and cary out actions. And using the Jobs queue in Laravel everything was handled without impacting the user flow.

Internal use only (for now). Everything is manageable from a REST API with the goal to build an angular frontend in a later stage.
Humble beginnings.

To keep things simple I used Laravels Mustache syntax to parse the arguments that were used by the service clients.

GDPR consents

As Eukles would become responsible for all outgoing communication, I found it fitting to store communication preferences and consents on Eukles instead of in the business logic. Eukles doesn’t really have the concept of a ‘user’, as user project models are sent in events and follow the same logic as any other event property. So I created an API endpoint that would show me all consents in a given project by a given project model. By adding this call to my registration process I was able to ask all users that logged in (existing or new) to give the required and optional consents.

By adding these as filters to segments and actions, I was able to block communications to users who opted out of these messages.

Segments

The goal of Segments was to make dynamic or static lists of project models and synchronize those lists with newsletter services like Mailchimp. For example: by defining segment rules and setting a parent model in the segment schema, Eukles is able to create a segment for each ‘event’ projectmodel, containing all ‘user’ projectmodels that have the ‘attending’ relationship set to it AND have given consent to receive newsletters.

Once every hour these segments are then synchronized with Moosend or Mailchimp, and whoever is in charge of sending out the newsletters can rely on those lists being up to date at all times.

This synchronization also checks for users who are marked as ‘unsubscribed’ in Mailchimp, and the applicable consent is removed from those specific users, resulting in remove them from that segment.

Templates

As all emails in QuizWitz use the same basic template, I have added the ability to define mustache-like ‘templates’ for each project. Anywhere parameters are parsed, these templates can be used (but they only really make sense in html-like contexts)

The email that is sent whenever a player starts a QuizWitz PRO game uses the standard ’email’ template.

Wait 1 hour. if !subject.hasLicense(event.license): then: send email.

And that is pretty much where the project stayed for a few years. My simple event -> trigger -> action model worked fine for sending transactions emails, notify us about certain events and even sending tweets with giphies based on event attributes (implementing services is trivial since the base architecture for defining actions and storing credentials is already provided).

All was well in Eukles world… until I found myself with a bit of time on my hands and I decided to implement some dark design patterns to try to raise the conversion rates in QuizWitz: when someone initalizes a license purchase but in the end decides not to complete the purchase, send them an email asking for feedback.

A perfect use-case for Eukles, where it not that I had gone for the quick event -> trigger -> actions design instead of a fully fledged ‘workflow’ state machine. So that is what I started building.

Instead of triggering an action, all events now check for ‘workflow triggers’ that initializes states in the workflow machine. Depending on the trigger configuration, Eukles also checks for duplicates to make sure that recurring events don’t cause too much triggers. A state then follows the configurable steps in the workflow and triggers actions that are defined in steps. Each trigger also has the ability to set one of the event properties as ‘subject’ that can be used anywhere in the workflow, to reduce the dependency on the initial event.

A simple shunting yard algorithm offers me the ability to define conditions in string format and makes it possible to setup decision trees in the workflow, while a special ‘wait’ step adds a delay to the state machine processing. The whole thing is using Laravel Job queues, so it was all fairly trivial to setup.

The table based interface is obviously a horrible choice for setting up workflows like this, but as for the moment I am the only one using this, it will do. No need to pour more hours in things that only marginally improve my life ๐Ÿ˜‰

And that basically is the current state of the Eukles project. My goal is to clean it up and release it under a GPL license in the future, but in order to do that I first want to improve the admin panel interface and write down some documentation. If that ever happens, I’ll definitely post it here.

Categories
English Mijn Wereld World Wide Web

“Ye, people, hear my story”

An English post, for English people. My blogging frequency goes up another notch and I’m quicly approaching the 3 blogs a year border!

But enough about my blogging rituals. As some of you guys, my frequent visitors, might know, the National Novel Writing Month. The idea is fairly simple: write a 50000 words novel in less than one month. An easy challenge indeed; how hard can it be to write a fifty thousands word story?

Since last year, I only heard about it when it was half over, I decided to give it a try. Since I had completely forgotten about the weird contest, I heard the deadline started yesterday… yesterday. In other words, I’m starting on this in the purest form possible: without a storyline in my mind nor a letter on paper.

The only way to keep myself motivated to do this is to proclaim my victory here now. This way, if I succeed, nothing happens. But when I fail, I will fail you guys, my worthy visitors.

But do not dispair, I will finish this novel before the 30th of this month. I’ve blogged, therefor I will succeed. Bring it on, Novel! I can take you!

Categories
English World Wide Web

LinkedIn

I had some trouble logging into LinkedIn lately, so I contacted customer support.

This is the answer I got:

Categories
World Wide Web

Gmail: Langezame verbinding

Annonieme bronnen melden dat Google een dezer dagen zijn eigen ADSL lijn zal uitbrengen. Het zou hier gaan over een erg goedkope verbinding om ook de minder bedeelden ter samenleving op het internet te krijgen. Google heeft zelfs al een naam voor zijn ADSL assortiment: de Langezame (reg. trademark) verbinding. Google heeft zijn bekende mail dienst GMail in ieder geval al uitgerust met sluikreclame (klik voor meer detail):

Categories
English World Wide Web

Thijs’ guide to UTF8

The Thijs way!
This might not be the best way, but it does work like a charm!

First of all: make sure your document is written in UTF8. This will make sure that all forms you submit (yes, even in xmlHttpRequest calls) will be encoded in UTF8.

Second: make sure that your database connection is using UTF8 aswell! Using UTF8 as charset in your fields and tables is one thing, you have to make sure that your connection is using UTF8 aswell! You can change the default charset in your mysql configuration files, but the easiest way to make sure you are using UTF8 is actually telling your connection to use UTF8.
mysql_connect (DB_SERVER, DB_USER, DB_PASS);
mysql_select_db (DB_DATABASE);
mysql_query ("SET NAMES utf8");

This way you will lose those ugly non-utf8 characters in your database.

Now let’s take a look at the xmlHttpRequest. This is fairly simple: what do you want to do? Submit a form ofcourse. What do we have to take in mind? & and = are special characters, so we can’t use those in the form. A simple escape in javascript should take care of that.

The html:
<textarea id="myfield">Some data</textarea>
<button onclick="submitForm();">Submit</button>

The javascript (function submitForm()):
// Only escape the form field
var myfield = encodeURIComponent(document.getElementById('myfield').value);
var req= getXmlHttpRequest (); // Use your own function here..
req.open ('post', 'index.php', true);
req.setRequestHeader("Method", "POST index.php HTTP/1.1");
req.setRequestHeader("Content-Type", "application/x-www-form-urlencoded");
req.send ('myfield='+myfield);

How to receive it in PHP:
// Just remove the special url chars
$myfield = rawurldecode ($_POST['myfield']);

And that’s it really. Since your document is in UTF8 and your form fields are encoded in UTF8, the only thing you have to keep in mind is the “split symbols”. No need to encode or decode anything.

Categories
English World Wide Web

Generating a tree from MySQL

A few days ago I was working on a navigation tree. The basic idea is simple: every navigation item is part of another navigation item or the root item.

Seems like a simple parent-child relation, so I made a MySQL table to store everything in. Every record has a parent_id and, ofcourse, an unique id. I wanted to load the data in a multi dimensional associated array in PHP, so that I could easily include them in my HTML templates (using recursive functions).

The general aproach for these kinds of data is recursive functions, but this leads to a huge amount of overhead and a lot of useless loops. That’s why I decided to take another approach, using references.

The basic idea is simple: for every parent you create, you write a reference to this parents’ children array in a seperate array.

Take in mind that, in order to use this approach, you need to sort your data on the depth level! That means you will have to store the level of a child (= count the parent, grandparent, …) in your table. Order your data ascending on depth level, and you should be alright. If you dont do this, the script will try to add records to unexisting parents.

You can find my example script here.

Categories
Mijn Mening Mijn Wereld World Wide Web

Scholen in Zwitserland kiezen Ubuntu

Vanaf september zullen 9000 schoolcomputers in Zwitserland overschakelen op Linux. Het gebruik van Ubuntu wordt er immers door de overheid aangeraden. Tot nu toe werd Ubuntu steeds in Dual Boot gezet, maar vanaf september vliegt Windows finaal naar de vergetelheid.

Dit is natuurlijk een keuze die ik enkel kan toejuigen: door Linux al op school tegen te komen, zal Linux ongetwijfeld veel minder vreemd overkomen. Ook is het, vanuit financieel standpunt, maar eerlijk dat op school gratis software gebruikt wordt: op die manier kan iedereen ook thuis gebruik maken van de op school geleerde computa skills.

Zwitserland promoot het gebruik van open software al langer, net zoals ook de Belgische overheid tracht te doen. Helaas blijft het in Belgiรซ bij het standaardiseren van ODF (opensource document format). Ook andere overheden richten zich steeds meer op open software; zo tekende de Australische overheid in december 2007 een groot contract met Novell om Linux in verschillende departementen in te voeren.

De toekomst ligt bij Microsoft? Ik dacht het niet ๐Ÿ˜‰

Bron: ZDNet

Categories
World Wide Web

Ebay koopt 1Brood

Met veel trots mag ik officieel aankondigen dat Ebay een indrukwekkende interesse getoond heeft in 1Brood, de 1-klik lunch bestel applicatie. Ebay heeft een voorstel gedaan dat ik niet kon weigeren en mag zich nu eigenaar van 1Brood noemen.

Ikzelf blijf head of development en zal 1Brood de nieuwe tijd inleiden, met hulp van een team ervaren programmeurs en een designer van Ebay zelf.

1Brood mag grootste tijden verwachtten. Voorlopig zal het aanbod beperkt blijven tot de Belgische markt, maar vanaf 1 april 2009 hopen we dit product ook internationaal te kunnen uitbrengen.
E-bay



Update: Uiteraard gaat het hier om een 1 april grap. Opvallend hoeveel mensen mij geloofden ๐Ÿ™‚

Categories
Spam World Wide Web

Nieuws is hip!

Blackberry nieuws

Het ziet er naar uit dat het webbedrijf achter “DeRedactie.be” nog niet klaar is met ongelofelijke, vreselijke updates naar ons hoofd te gooien. Woensdag avond wou ik nog snel even kijken wat ik de laatste dagen gemist had in de wereld, krijg ik een gigantische blackberry tegen m’n hoofd gesmeten!

Bron van screenshot: Emich.