EDI

Table of Contents

Introduction

The core of the architecture is the rabbitmq amqp message server. Every piece of code connects in some way to it.

Most services share a couple of well defined exchanges. See Well-defined Exchanges for a description.

Setup

In theory(tm). Expect some settings to still be hard coded somewhere.

  1. Install AMQP Server. Best use RabbitMQ
  2. git clone this repo
  3. Check configuration in conf/. Especially the environment variables
  4. Create symlinks from sv/available to sv/enabled for the desired subsystems to run
  5. …
  6. Start EDI with run

Glossary

source
Apps that only/mainly produce messages
sink
Apps that only/mainly consume messages
processor
Apps that transform messages. Consume -> Produce.
bot
Consumer/Producer that add external/foreign interfaces to the system. Like IRC.
app
EDI application.

Well-defined Exchanges

msg   public

Raw messages received from somewhere. This should be something that can be parsed to a command.

Type: topic

Routing Keys

In general: Bots use protocol.bot-name.{send,recv,presence}.channel Apps listening for msg replies use app.direction.random_id.

Examples
  • irc.EDI.recv._channel_
  • irc.EDI.send._channel_
  • irc.EDI.presence
  • edish.send.13213
  • webappdemo.send.af234fabc234
Message direction

Imagine the direction, e.g send or recv, from the point of an irc bot. recv means that the bot received the message from IRC. The bot subscribes for the send direction and forwards this messages to IRC.

Messages

#.send.*

Two content types are allowed: application/json and text/plain:

  • Content-Type: application/json
    • Mandatory entries
      msg
      Message body
      user
      Destination user
    • Optional entries
      data
      Machine readable content
  • Content-Type: text/plain
    body
    Message
#.recv.*
  • Mandatory entries
    msg
    Message body
    user
    Message sender
  • Common entries
    uflags
    User Flags. Bot/App specific properties the user has in the context of the bot. The IRC bot, for example, adds the flag "op" for users with operator status in the bot's channel.
    bot
    The user name of the bot.
    chan
    Channel the message originated from
    type
    Type of message received. e.g "IRC /me", IRC privmesg, Twitter Timeline..

Apps

IRC Bot - mqbot.py
  • IRC Messages → EDI MSG with direction recv
  • Subscribes to irc.BOT_NAME.{send,presence,action}.* messages
    send
    IRC message, may be directed at a user or a channel
    presence
    Set away status
    action
    IRC action. "/me".
msg-to-cmd

Transform !<command> to cmd Messages. (See cmd Exchange)

cmd   public

Messages that do something :)

Type: topic

Messages

Content-Type: application/json

The following lists standard entries to the body's JSON:

Mandatory entries
cmd
Usually the same as the routing key when parsed from msg Messages. Could be different. Not sure why I include it. The clojure tools use the to dispatch handlers..
args
Argument string.
user
User that send the command. The command may use this to log.
Optional entries
src
Origin. Replies will be send here with the word recv replaced by send. See Msg Reply Mechanism for details
dst
Where to put the result. Implement a same default like reply based on src or default destination.

Inspect

Every command should be inspectable, e.g implement the command inspect.

A msg reply to inspect contains a machine readable description of the app and its commands. Apps can use this description to generate help and user interfaces.

Structure
app
Application short name
descr
Short description of the app
cmds
Object with keys command.
  • Each command contains an object with keys:
    descr
    Command description
    args
    Data type of the args key in cmd messages
    attribs
    Object with keys attrib. Key in cmd messages that the command uses appart from args and cmd. Apps commonly use the user attribute.
  • args types
    NONE
    No argument
    TEXT
    Arbitrary Text
    TIME
    Time string
    DATE
    Date string
    COMPLEX
    Complex Arguments. Maybe handled by getops/argparse in the app.
    WORD
Example

The app thehonestbookoftruth returns the following on inspect: 

{
   "app" : "thehonestbookoftruth",
   "descr" : "Carbon entity presence"
   "cmds" : {
      "logout-all" : {
         "descr" : "Logout all users",
         "args" : "NONE",
         "attribs" : {}
      },
      "ul" : {
         "args" : "NONE",
         "descr" : "Return list of logged in users and ETAs",
         "attribs" : {}
      },
      "uneta" : {
         "args" : "NONE",
         "descr" : "Remove ETA",
         "attribs" : {
            "user" : "Remove ETA from this user"
         }
      },
      "eta" : {
         "descr" : "Set ETA. Supports HHMM, HH:MM, HH:MM:SS, HHMMSS",
         "args" : "TIME",
         "attribs" : {
            "user" : "User to set ETA for"
         }
      },
      "logout" : {
         "attribs" : {
            "user" : "User to log out"
         },
         "descr" : "Logout user",
         "args" : "NONE"
      },
      "login" : {
         "attribs" : {
            "user" : "User to log in"
         },
         "descr" : "Login user",
         "args" : "NONE"
      }
   },
}

Msg Reply Mechanism

To reply data back to the command's origin the src field of the command be used. The src field however is optional, the originator may no be able/interested in replies.

To reply to commands create a edi msg with:

Routing key
cmd.src with string recv replaced with send
(no term)
Entries:
user
User from cmd
msg
Message payload

Consider adding a data entry with machine readable data.

See also #.recv.*

notify   public

Sink exchange for notifications.

Routing Keys

  • audio
  • text

Sinks

mplayer one-liner
amqp-consume --url="amqp://mopp" --exchange="notify" --routing-key="audio" mplayer -

Messages

Content-Type depending on exchange keys. Should be directly usable by the sink (e.g mp3 file to hand over to mplayer).

subinit   private

Sink exchange for subinit messages

Type: topic

Messages

Content-type: text/plain

Must always contain the same as the routing key.

Payload/Routing Key

rc.$level.$action

level
Integer
action
start or stop
Example Payloads
rc.2.start
Execute start scripts for runlevel 2
rc.4.stop
Execute stop scripts for runlevel 4

act_433mhz   private

Sink exchange to signal 433mhz transmitter.

Type: fanout

Messages

Commandline arguments for `rcswitch-pi`.

act_mpd   private

Sink exchange. Forwards payload to local mpc tool.

Type: direct Routing key: Instance name, e.g "subraum"

act_dmx   private

Sink exchange for DMX.

Routing Keys and Payloads

dmx.lamp.$INSTANCE.control

Payload: on or off

dmx.lamp.$INSTANCE.$ID

Payload regex: (\d,\d,\d|html-farbe|programmname)

Example

Routing key: dmx.lamp.subraum.8, Payload: background. Run DMX program background for lamp with id 8.

sens_token   private

Token = Something someone has. Think of smart card ids, Ethernet MAC addresses, etc.

The exchange is for producers, like smart card readers or DHCP servers, to communicate with processors like token login.

Routing Key

location.protocol.action

Actions: add, /del Example: subraum.ethernet.add

Payload

Token in a format specific to the protocol field in the routing key.

Example: For protocol ethernet the payload is the mac address in colon hex format.

Libraries and Tools

listen_commands

Helper tool to connect arbitrary tools to edi without using AMQP directly. Executes an app for every command received.

The commands payload is passed in a format specified with --data via stdin.

Environment Variables

listen_commands passes some command data through environment variables:

EDI_CMD
cmd entry in cmd message
EDI_USER
user entry in cmd message
EDI_CMD_ARGS
args entry in cmd message
EDI_DATA_FD
When --autoreply is specified listen_commands reads machine readable reply data on this ft.

Example

listen_command \
    --cmd wetter \
    --autoreply \
    --name "weather" \
    --description "Wetterbericht - Subraum und das da draussen" \
    --exe "weather.sh"

Executes the shell script weather.sh for every command wetter received. It also registers a inspect command from the meta data provided.

Note the --autoreply: The programs stdout is captured and replied back using the msg reply mechanism.

pyedi

Pyedi is a utility library for python that supports registering commands, message handlers and emitting various defined message types.

See it's documentation in lib/python/README.org

Software, Libs, etc.

Debian packages

  • rabbitmq-server
  • python-amqplib
  • amqp-tools

docker

For development docker seems a good choice: 

sudo docker run -p :5672 -p :15672 -v /scratch/docker-data/rabbitmq:/var/lib/rabbitmq/mnesia f04150b0661e
sudo docker build github.com/mikaelhg/docker-rabbitmq.git

Note that the exchanges are configured by hand..

Use mopp, running on the dell netbook.

Development

Install requirements. Setup exchanges in rabbitmq. The web interfaces comes in handy here ;)

Repository Organization

apps
EDI applications
bin
Common EDI tools
conf
Configuration for apps
doc
Documentation
etc
Misc tools and example code
lib
EDI helper libraries like pyedi
log
Log output for sv/enabled daemons.
sv
Scripts to run EDI apps

Most larger tools are subtree merged from elsewhere. This repo is kind of the collected deployment branch.

Have something to add? Let me pull your repo!

Libraries

Python

pika
http://pika.readthedocs.org/en/latest/ Documented, Async lib
amqplib
simpler non-threaded library; documentation shipped in the .py files. Which are quite readable ;)
pyedi
See Libraries and Tools

Commandline

amqp-tools
Make sure you get the recent ones. Debian testing works quite well. Debian stable not so.

Clojure

langohr
http://clojurerabbitmq.info/ Excellent library.

Created: 2014-09-25 Do 16:36

Validate