domingo, 29 de abril de 2012

Buddycloud Media Server Architecture

After some conversations with my mentor (dodo), we finally have a bigger picture of how should work the media server communication flow, here is a short description of the "posting" media functionality:

  1. Client discovers the channels media server;
    • Since it is a XMPP Component, it should be "discoverable";
  2. Negotiate media url; 
  3. Client receives a link for public channels and a cookie for private ones;
  4. Upload data, PUT or POST request to the media server http static file server.

Here is an architecture overview diagram:

Where:
  • Media Server:
    • HTTP FIle Server: static file server, where clients will be able to GET and PUT (or POST) files;
    • XMPP Component: component that will evaluate user's permissions and also generate valid URLs for the client download or upload files through the HTTP File Server.
  • Storage:
    • Medatada: stores files metadata, like checksum, upload date, etc;
    • Binary Files: stores the files themselves (and also their previews).
That means, that both Media Server and Client should talk two "languages", HTTP and XMPP.  But this would change (or not) after Denis, the other buddycloud's GSoC student, finishes his project.

quarta-feira, 25 de abril de 2012

Google Summer of Code 2012

This year I'm participating of the Google Summer of Code for the first time, I'll be working with an exciting XMPP project called buddycloud.

The project title is An XMPP Media Server, it basically consists of the development of an XMPP based media server that will have a HTTP face to facilitate the use by different types of clients.

The media server should be responsible for getting, adding and deleting of files (images, videos and documents), treating properly of the authentication process and also providing previews if the client doesn't want to download the whole stored file. That's why this project should bring even more developers for the XMPP world.

I will be reporting all the process evolution, problems, challenges and ideas at this blog =)

sexta-feira, 6 de abril de 2012

Hadoop + Amazon EC2 - Using Apache Whirr

Apache Whirr is a set of libraries for running cloud services. You can use it to create EC2 instances running Hadoop, just like the previous tutorial. But, Whirr also provides a lot of other interesting functionality that worth to give a try.

Other advantage is the possibility to run into different cloud providers without worrying about each one configuration details. Of course that you need an active account.

You can find a simple tutorial at Whirr's homepage: Getting Started with Whirr.

quarta-feira, 14 de março de 2012

Hadoop + Amazon EC2 - An updated tutorial

There is an old tutorial placed at Hadoop's wiki page: http://wiki.apache.org/hadoop/AmazonEC2, but recently I had to follow this tutorial and I noticed that it doesn't cover some new Amazon functionality.

To follow this tutorial is recommended that you are already familiar with the basics of Hadoop, a very useful "how to start" tutorial can be found at Hadoop's homepage: http://hadoop.apache.org/. Also, you have to be familiar with at least Amazon EC2 internals and instance definitions.

When you register an account at Amazon AWS you receive 750 hours to run
t1.micro instances, but unfortunately, you can't successfully run Hadoop in such machines.

On the following steps, when a command starts with $ means that it should be executed into the local machine, and with # into the EC2 instance.

Create an X.509 Certificate


Since we gonna use ec2-tools, our account at AWS needs a valid X.509 Certificate:
  • Create .ec2 folder:
  • $ mkdir ~/.ec2
  • Login in at AWS
    • Select “Security Credentials” and at "Access Credentials" click on "X.509 Certificates";
    • You have two options:
      • Create certificate using command line:
      • $ cd ~/.ec2; openssl genrsa -des3 -out my-pk.pem 2048
        $ openssl rsa -in my-pk.pem -out my-pk-unencrypt.pem
        $ openssl req -new -x509 -key my-pk.pem -out my-cert.pem -days 1095
        • It only works if your machine date is ok.
      • Create the certificate using the site and download the private-key (remember to put it at ~/.ec2).

Setting up Amazon EC2-Tools

  • Download and unpack ec2-tools;
  • Edit your ~/.profile to export all variables needed by ec2-tools, so you don't have to do it every time that you open a prompt:
    • Here is an example of what should be appended to the ~/.profile file:
      • export JAVA_HOME=/usr/lib/jvm/java-6-sun
      • export EC2_HOME=~/ec2-api-tools-*
      • export PATH=$PATH:$EC2_HOME/bin
      • export EC2_CERT=~/.ec2/my-cert.pem
    • To access an instance, you need to be authenticated (obvious security reasons), in this way, you have to create a Key Pair (public and private keys):
      • At https://console.aws.amazon.com/ec2/home, click on "Key Pairs", or
      • You can run the following commands:
      • $ ec2-add-keypair my-keypair | grep –v KEYPAIR > ~/.ec2/id_rsa-keypair
        $ chmod 600 ~/.ec2/id_rsa-keypair

Setting up Hadoop


After download and unpack Hadoop, you have to edit the EC2 configuration script present at src/contrib/ec2/bin/hadoop-ec2-env.sh.
  • AWS variables
    • These variables are related to your AWS account (AWS_ACCOUNT_ID, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY), they can be found at logging at your account, in Security Credentials;
      • The AWS_ACCOUNT_ID is your 12 digit account number.
  • Security variables
    • The security variables (EC2_KEYDIR, KEY_NAME, PRIVATE_KEY_PATH), are the ones related to the launch and access of an EC2 instance;
    • You have to save the private key into your EC2_KEYDIR path.
  • Select an AMI
    • Depending on Hadoop's version that you want to run (HADOOP_VERSION) and the instance type (INSTANCE_TYPE), you should use a properly image to deploy your instance:
    • There are many public AMI images that you can use (they must suit the needs for most users), to list, type
    • $ ec2-describe-images -x all | grep hadoop
    • Or you can build your own image, and upload it to an Amazon S3 bucket;
    • After selecting the AMI you will use, there are basically three variables to edit at hadoop-ec2-env.sh:
      • S3_BUCKET: the bucket where is placed the image that you will use, example hadoop-images,
      • ARCH: the architecture of the AMI image you have chosen (i386 or x84_64) and
      • BASE_AMI_IMAGE: the unique code that maps an AMI image, example ami-2b5fba42.
    • Other configurable variable is the JAVA_VERSION, there you can define which version will be installed along with the instance:
      • You can also provide a link where would be located the binary (JAVA_BINARY_URL), for instance, if you have JAVA_VERSION=1.6.0_29, an option is use JAVA_BINARY_URL=http://download.oracle.com/otn-pub/java/jdk/6u29-b11/jdk-6u29-linux-i586.bin.

Running!

  • You can add the content of src/contrib/ec2/bin to your PATH variable so you will be able to run the commands indepentend from where the prompt is open;
  • To launch a EC2 cluster and start Hadoop, you use the following command. The arguments are the cluster name (hadoop-test) and the number of slaves (2). When the cluster boots, the public DNS name will be printed to the console.
  • $ hadoop-ec2 launch-cluster hadoop-test 2
  • To login at the master node from your "cluster" you type:
  • $ hadoop-ec2 login hadoop-test
  • Once you are logged into the master node you will be able to start the job:
    • For example, to test your cluster, you can run a pi calculation that is already provided by the hadoop*-examples.jar:
    • # cd /usr/local/hadoop-*
      # bin/hadoop jar hadoop-*-examples.jar pi 10 10000000
  • You can check your job progress at http://MASTER_HOST:50030/. Where MASTER_HOST is the host name returned after the cluster started.
  • After your job has finished, the cluster remains alive. To shutdown you use the following command:
  • $ hadoop-ec2 terminate-cluster hadoop-test
  • Remember that in Amazon EC2, the instances are charged per hour, so if you only wanted to do tests, you can play with the cluster for some more minutes.

quarta-feira, 10 de agosto de 2011

Audio player toy using Gstreamer

In order to learn more about some other Open Source projects and different development environments, I started to read about Gstreamer.

The idea was implement a very simple player, where the medias that will be played are passed by argument in the command line.

The main pipeline has only one element, the playbin2, witch has the autonomy to autodetect the file extension and, for example, choose the most appropriate outputstream. Because of this, the player has the "capacity" of playing both audio and video files.

Was very interesting playing with an area of computer science that I never had any kind of experience before.

Here is the code:


#include
#include


GstTagList *tags;


static void
print_tags()
{
gchar *artist = "Artist";
gchar *title = "Music";

/* If already known, get the Artist and Title tags of the media */
gst_tag_list_get_string(tags, GST_TAG_ARTIST, &artist);
gst_tag_list_get_string(tags, GST_TAG_TITLE, &title);
g_print("%s - %s :", artist, title);
}

static gboolean
cb_print_position (GstElement *pipeline)
{
GstFormat fmt = GST_FORMAT_TIME;
gint64 pos, len;

/* Keep printing informations about the media that is playing */

print_tags();

if (gst_element_query_position (pipeline, &fmt, &pos)
&& gst_element_query_duration (pipeline, &fmt, &len)) {
g_print (" %" GST_TIME_FORMAT " / %" GST_TIME_FORMAT "\r",
GST_TIME_ARGS (pos), GST_TIME_ARGS (len));
}

/* call me again */
return TRUE;
}

static gboolean
bus_call (GstBus *bus,
GstMessage *msg,
gpointer data)
{
GMainLoop *loop = (GMainLoop *) data;

switch (GST_MESSAGE_TYPE (msg)) {

case GST_MESSAGE_EOS:
g_print ("\nEnd of stream\n");
g_main_loop_quit (loop);
break;

case GST_MESSAGE_ERROR: {
gchar *debug;
GError *error;

gst_message_parse_error (msg, &error, &debug);
g_free (debug);

g_printerr ("Bus error: %s\n", error->message);
g_error_free (error);

g_main_loop_quit (loop);
break;
}

case GST_MESSAGE_TAG: {
GstTagList *newTags = gst_tag_list_new();
gst_message_parse_tag(msg, &newTags);
tags = gst_tag_list_merge(tags, newTags, GST_TAG_MERGE_PREPEND);

gst_tag_list_free(newTags);

break;
}

default:
break;
}

/* call me again */
return TRUE;
}

static void
add_bus_watch(GstElement *pipeline,
GMainLoop *loop)
{
GstBus *bus;
bus = gst_pipeline_get_bus (GST_PIPELINE (pipeline));
gst_bus_add_watch (bus, bus_call, loop);
gst_object_unref (bus);
}

static GstElement *
init_pipeline(GMainLoop *loop)
{
GstElement *playbin;

/* Create gstreamer elements */
playbin = gst_element_factory_make ("playbin2", "play");

if (!playbin) {
g_printerr ("One element could not be created. Exiting.\n");
return NULL;
}

add_bus_watch(playbin, loop);

return playbin;
}

gboolean
set_uri(GstElement *playbin,
const char *uri)
{
/* Set the input filename to the source element */
GError *error = NULL;
g_object_set (G_OBJECT (playbin), "uri",
gst_filename_to_uri(uri, &error), NULL);

if (error) {
g_print("Filename to URI error: %s\n", error->message);
g_error_free(error);

return FALSE;
}

return TRUE;
}

gint
main (int argc,
char *argv[])
{
GMainLoop *loop;
GstElement *pipeline;
gint i;

/* Initialisation */
gst_init (&argc, &argv);

if (argc < 2)
g_error ("Usage: %s ...", argv[0]);

loop = g_main_loop_new (NULL, FALSE);
pipeline = init_pipeline(loop);

if (!pipeline)
return -1;

g_timeout_add (200, (GSourceFunc) cb_print_position, pipeline);

for (i = 1; i < argc; i++) {
if (!set_uri(pipeline, argv[i])) {
continue;
}

tags = gst_tag_list_new();

/* Start playing */
g_print ("Running...\n");
gst_element_set_state (pipeline, GST_STATE_PLAYING);
g_main_loop_run (loop);

/* Out of the main loop, clean up nicely */
g_print ("Returned, stopping playback\n");
gst_element_set_state (pipeline, GST_STATE_NULL);
gst_tag_list_free(tags);

/* Has the execution ended? */
g_print("\n");
if (i < argc - 1) {
g_print("Next track...\n\n");
} else {
g_print ("Deleting pipeline\n");
gst_object_unref (GST_OBJECT (pipeline));
g_print("End of execution!\n");
}
}

return 0;
}

sexta-feira, 29 de julho de 2011

Playing with Linked Lists

It seems silly post some code that lies in a such basic concept, but, lately I was asking myself about my basic coding skills, and decided to practice to see how I'm doing.

Since my first semester in College, I had never implemented again a Linked List manually, actually, I had never implemented it in C, only in Pascal. During the coding process, I became impressed on how much I had to struggle to simply add new elements in order to the list.

Here is the code (very ugly, btw):


#include
#include

typedef struct nodeT {
int value;
struct nodeT *next;
} node;

//insert always at the head
void insertBeginning(int number, node **head) {
node *newNode = (node *) malloc(sizeof(node));
newNode->value = number;
newNode->next = *head;
*head = newNode;
}

//insert in order
void insertInOrderRecursive(int number, node **head) {
//walk through the tail
if (!(*head)->next) {
node *newNode = (node *) malloc(sizeof(node));
newNode->value = number;
if ((*head)->value > number) {
newNode->next = *head;
*head = newNode;
} else {
newNode->next = NULL;
(*head)->next = newNode;
}
//found the right place
} else if ((*head)->value >= number) {
node *newNode = (node *) malloc(sizeof(node));
newNode->value = number;
newNode->next = *head;
*head = newNode;
} else {
//walk to the next node
insertInOrderRecursive(number, &(*head)->next);
}
}

void walkThrough(node *head) {
if (!head->next) {
printf(" %d\n", head->value);
return;
}

printf("%d -> ", head->value);
walkThrough(head->next);
}

int main() {

//init the linked list
node *list = (node *) malloc(sizeof(node));
list->next = NULL;

printf("Enter a number:\n");
int number;
scanf("%d", &number);

list->value = number;

//insert numbers
while (1) {
printf("Enter a number: (-1 to stop)\n");
scanf("%d", &number);

insertInOrderRecursive(number, &list);
walkThrough(list);

if (number == -1) {
break;
}
}

return 0;
}

quinta-feira, 28 de julho de 2011

OurGrid - Bug 871

This bug was consequence of a strange behavior of Openfire XMPP Server. Some s2s messages were arriving out of order.

We struggled through different approaches and ideas until find out "the truth", here is the complete history:

A new Peer wanted to join OurGrid's community, in this way, it joined to LSD's Discovery Service and began to be shown at http://status.ourgrid.org/, but, sometimes it strangely disappears and appears again. This Peer had a particular feature, it was not connecting to xmpp.ourgrid.org [1], but in another one (Openfire's too).

Changing the logs level to debug, we verified that LSD's Discovery Service was receiving failure notifications of this Peer, what was not expected, because it was never stopped. After some debugging, we discovered that Commune was sending failure notifications because it was receiving messages with unexpected sequence number (Commune should shutdown the connection with an external component that sends messages with sequence numbers different of those that are expected).

The first approach was, of course, verify Smack (that is used by Commune) to find possible bugs that could interfere at the order of arriving messages, but, after some Google research, we discovered that this issue was a known and already registered bug of Openfire. In this way, we tried to install other XMPP Servers, like ejabberd and Tigase, although, they are very different of Openfire, witch has a pretty and user friendly webconsole (where you can change all the configurations). Tigase did not work at all and ejabberd was very "slow" and also, some control messages like the components status, were not being delivered.

After those tries, the only solution envisioned was the implementation of a buffer for reordering messages at Application's level (Commune). So, every time that a message arrives, the application verifies if it has the expected sequence number, if not, the message should be buffered and wait for a limited timeout for the right sequence message.

The solution was implemented, tested and util there, everything worked fine.
The OurGrid's version 4.2.4 was released and deployed...
Users started to run their jobs...
The community froze, the jobs were not ending and some Workers were being allocated for the Brokers permanently...
Let's do some log analyse...
Messages are getting lost!
Hmm... Seems that the out of order messages buffer is not working very well.

What do to now? After doing some "backtracking", we remembered that ejabberd is the most used XMPP Server by all kind of organizations and has a very active community, but... Why? It is very slow... And what about some larger messages, like the OurGrid's components status?

The answer: you're doing it wrong! There is a lot of configuration properties at ejabberd, like c2s and s2s shappers, max_stanza_size for file transfers, and etc. After properly configurations, ejabberd is working very well without out of order messages traded between distinct servers.

The truth: Openfire has a very strange bug, maybe caused by a concurrency problem and not fixed until the most recent release, ejabberd hasn't, maybe for this (and a lot of other reasons), ejabberd is currently the most used.

The lesson: doesn't matter how much your boss is putting pressure on you, never leave a tentative (specially the one that is more likely to succeed) until you have investigate it as deep as possible (or until your boss commands you to, of course).

--

[1] - OurGrid's public XMPP Server