Running Rainy on Raspbian/arm-hf on my Raspberry Pi

For some time, mono did only support the arm-el (soft-float) platform. But luckily, thanks to the works of Alex Rønne Peterson the latest mono git master code is capable of running on the arm-hf (hard-float) platform. This is important for most RaspberryPi users that are running Raspbian. Raspbian is a Debian flavor recompiled for arm-hf to exploit the RaspberryPi’s hard-float capabilities that quite some RPi users have installed (including myself). So up until now, your only choice to run Rainy was to stick with Debian Wheezy for armel. But given the latest additions to the mono runtime this is not the case anymore.

To run Rainy on the more popular Raspbian, you’d need to fetch latest mono master from  GitHub and compile from scratch. This took around 18h on my RPi. Sounds much? Yes, indeed. That is why I created a precompiled bundle of Rainy 0.5.0 compiled into native (arm-hf) code. You don’t even need to install mono at all as it has the mono runtime embedded into the fat binary, which is thus 12 Megabyte large.

Get precompiled Rainy for Raspbian 0.5.0 from here.

UPDATE: I was notified that there are issues with this binary, complaining about missing mscorlib.dll. I am investigating this and will post updates in this blog as soon as there are news on this.

UPDATE2: The issue with missing mscorlib.dll was fixed in newer mono versions, run “make linux_bundle” on the Pi to create a statically linked version of Rainy. Binaries for download will soon be available.

I will soon move the Rainy testserver installation  to my Raspberry Pi, depending on the stability of the mono build.
UPDATE: While Rainy runs fine on the RaspberryPi, it is somewhat slow. Although it is sufficient for a small group of users, the Demoserver is used by many to get a first glance of Rainy and I want it to perform as good as possible. So I’ve moved the Demoserver  to the smallest instance of a DigitalOcean SSD based droplet which costs only $5/month. You can get your own Droplet and support Rainy’s development by using this referral link to DigitalOcean.

GSoC – Final report

GSoC 2013 is coming to an end, and in this posting I’d like to inform you about the new things that I added to Rainy since Midterm.

The second half of the GSoC was mainly about adding HTML5 interfaces. Although there are now two different interfaces – admin and user (note editing) client – they use the same technologies. After some evaluation, I setup both with AngularJS and Bootstrap. Bootstrap was especially helpful, since I am by no means a creative designer that could come up with a nice design, so using the (nowadays somewhat common) bootstrap default elements resulted in an acceptable overall look.

Single page applications – single executable

The interfaces are pure single-page applications (SPA) – that means they interact solely via JSON/REST with the backend, there is no server side HTML generation involved. So the major code parts of the interfaces are Javascript codes and HTML/Bootstrap skeletons. This has the advantage that they are just plain static files, which can be served by a tiny webserver – and directly via the HttpListener with a custom ServiceStack handler that is spawned by Rainy.

This is the part I like the most in this solution: All website files (.html, .js, .css etc.) are embedded as a resource into the Rainy.exe file and served by the builtin webserver. That means: No messy folders with assets, no setting up pathes in any configuration whatsoever. Deployment is still a single Rainy.exe file that only requires mono as a dependency.

Admin interface

The admin interface is the first thing that will pop open in your browser if you start Rainy on a system with a graphical environment (read: have a $DISPLAY variable set). It can be used to add, remove and edit users, as well as temporarily deactivate user accounts. The admin password is global, and needs to be specified in the settings.conf file prior to starting Rainy.

NOTE EDITING

The note editing client was the more difficult part, especially the WYSIWYG editor. Tomboy uses a custom note XML format, and converting this to HTML and back without messing up the markup is not that easy. The bootstrap-wysihtml5 widget was of good help, but unfortunately has some bugs and the projects seems to be abandoned (wysihtml5, which is part of bootstrap-wysihtml5) on github. As a result, the WYSIWYG sometimes messes up the note formatting when saving, but I am confident that this can be sorted out in the future.

Nonetheless, viewing notes with custom markup works now pretty good. There is also a search field which responds as you type, so no server callbacks are involved. 

As soon as you change a note’s text, create or delete a note, the Sync button will become available and start a timer of 300 seconds. After 300 seconds or when you manually click the sync button the notes are synced with the upstream server version. You don’t have to sync after editing every note, changes are temporarily saved. So you can switch back and forth between the notes and make changes. Sync will only happen when the timer reaches zero or you manually press the sync button. For future versions I plan a complete offline mode that syncs the notes to local storage, but I will have to figure out what is the best way to do this.

Host it for your friends and family

For smaller groups up to 20 people, the SQLite backend will perform sufficiently well. But with the PostgreSQL support added before Midterm, Rainy is now ready to serve many more users at once. That is why I added the possibility to signup for an account. In the settings.conf, you can specifiy if signup is allowed (default: off) and whether or not users need to be manually activated by an admin after signup (default: on). This allows you to invite your friends, family members or co-workers to use your custom Rainy installation – hosted in a secure environment at home or at your data center of choice (remember that storage encryption was also added as part of the GSoC). You can even host it on a RaspberryPi running Debian! (sorry, no Raspbian yet due to missing ARMHF support in mono)

Give it a try – the public test server

The public test server that has been in place for months is updated to the latest release, and I try to keep it aligned with the latest git master. The HTML5 note interface is available for testing, too. Head over to the public test server page in the Rainy wiki or directly go to the login.

New RELEASE: v0.5.0

I’ve just uploaded a new release which contains all new features up to today. The v0.5.0 is the unstable branch that should evolve into a v0.6 stable release later after more testing. Get it from the Rainy download page and let me know of any bugs or issues that you have in the github issue tracker.

GSoC 2013 – First progress report

As you might know, I was accepted to work on Rainy during Google Summer of Code 2013. It is a great opportunity to turn Rainy into something great, and I am happy to give my first status update about what I’ve been doing the last three weeks and what I am up to in the next time.

Rainy has grown to an extend where I thought it might be a good idea to have public website (not only a blog about it) online, and for some month the home of Rainy has been and still is located at http://dynalon.github.io/Rainy. Check for releases, news, infos and updates that are Rainy specific.

Week 1: PostgreSQL Support & Cleanup

In the first week, I planned to add PostgreSQL support. Since I plan to add more database backends in the future (MySQL, MSSQL) I first had to refactor some classes and interfaces that allow to plug in other backends at a later point. This was the first time I got in contact with the Inversion of Control pattern, as I ended up refactoring most of Rainy’s classes to use constructor-injection and auto wiring using ServiceStack’s Funq IoC container implementation. This was very fun to learn, and I like the concept, especially since ServiceStack uses the IoC  constructor injection pattern everywhere and you can easily hook into it.

After the refactoring, I added the PostgreSQL backend, which was very easy since I already used ServiceStack.OrmLite as an O/R mapper which brings the necessary abstractions.

So, Rainy can now be used using either the SQLite or PostgreSQL backend, allowing to scale for a much larger number of users. My first benchmarks show that SQLite is actually faster than Postgre, but I measured only using a small number of notes and users. For larger databases I think PostgreSQL will outperform SQLite.

Week 2 & 3: Server side encryption

This is one of my favorite tasks that I added to the GSoC proposal, and I think will be the most valueable one. Especially since recent events in the media should have raised awareness of data privacy in the cloud to a broader audience 🙂

Unfortunately, Tomboy and Tomdroid do not yet support full client side encryption of notes, that would enable us to store them safely in the cloud. To overcome this, we can at least implement server-side encryption in Rainy. Notes are encrypted on the fly using AES256 before they are stored in the database, and decrypted transparently upon requesting the notes. The required keys should of course not be stored in plaintext in Rainy’s database, so that it is non-fatal if someday a Rainy server-instance gets hacked and the database is made public.

The crypto implementation was one of the harder pieces in my proposal, and I am happy to tell that it is done and has landed in the git master. For anybody interested in the details how the crypto and the key management works, I wrote developer docs that can be viewed here. Since a basic understanding of encryption is required, I will discus only a easy-to-understand approach here, for details see the developer docs.

What the encryption does

• works completely in the background and on the fly, no setup on the clients is necessary
• encrypts every note with a per-note key, which is itself encrypted with the users master key
• the master key is not stored in the database, but part of the users oauth credentials. That way, if the database gets public the notes can’t be decrypted without the users master key
• if you lose a device (say your cell phone with Tomdroid) and you immediatley revoke the access token for the device before an attacker finds it, the attacker cannot sync again (but might access the notes that are stored on your phone). No need for changing the password or revoke all tokens
• Change your password at any time, without re-authenticating any device or client

What the encryption does not

• replace client-side encryption (which is sadly not available yet)
• protect your from malicous server administrators – they can still modify Rainy’s code and hijack your keys which should only reside in the RAM

What’s next

For the next weeks I’ve planned to add a Time-Machine like “Go back to version” function into Rainy, that will save all note versions into the database and allows to access any older revision. Very much like the TimeMachine mode os OS X or Dropbox allows to go back to any previous version of a file.

This requires the Tomboy REST API to be extended and might not land into the client software too soon. But it  will  be in the  HTML5 Webinterface client that I will also be doing as part of the 2nd half of the GSoC. More about that in a future blog post 😉

Rainy’s first release, Tomboy sync server deployment made easy

** UPDATE: All Rainy related stuff can be found in a Wiki at http://dynalon.github.io/Rainy – See the NEWS section there for latest release and feature information **

It’s been a while since I blogged about me hacking on Rainy, a note synchronisation/cloud server for using with Tomboy and drop-in replacement for the somewhat unmaintained Snowy. But though not blogging, development  went further and I came up with a releasable first version, that I invite everyone to try out (source & binary download links are at the bottom of this post).

Attention: Canonical has announced removal of the public note synchronization service offered with UbuntuOne for around end of February, so now would be the perfect time to help testing and report bugs, so that Rainy can be shipped as a stable and mature, free alternative for all Tomboy (and Tomdroid) users!

Single-file deployment

Notes are usually very personal, and not everyone feels comfortable storing them in the cloud, which is IMHO why Tomboy is still a good competitor to Evernote and alike. This is why I want the deployment/installation of Rainy to be easy and hassle-free as much as possible, so that everyone can setup his own instance on his own private home or cloud server.

That is why I am so happy I found the IL-Repack project on GitHub, which enables me to link all required assemblies (including those from the System.* namespace) together with my own code into a single, dependency-free CIL executable (“Rainy.exe”), which is less than 900kb of size.  You will still need the mono runtime installed, but that is nothing difficult to do, a simple “apt-get install mono-complete” or “zypper install mono-complete” should do the trick on Ubuntu/Debian or openSUSE. If you are running Tomboy, you are already running mono, so no need to install anything!

This Rainy.exe is the preferred deployment target, as it runs on all system with mono available. The intermediate code it is compiled to, makes a binary deployment quite easy, and there is not much need for a user to compile from source. I’ve tested it on Linux & Mac, and it also runs on Windows if  you (a) use mono (since we need Mono.Data.Sqlite) and (b) install a feasible native sqlite3.dll.

Building the packed executable is already part of the Makefile (“make pack” ), so feel free to roll your own blobs from the source code.

What else is new?

Compared to the initial state that I had come up within the first two weeks, there are some major changes:

• Introduction of sqlite3 as backend, which replaces the basic JSON plain-file which lacked scalability and safe-transactions
• Using ServiceStack.OrmLite as an OR mapper, so that sqlite can be easily replaced by mysql/postgres/ms-sql in the near future
• Two backends can be configured to use:
  • full sqlite, storing all notes and OAuth metadata into a single “rainy.db” sqlite3 file. Makes backups very easy, as only the “rainy.db” file needs to be backuped.
  • the XML backend that comes with tomboy-library, which stores each note in readable, plain XML files. This is perfect for small installations, as notes stay readable in any text editor and it is easy to write 3rd party Apps. However the OAuth metadata is still stored in a sqlite database when using this backend.
• Lots of additional unit tests, modelling different synchronization scenarios

How stable is it?

This is actually the very first release of Rainy, as I’ve only maintained a source tree on GitHub. I’ve tagged it “0.1”, and to sum up the stability and maturity: I’ve tested it on my private machines, using Tomboy on two different Ubuntu 12.04 and latest Tomdroid (0.7.2). Tomboy works good, I did not lose any notes in my testing, nor did I encounter any crashes. Tomdroid has some bugs, but if you start with a note and hit refresh a few times upon setup, you will get to a point where syncing works without any noticeable flaws, at least to the extend I’ve tested (which isn’t that big, I have to admit). But nevertheless: Be warned that this is beta quality software, and always do backups before testing.

Please report any bugs or issues you may find into the GitHub issue tracker. And of course, if you are a C# coder, contributions are always welcome! Find the code at http://github.com/Dynalon/Rainy

Public open beta server

I own a small virtual server instance I installed Rainy upon, which I am hereby offering as a testing playground to the public. While Rainy still lacks HTTPS support, (though this can be worked around through setting up a secure nginx reverse proxy, which I will blog about if there is demand), Tomboy does not yet have client-side note encryption, so I do not even start to pretend that this public instance is a secure cloud space. It is just not. I provide it for testing and experimentation only. Since it is not meant to be private, I have created 50 public username / password combinations that can be used by everyone. If you randomly pick one for testing, there is a good chance you won’t be disturbed by anybody else while testing 🙂

Click here for more info about the public server

Downloads & Docs

The Documentation reside in the source code as Markdown files in the GitHub repo and are also viewable in a wiki-style layout on http://dynalon.github.io/Rainy

The single Rainy.exe binary releases can be found at http://rainy.notesync.org/release/

Announcing Rainy – a poor man’s Tomboy sync server

In this post I’d like to announce Rainy, a new synchronization/cloud server for Tomboy which I’ve been working on at the Monkeyspace Hackfest and the few weeks up to today.

To keep the expenses down for my stay at Boston, I shared the hotel room at Boston Marriott with Stefan Hammer, one of the Tomdroid maintainers (Tomdroid is a Tomboy-like client running on Android). So I got to know Stefan throughout my stay in Boston (and New York, which we both visited afterwards for sightseeing), and also got to know Jared Jennings who is working on Tomboy (and it’s Cocoa based OS X counterpart ‘Macboy’). That was the first the time I heard from Tomdroid, as well as saw it in action on Stefan’s Nexus 7 tablet. He told me how they have issues syncing the notes with Ubuntu One, and that the existing Snowy server is somewhat unmaintained and hard to get up and running.

After I heard Demis Bellot‘s talk at Monkeyspace about ServiceStack, I wanted do try it out, and actually do real-world project with it (I did experiment with it before). So I thought, why not use ServiceStack to create a drop-in replacement for Snowy. I started hacking on it at the Monkeyspace hackfest, as well as in the 4 hour ride by train to NY, and got curious. I finished it to a releasable first alpha version in the last weeks, and pushed it to github along with build instructions yesterday.

So, here are some facts about Rainy’s current state:

• written in C#
• re-uses tomboy-library, a C# library created by Jared to bundle the note management logic and data storage, that is intended to be used by Tomboy one day
• is fully compatible to the Tomboy REST API 1.0, no changes to Tomboy or Tomdroid required
• multi-user capable
• uses tomboy-library on-disk storage format, which will store the notes as flat XML files within a given directory (on a per-user basis)
• two-way syncing is working with Tomboy, tested with Ubtuntu 12.04 LTS on 2 machines accessing the same notes on the server
• creating, modifying and deleting notes is replayed on each Tomboy upon sync, so your single note repository should always be in consistent state

While Snowy is written Python/Django and uses a SQL database as backend, Rainy is designed to be more leightweight. The data store is directory based, so no database setup is required. Rainy is a command line program that will create it’s own http listener, so you don’t need to setup a webserver application. All dependencies besides mono are also C# based and 100% managed code, integrated as git submodules, that will build together with Rainy. So the current build requirements are only a recent mono install and git for checking out the sources.

Syncing with Tomdroid has some issues on the Tomdroid side, but I’ve reported the bugs and am in close contact with Stefan, who promised me to works those out  ASAP. You need to use the latest development build, not the version from the google play store.

Of course, this first release is considered alpha quality, and you should not rely on it to safely store your notes. When using Rainy, always create backups of your notes before syncing, and don’t blame me (but send bug reports) if Rainy sucks up your notes into nirvana.

If you want to give Rainy a spin (and maybe report some bugs that you find), checkout the github project page.