Tag Archives: mono

Running Rainy on Raspbian/arm-hf on my Raspberry Pi

For some time, mono did only support the arm-el (soft-float) platform. But luckily, thanks to the works of Alex Rønne Peterson the latest mono git master code is capable of running on the arm-hf (hard-float) platform. This is important for most RaspberryPi users that are running Raspbian. Raspbian is a Debian flavor recompiled for arm-hf to exploit the RaspberryPi’s hard-float capabilities that quite some RPi users have installed (including myself). So up until now, your only choice to run Rainy was to stick with Debian Wheezy for armel. But given the latest additions to the mono runtime this is not the case anymore.

To run Rainy on the more popular Raspbian, you’d need to fetch latest mono master from  GitHub and compile from scratch. This took around 18h on my RPi. Sounds much? Yes, indeed. That is why I created a precompiled bundle of Rainy 0.5.0 compiled into native (arm-hf) code. You don’t even need to install mono at all as it has the mono runtime embedded into the fat binary, which is thus 12 Megabyte large.

Get precompiled Rainy for Raspbian 0.5.0 from here.

UPDATE: I was notified that there are issues with this binary, complaining about missing mscorlib.dll. I am investigating this and will post updates in this blog as soon as there are news on this.

UPDATE2: The issue with missing mscorlib.dll was fixed in newer mono versions, run “make linux_bundle” on the Pi to create a statically linked version of Rainy. Binaries for download will soon be available.

I will soon move the Rainy testserver installation  to my Raspberry Pi, depending on the stability of the mono build.
UPDATE: While Rainy runs fine on the RaspberryPi, it is somewhat slow. Although it is sufficient for a small group of users, the Demoserver is used by many to get a first glance of Rainy and I want it to perform as good as possible. So I’ve moved the Demoserver  to the smallest instance of a DigitalOcean SSD based droplet which costs only $5/month. You can get your own Droplet and support Rainy’s development by using this referral link to DigitalOcean.

GSoC – Final report

GSoC 2013 is coming to an end, and in this posting I’d like to inform you about the new things that I added to Rainy since Midterm.

The second half of the GSoC was mainly about adding HTML5 interfaces. Although there are now two different interfaces – admin and user (note editing) client – they use the same technologies. After some evaluation, I setup both with AngularJS and Bootstrap. Bootstrap was especially helpful, since I am by no means a creative designer that could come up with a nice design, so using the (nowadays somewhat common) bootstrap default elements resulted in an acceptable overall look.

Single page applications – single executable

The interfaces are pure single-page applications (SPA) – that means they interact solely via JSON/REST with the backend, there is no server side HTML generation involved. So the major code parts of the interfaces are Javascript codes and HTML/Bootstrap skeletons. This has the advantage that they are just plain static files, which can be served by a tiny webserver – and directly via the HttpListener with a custom ServiceStack handler that is spawned by Rainy.

This is the part I like the most in this solution: All website files (.html, .js, .css etc.) are embedded as a resource into the Rainy.exe file and served by the builtin webserver. That means: No messy folders with assets, no setting up pathes in any configuration whatsoever. Deployment is still a single Rainy.exe file that only requires mono as a dependency.

Admin interface

The admin interface is the first thing that will pop open in your browser if you start Rainy on a system with a graphical environment (read: have a $DISPLAY variable set). It can be used to add, remove and edit users, as well as temporarily deactivate user accounts. The admin password is global, and needs to be specified in the settings.conf file prior to starting Rainy.

admin1

admin2

NOTE EDITING

The note editing client was the more difficult part, especially the WYSIWYG editor. Tomboy uses a custom note XML format, and converting this to HTML and back without messing up the markup is not that easy. The bootstrap-wysihtml5 widget was of good help, but unfortunately has some bugs and the projects seems to be abandoned (wysihtml5, which is part of bootstrap-wysihtml5) on github. As a result, the WYSIWYG sometimes messes up the note formatting when saving, but I am confident that this can be sorted out in the future.

Nonetheless, viewing notes with custom markup works now pretty good. There is also a search field which responds as you type, so no server callbacks are involved. notes1

As soon as you change a note’s text, create or delete a note, the Sync button will become available and start a timer of 300 seconds. After 300 seconds or when you manually click the sync button the notes are synced with the upstream server version. You don’t have to sync after editing every note, changes are temporarily saved. So you can switch back and forth between the notes and make changes. Sync will only happen when the timer reaches zero or you manually press the sync button. For future versions I plan a complete offline mode that syncs the notes to local storage, but I will have to figure out what is the best way to do this.

Host it for your friends and family

For smaller groups up to 20 people, the SQLite backend will perform sufficiently well. But with the PostgreSQL support added before Midterm, Rainy is now ready to serve many more users at once. That is why I added the possibility to signup for an account. In the settings.conf, you can specifiy if signup is allowed (default: off) and whether or not users need to be manually activated by an admin after signup (default: on). This allows you to invite your friends, family members or co-workers to use your custom Rainy installation – hosted in a secure environment at home or at your data center of choice (remember that storage encryption was also added as part of the GSoC). You can even host it on a RaspberryPi running Debian! (sorry, no Raspbian yet due to missing ARMHF support in mono)

signup

loginwsignup

Give it a try – the public test server

The public test server that has been in place for months is updated to the latest release, and I try to keep it aligned with the latest git master. The HTML5 note interface is available for testing, too. Head over to the public test server page in the Rainy wiki or directly go to the login.

New RELEASE: v0.5.0

I’ve just uploaded a new release which contains all new features up to today. The v0.5.0 is the unstable branch that should evolve into a v0.6 stable release later after more testing. Get it from the Rainy download page and let me know of any bugs or issues that you have in the github issue tracker.

GSoC 2013 – First progress report

As you might know, I was accepted to work on Rainy during Google Summer of Code 2013. It is a great opportunity to turn Rainy into something great, and I am happy to give my first status update about what I’ve been doing the last three weeks and what I am up to in the next time.

Rainy has grown to an extend where I thought it might be a good idea to have public website (not only a blog about it) online, and for some month the home of Rainy has been and still is located at http://dynalon.github.io/Rainy. Check for releases, news, infos and updates that are Rainy specific.

Week 1: PostgreSQL Support & Cleanup

In the first week, I planned to add PostgreSQL support. Since I plan to add more database backends in the future (MySQL, MSSQL) I first had to refactor some classes and interfaces that allow to plug in other backends at a later point. This was the first time I got in contact with the Inversion of Control pattern, as I ended up refactoring most of Rainy’s classes to use constructor-injection and auto wiring using ServiceStack’s Funq IoC container implementation. This was very fun to learn, and I like the concept, especially since ServiceStack uses the IoC  constructor injection pattern everywhere and you can easily hook into it.

After the refactoring, I added the PostgreSQL backend, which was very easy since I already used ServiceStack.OrmLite as an O/R mapper which brings the necessary abstractions.

So, Rainy can now be used using either the SQLite or PostgreSQL backend, allowing to scale for a much larger number of users. My first benchmarks show that SQLite is actually faster than Postgre, but I measured only using a small number of notes and users. For larger databases I think PostgreSQL will outperform SQLite.

Week 2 & 3: Server side encryption

This is one of my favorite tasks that I added to the GSoC proposal, and I think will be the most valueable one. Especially since recent events in the media should have raised awareness of data privacy in the cloud to a broader audience 🙂

Unfortunately, Tomboy and Tomdroid do not yet support full client side encryption of notes, that would enable us to store them safely in the cloud. To overcome this, we can at least implement server-side encryption in Rainy. Notes are encrypted on the fly using AES256 before they are stored in the database, and decrypted transparently upon requesting the notes. The required keys should of course not be stored in plaintext in Rainy’s database, so that it is non-fatal if someday a Rainy server-instance gets hacked and the database is made public.

The crypto implementation was one of the harder pieces in my proposal, and I am happy to tell that it is done and has landed in the git master. For anybody interested in the details how the crypto and the key management works, I wrote developer docs that can be viewed here. Since a basic understanding of encryption is required, I will discus only a easy-to-understand approach here, for details see the developer docs.

What the encryption does

  • works completely in the background and on the fly, no setup on the clients is necessary
  • encrypts every note with a per-note key, which is itself encrypted with the users master key
  • the master key is not stored in the database, but part of the users oauth credentials. That way, if the database gets public the notes can’t be decrypted without the users master key
  • if you lose a device (say your cell phone with Tomdroid) and you immediatley revoke the access token for the device before an attacker finds it, the attacker cannot sync again (but might access the notes that are stored on your phone). No need for changing the password or revoke all tokens
  • Change your password at any time, without re-authenticating any device or client

What the encryption does not

  • replace client-side encryption (which is sadly not available yet)
  • protect your from malicous server administrators – they can still modify Rainy’s code and hijack your keys which should only reside in the RAM

What’s next

For the next weeks I’ve planned to add a Time-Machine like “Go back to version” function into Rainy, that will save all note versions into the database and allows to access any older revision. Very much like the TimeMachine mode os OS X or Dropbox allows to go back to any previous version of a file.

This requires the Tomboy REST API to be extended and might not land into the client software too soon. But it  will  be in the  HTML5 Webinterface client that I will also be doing as part of the 2nd half of the GSoC. More about that in a future blog post 😉

Rainy’s first release, Tomboy sync server deployment made easy

** UPDATE: All Rainy related stuff can be found in a Wiki at http://dynalon.github.io/Rainy – See the NEWS section there for latest release and feature information **

It’s been a while since I blogged about me hacking on Rainy, a note synchronisation/cloud server for using with Tomboy and drop-in replacement for the somewhat unmaintained Snowy. But though not blogging, development  went further and I came up with a releasable first version, that I invite everyone to try out (source & binary download links are at the bottom of this post).

Attention: Canonical has announced removal of the public note synchronization service offered with UbuntuOne for around end of February, so now would be the perfect time to help testing and report bugs, so that Rainy can be shipped as a stable and mature, free alternative for all Tomboy (and Tomdroid) users!

Single-file deployment

Notes are usually very personal, and not everyone feels comfortable storing them in the cloud, which is IMHO why Tomboy is still a good competitor to Evernote and alike. This is why I want the deployment/installation of Rainy to be easy and hassle-free as much as possible, so that everyone can setup his own instance on his own private home or cloud server.

That is why I am so happy I found the IL-Repack project on GitHub, which enables me to link all required assemblies (including those from the System.* namespace) together with my own code into a single, dependency-free CIL executable (“Rainy.exe”), which is less than 900kb of size.  You will still need the mono runtime installed, but that is nothing difficult to do, a simple “apt-get install mono-complete” or “zypper install mono-complete” should do the trick on Ubuntu/Debian or openSUSE. If you are running Tomboy, you are already running mono, so no need to install anything!

This Rainy.exe is the preferred deployment target, as it runs on all system with mono available. The intermediate code it is compiled to, makes a binary deployment quite easy, and there is not much need for a user to compile from source. I’ve tested it on Linux & Mac, and it also runs on Windows if  you (a) use mono (since we need Mono.Data.Sqlite) and (b) install a feasible native sqlite3.dll.

Building the packed executable is already part of the Makefile (“make pack” ), so feel free to roll your own blobs from the source code.

What else is new?

Compared to the initial state that I had come up within the first two weeks, there are some major changes:

  • Introduction of sqlite3 as backend, which replaces the basic JSON plain-file which lacked scalability and safe-transactions
  • Using ServiceStack.OrmLite as an OR mapper, so that sqlite can be easily replaced by mysql/postgres/ms-sql in the near future
  • Two backends can be configured to use:
    • full sqlite, storing all notes and OAuth metadata into a single “rainy.db” sqlite3 file. Makes backups very easy, as only the “rainy.db” file needs to be backuped.
    • the XML backend that comes with tomboy-library, which stores each note in readable, plain XML files. This is perfect for small installations, as notes stay readable in any text editor and it is easy to write 3rd party Apps. However the OAuth metadata is still stored in a sqlite database when using this backend.
  • Lots of additional unit tests, modelling different synchronization scenarios

How stable is it?

This is actually the very first release of Rainy, as I’ve only maintained a source tree on GitHub. I’ve tagged it “0.1”, and to sum up the stability and maturity: I’ve tested it on my private machines, using Tomboy on two different Ubuntu 12.04 and latest Tomdroid (0.7.2). Tomboy works good, I did not lose any notes in my testing, nor did I encounter any crashes. Tomdroid has some bugs, but if you start with a note and hit refresh a few times upon setup, you will get to a point where syncing works without any noticeable flaws, at least to the extend I’ve tested (which isn’t that big, I have to admit). But nevertheless: Be warned that this is beta quality software, and always do backups before testing.

Please report any bugs or issues you may find into the GitHub issue tracker. And of course, if you are a C# coder, contributions are always welcome! Find the code at http://github.com/Dynalon/Rainy

Public open beta server

I own a small virtual server instance I installed Rainy upon, which I am hereby offering as a testing playground to the public. While Rainy still lacks HTTPS support, (though this can be worked around through setting up a secure nginx reverse proxy, which I will blog about if there is demand), Tomboy does not yet have client-side note encryption, so I do not even start to pretend that this public instance is a secure cloud space. It is just not. I provide it for testing and experimentation only. Since it is not meant to be private, I have created 50 public username / password combinations that can be used by everyone. If you randomly pick one for testing, there is a good chance you won’t be disturbed by anybody else while testing 🙂

Click here for more info about the public server

Downloads & Docs

The Documentation reside in the source code as Markdown files in the GitHub repo and are also viewable in a wiki-style layout on http://dynalon.github.io/Rainy

The single Rainy.exe binary releases can be found at http://rainy.notesync.org/release/

Announcing Rainy – a poor man’s Tomboy sync server

In this post I’d like to announce Rainy, a new synchronization/cloud server for Tomboy which I’ve been working on at the Monkeyspace Hackfest and the few weeks up to today.

To keep the expenses down for my stay at Boston, I shared the hotel room at Boston Marriott with Stefan Hammer, one of the Tomdroid maintainers (Tomdroid is a Tomboy-like client running on Android). So I got to know Stefan throughout my stay in Boston (and New York, which we both visited afterwards for sightseeing), and also got to know Jared Jennings who is working on Tomboy (and it’s Cocoa based OS X counterpart ‘Macboy’). That was the first the time I heard from Tomdroid, as well as saw it in action on Stefan’s Nexus 7 tablet. He told me how they have issues syncing the notes with Ubuntu One, and that the existing Snowy server is somewhat unmaintained and hard to get up and running.

After I heard Demis Bellot‘s talk at Monkeyspace about ServiceStack, I wanted do try it out, and actually do real-world project with it (I did experiment with it before). So I thought, why not use ServiceStack to create a drop-in replacement for Snowy. I started hacking on it at the Monkeyspace hackfest, as well as in the 4 hour ride by train to NY, and got curious. I finished it to a releasable first alpha version in the last weeks, and pushed it to github along with build instructions yesterday.

So, here are some facts about Rainy’s current state:

  • written in C#
  • re-uses tomboy-library, a C# library created by Jared to bundle the note management logic and data storage, that is intended to be used by Tomboy one day
  • is fully compatible to the Tomboy REST API 1.0, no changes to Tomboy or Tomdroid required
  • multi-user capable
  • uses tomboy-library on-disk storage format, which will store the notes as flat XML files within a given directory (on a per-user basis)
  • two-way syncing is working with Tomboy, tested with Ubtuntu 12.04 LTS on 2 machines accessing the same notes on the server
  • creating, modifying and deleting notes is replayed on each Tomboy upon sync, so your single note repository should always be in consistent state

While Snowy is written Python/Django and uses a SQL database as backend, Rainy is designed to be more leightweight. The data store is directory based, so no database setup is required. Rainy is a command line program that will create it’s own http listener, so you don’t need to setup a webserver application. All dependencies besides mono are also C# based and 100% managed code, integrated as git submodules, that will build together with Rainy. So the current build requirements are only a recent mono install and git for checking out the sources.

Syncing with Tomdroid has some issues on the Tomdroid side, but I’ve reported the bugs and am in close contact with Stefan, who promised me to works those out  ASAP. You need to use the latest development build, not the version from the google play store.

Of course, this first release is considered alpha quality, and you should not rely on it to safely store your notes. When using Rainy, always create backups of your notes before syncing, and don’t blame me (but send bug reports) if Rainy sucks up your notes into nirvana.

If you want to give Rainy a spin (and maybe report some bugs that you find), checkout the github project page.

GSoC Progress Report #4 – MidTerm passed

This is the 4th post in the series of my biweekly gsoc status reports. Last week was MidTerm deadline, and I am happy to say that I passed my MidTerm evaluation. I am a little ahead of time, actually. I proposed to have a Banshee.app ready with no working Mass Storage yet, but preview USB plug-in detection. The last alpha3 build already had full usb mass storage support working. So I had the time in the last two weeks to polish the hardware support a little.

Sync your music to network shares with Banshee

I’ve modified the OS X specifc hardware detection code a little, so that not only USB devices get recognized, but all kind of mountable volumes that have an .is_audio_player file on them. This allows one to create a fake mass storage device to sync your media to, but it’s actually just a folder on a network share. This can be any kind of network share, like smb or afp mounts, or as I am going to show step-by-step, a remote SSH location mounted by sshfs/MacFusion.

Howto: Syncing to a remote machine over SSH

I want to be able to sync my media collection and selected playlists from Banshee to my linux powered File-/DLNA-/ Server where I store all my music. I’ll only want to carry a subset of tracks on my Macbook, which has only limited disk space. This is how we can enable syncing via SSH (provided you use at least alpha4 build):

  1. Login via ssh to the machine to want to sync to, and create a folder for your Music and put an .is_audio_player into it:
    mkdir ~/SharedMedia/
    mkdir ~/SharedMedia/Music
    mkdir ~/SharedMedia/Playlists
    touch ~/SharedMedia/.is_audio_player
  2. Edit the .is_audio_player file with your favorite editor and have it look like this:
    audio_folders=Music/
    folder_depth=2
    output_formats=audio/ogg,audio/x-ms-wma,audio/mpeg,audio/wav,audio/x-flac
    playlist_path=Playlists/
  3. On your OS X system, install MacFusion (successor of MacFuse)
  4. From the MacFusion tray icon, select “Connect to Server” and enter the ssh:// address of your remote machine. In my case, this is “ssh://timo@orion/home/timo/SharedMedia/”. Replace “timo” with your username and “orion” with your  hostname of your machine. You will need to have setup ssh public key authentication for the connection to succeed!
  5. Banshee will detect the network share as a generic mass storage device. You can now sync your whole media collection or single playlists between the network share and your banshee media database! 🙂

Other improvements in the hardware backend include stability fixes as well as making the “Eject” button working for the external devices.

Native file dialogs

Also new in alpha4 is the support for native file dialogs. Right now, the Gtk file open dialog is just broken on OS X. If you own a system with more than one harddrive (no matter if internal or external through usb/firewire/thunderbolt) the Gtk Filechooser will not display it. Same holds true for any network shares. Only way to access external storage is, if you know that OS X mounts those volumes in your /Volumes path – a fact that is not known to lots of OS X users (and there is no need to, as you got your external Volumes always in your sidebar). Currently only the Import Media->From File/Folder uses the OS X native file open dialog. At other points, its more complicated due to Gtk Widgets being embedded in the chooser. But those dialogs are usually less often used (export playlist dialog, choose cover art dialog).

As usual, with this bi-weekly report I’ve rolled a new alpha bundle which included my latest development progress and all features talked about in this post, as well as all previous posts. Get it here or follow this direct download link to test it yourself!

Source Code

Since the source for the hardware support and the native file choosers will likely change within the GSoC period, I’ve not submitted patches to upstream, yet. However, I’ve setup a mirror on GitHub including latest banshee master branch from gnome, and a branch with my changes (nicely formatted and rebased). A pull request against myself can be used to retrieve a single patch including all my changes for this GSoC assignment. I’ve also updated the banshee-git package in bockbuild to auto-retrieve this patch and build when doing a release build.

GSoc progress report #3

This is my 3rd post in the series of my bi-weekly status reports for my GSoC banshee OS X assignment. This two weeks period was actually quite more fun than the weeks before, as I shifted from doing packaging and bugfixing work (which is sometimes very time consuming and frustrating when dealing autotools and  recompiling packages over and over again to check for regressions) to actual coding work.

Hardware support

Part of my self-set midterm goal is bringing basic support for hardware devices in OSX banshee, in particular to USB mass storage devices. Mass storage devices are very common: From the first generation of mp3 players, the mass storage usb  driver was used to put mp3 files into the players flash memory and it is still widely used today: All recent Android phones support it, as well as low-cost consumer mp3 player sticks and more sophisticated “feature phones” are also often loadable via usb mass storage. That makes having mass storage support in OS X banshee very important.

OS X disk arbitration library

After little research, I found there are several methods to access usb devices on OS X. There is the IOKit framework, that can be used to handle everything usb related, as far as low level device driver programming. Luckily, there is another Framework that is a little higher on the device layer while not being bound too strictly to USB and fits perfectly for our purposes: the DiskArbitration framework. You can use it to hook callbacks whenever a disk device (being a USB/Firewire mountable volume like a usb stick or harddrive, network shares or .dmg images) appears or disappears to the system. So I chose to go for it and started research.

Unfortunately, neither the IOKit nor the DiskArbitration framework are yet bound in MonoMac. That meant I had a number of choices:

  • writing myself a C based glue layer to interface with the frameworks
  • bind the needed function calls myself with DllImport P/Invokes and wrap the datatypes

The first way is usually an easier one from a programmers perspective, as I could do everything in C and just write a very basic public interface that is marshaled by the mono runtime. Future changes in the framework’s public ABI can be spotted very easily, as the C compiler will complain about missing functions or changes datatypes.

However, I chose the second way as usually prefer keeping one language in a project. Banshee uses quite some C gluelayers at the moment, bug there are efforts to remove them and someday have a completely C# based codebase.

Although MonoMac did not bind the IOKit/DiskArbitration framework, there is limited support for the CoreFoundation framework which helped doing a managed-only solution. I did never work on low-level OS X APIs before, neither did I work with MonoMac, so I spent nearly a full-time week digging into the Header files (which turned out to be the best documentation for the API) and finding out about MonoMac, and how I could handle the native CFObject types and marhsal them into C#.

If I had more knowledge about MonoMac and OS X internals, I’d have tried to contribute them to MonoMac by binding the missing frameworks. I am however glad I didn’t chose that in the first place, as it turned out I only needed a handful of funtions wrapped, and especially IOKit is a huge beast with far more complexity needed for my purposes. So right now, there are two small static IOKit and DiskArbitation classes that expose the needed library calls via the public static extern DllImport mechanism.

Hooking into disk arbitration wasn’t the difficult part. Turned out, for device recognition I needed the usb vendorId/productId pairs, and those were not provided by disk arbtitration (as its not bound to the USB bus). It took me days to figure out how to do that, and I had a chance to catch up with my rusty C abilities, as I did the prototype completely in C, and then rebuild it in C#.

Outcome

I managed to get USB device recognition working quite well (as far as I could test). In the latest alpha build (see below), you can plugin usb mass storage devices at will, before or during banshee runs, and we hook into all necessary events (device unmounting, hard usb removal because life is to short to remove usb safely) and banshee will refresh the devices list.

The usb vendor/product ID pairs are exposed, so lots of device abilities are detected automatically. My Samsung Galaxy S (with CM9) is correctly detected as an android device, and I tested a simple USB stick with an .is_audio_player file to pretend it being a MP3 player. There might be glitches with devices having more than one partition on them, I have yet to test that.

Syncing works (again as far as I could test) although I had to apply two non-hardware backend related patches (1,2) to make the process work seamlessly.

As I only have a limited number of devices, I invite everyone to test with your devices, and give me feedback so I can further improve the mass storage support.

Minor improvements

The alpha3 build I put together includes, besides the very large hardware-support patch (that is not yet in upstream as it will likely change in the next weeks), enabled  UPnP/DLNA client support . There was not much to do, just create a bockbuild package dependency and it worked out of the box, same way as it does on Linux.

As the folks from xamarin pointed out to me in IRC, there are known problems with Glib 2.32 on OS X which severely impact performance. The (quite old) gcc Apple ships seems to have problem with gcc interlocked/atomic intrinsic (compare-and-swap and alike). I downgraded to Glib 2.30 by reusing the xamarin package and to my great surprise I could actually feel it. The UI is a little more responsive, but the great change is the playback: While I often had sound stuttering when the system was under load, that is completely gone now. That change is also included in the latest alpha build.

New alpha build release

As I had done with the two progress reports before, I’ve bundled a new Banshee.app (this time alpha3) ready for everyone to download and test. Get it from here and do not forget to read the instructions provided. Feedback is greatly welcome, you could use the banshee mailing list for that.