Rainy’s first release, Tomboy sync server deployment made easy

** UPDATE: All Rainy related stuff can be found in a Wiki at http://www.notesync.org – See the NEWS section there for latest release and feature information **

It’s been a while since I blogged about me hacking on Rainy, a note synchronisation/cloud server for using with Tomboy and drop-in replacement for the somewhat unmaintained Snowy. But though not blogging, development  went further and I came up with a releasable first version, that I invite everyone to try out (source & binary download links are at the bottom of this post).

Attention: Canonical has announced removal of the public note synchronization service offered with UbuntuOne for around end of February, so now would be the perfect time to help testing and report bugs, so that Rainy can be shipped as a stable and mature, free alternative for all Tomboy (and Tomdroid) users!

Single-file deployment

Notes are usually very personal, and not everyone feels comfortable storing them in the cloud, which is IMHO why Tomboy is still a good competitor to Evernote and alike. This is why I want the deployment/installation of Rainy to be easy and hassle-free as much as possible, so that everyone can setup his own instance on his own private home or cloud server.

That is why I am so happy I found the IL-Repack project on GitHub, which enables me to link all required assemblies (including those from the System.* namespace) together with my own code into a single, dependency-free CIL executable (“Rainy.exe”), which is less than 900kb of size.  You will still need the mono runtime installed, but that is nothing difficult to do, a simple “apt-get install mono-complete” or “zypper install mono-complete” should do the trick on Ubuntu/Debian or openSUSE. If you are running Tomboy, you are already running mono, so no need to install anything!

This Rainy.exe is the preferred deployment target, as it runs on all system with mono available. The intermediate code it is compiled to, makes a binary deployment quite easy, and there is not much need for a user to compile from source. I’ve tested it on Linux & Mac, and it also runs on Windows if  you (a) use mono (since we need Mono.Data.Sqlite) and (b) install a feasible native sqlite3.dll.

Building the packed executable is already part of the Makefile (“make pack” ), so feel free to roll your own blobs from the source code.

What else is new?

Compared to the initial state that I had come up within the first two weeks, there are some major changes:

• Introduction of sqlite3 as backend, which replaces the basic JSON plain-file which lacked scalability and safe-transactions
• Using ServiceStack.OrmLite as an OR mapper, so that sqlite can be easily replaced by mysql/postgres/ms-sql in the near future
• Two backends can be configured to use:
  • full sqlite, storing all notes and OAuth metadata into a single “rainy.db” sqlite3 file. Makes backups very easy, as only the “rainy.db” file needs to be backuped.
  • the XML backend that comes with tomboy-library, which stores each note in readable, plain XML files. This is perfect for small installations, as notes stay readable in any text editor and it is easy to write 3rd party Apps. However the OAuth metadata is still stored in a sqlite database when using this backend.
• Lots of additional unit tests, modelling different synchronization scenarios

How stable is it?

This is actually the very first release of Rainy, as I’ve only maintained a source tree on GitHub. I’ve tagged it “0.1″, and to sum up the stability and maturity: I’ve tested it on my private machines, using Tomboy on two different Ubuntu 12.04 and latest Tomdroid (0.7.2). Tomboy works good, I did not lose any notes in my testing, nor did I encounter any crashes. Tomdroid has some bugs, but if you start with a note and hit refresh a few times upon setup, you will get to a point where syncing works without any noticeable flaws, at least to the extend I’ve tested (which isn’t that big, I have to admit). But nevertheless: Be warned that this is beta quality software, and always do backups before testing.

Please report any bugs or issues you may find into the GitHub issue tracker. And of course, if you are a C# coder, contributions are always welcome! Find the code at http://github.com/Dynalon/Rainy

Public open beta server

I own a small virtual server instance I installed Rainy upon, which I am hereby offering as a testing playground to the public. While Rainy still lacks HTTPS support, (though this can be worked around through setting up a secure nginx reverse proxy, which I will blog about if there is demand), Tomboy does not yet have client-side note encryption, so I do not even start to pretend that this public instance is a secure cloud space. It is just not. I provide it for testing and experimentation only. Since it is not meant to be private, I have created 50 public username / password combinations that can be used by everyone. If you randomly pick one for testing, there is a good chance you won’t be disturbed by anybody else while testing

Click here for more info about the public server

Downloads & Docs

The Documentation reside in the source code as Markdown files in the GitHub repo and are also viewable in a wiki-style layout on http://rainy.notesync.org/

The single Rainy.exe binary releases can be found at http://rainy.notesync.org/release/

Announcing Rainy – a poor man’s Tomboy sync server

In this post I’d like to announce Rainy, a new synchronization/cloud server for Tomboy which I’ve been working on at the Monkeyspace Hackfest and the few weeks up to today.

To keep the expenses down for my stay at Boston, I shared the hotel room at Boston Marriott with Stefan Hammer, one of the Tomdroid maintainers (Tomdroid is a Tomboy-like client running on Android). So I got to know Stefan throughout my stay in Boston (and New York, which we both visited afterwards for sightseeing), and also got to know Jared Jennings who is working on Tomboy (and it’s Cocoa based OS X counterpart ‘Macboy’). That was the first the time I heard from Tomdroid, as well as saw it in action on Stefan’s Nexus 7 tablet. He told me how they have issues syncing the notes with Ubuntu One, and that the existing Snowy server is somewhat unmaintained and hard to get up and running.

After I heard Demis Bellot‘s talk at Monkeyspace about ServiceStack, I wanted do try it out, and actually do real-world project with it (I did experiment with it before). So I thought, why not use ServiceStack to create a drop-in replacement for Snowy. I started hacking on it at the Monkeyspace hackfest, as well as in the 4 hour ride by train to NY, and got curious. I finished it to a releasable first alpha version in the last weeks, and pushed it to github along with build instructions yesterday.

So, here are some facts about Rainy’s current state:

• written in C#
• re-uses tomboy-library, a C# library created by Jared to bundle the note management logic and data storage, that is intended to be used by Tomboy one day
• is fully compatible to the Tomboy REST API 1.0, no changes to Tomboy or Tomdroid required
• multi-user capable
• uses tomboy-library on-disk storage format, which will store the notes as flat XML files within a given directory (on a per-user basis)
• two-way syncing is working with Tomboy, tested with Ubtuntu 12.04 LTS on 2 machines accessing the same notes on the server
• creating, modifying and deleting notes is replayed on each Tomboy upon sync, so your single note repository should always be in consistent state

While Snowy is written Python/Django and uses a SQL database as backend, Rainy is designed to be more leightweight. The data store is directory based, so no database setup is required. Rainy is a command line program that will create it’s own http listener, so you don’t need to setup a webserver application. All dependencies besides mono are also C# based and 100% managed code, integrated as git submodules, that will build together with Rainy. So the current build requirements are only a recent mono install and git for checking out the sources.

Syncing with Tomdroid has some issues on the Tomdroid side, but I’ve reported the bugs and am in close contact with Stefan, who promised me to works those out  ASAP. You need to use the latest development build, not the version from the google play store.

Of course, this first release is considered alpha quality, and you should not rely on it to safely store your notes. When using Rainy, always create backups of your notes before syncing, and don’t blame me (but send bug reports) if Rainy sucks up your notes into nirvana.

If you want to give Rainy a spin (and maybe report some bugs that you find), checkout the github project page.

Back from Monkeyspace conference

I’ve arrived back in Germany yesterday evening, after a very great Monkeyspace Conference (and 3 more days of sightseeing in New York City). The conference was great, I’ve met some very cool people and caught lots of interesting talks. Thanks to all the Sponsors that made this great event possible:

At the Hackfest, David had set me up with a recent Mountain Lion setup which allowed me to eventually fix the bug that prevented  Banshee to run on ML. I’ve already twittered an experimental build which was done on David’s test machine on ML. It should be considered experimental as I usually did the builds and testing on Lion, and I cannot assure there build issues with the dependencies on ML. I would have already done a new build, but regretably British Airways or AirBerlin lost my luggage upon arrival. I’fe made a lost-luggage clain and am actively waiting for their response. In the meantime I can’t recharge my Macbook, as the power adapter was in my lost bag

 

 

I’m going to the MonkeySpace Conference in Boston, yay!

I will be attending the MonkeySpace Conference (formerly known as MonoSpace) this year. The organizators of the Conference are generously covering the entrance fee, flight and hotel costs so that I can take part in this great event. Besides the talks and discussions, there will be a hackfest with fellow C#/.NET/Mono hackers, where I will work on Banshee and showcase some of the work I did during GSoC.

I’m really looking forward to the event, and want to thank all sponsors of the conference, that make my stay possible, which I couldn’t have afforded all by myself:

The Conference will be kindly hosted at Microsoft New England Research and Development Center (NERD Center), so a special thanks goes out to Microsoft as well!

GSoC final report

Its been great three months, but everything has to come to an end someday. I took the time last week to rebase my patches against latest banshee git master, and submitted all my changes (that were not yet merged like some other patches I submitted during GSoC) to the bugzilla. Thanks to Betrand’s hard work with reviewing them during the hottest weekend this year in central europe, they are (with two exceptions) now fully merged and will be part of the upcoming Banshee 2.5 development release and (if no major issues arise) the next 2.6 stable release.

As a result I just updated the bockbuild tree at github with an up-2-date banshee.py package.

Roundup

I am very satisfied with my work during the period. If accomplished the two major tasks that I laid out in my original proposal: Adapting the bockbuild buildsystem to build banshee and a selected subset of the community extensions (BCE), as well as introducing hardware support on OS X. I am very proud with the hardware support, as this part was a major coding task and involved lot of reading low level documentation, reading C based libraries and OS X kernel framework headers. From a users point of view, it is also a good feature. iTunes, which is the usually the most used player on OS X, just has no support for syncing to Android devices or other mp3 players, locking the user very much to iPhone/iPods. With the now available implementation, users can shift to Banshee  on OS X if they don’t own an iDevice but want a sync capability.

Additionally, lots of bugs that were very obvious and annoying are now gone on OS X, making the overall use far more pleasant than before my GSoC time. I implemented native “File open” dialogs for importing media, which now allows the user to select from external drives, spotlight searches, network shares etc.

Although the GSoC period is over, I will continue to work on the OS X port (and Banshee in general). After the 2.5 feature freeze I will try to work out an annoying gtk-quartz which withholds me from calling banshee “stable” on OS X. It is hard to track down and not related to banshee but to the gtk-quartz implementation, and crashes banshee when doing drag-n-drop.

Thanks to Google for sponsoring the awesome Summer of Code program, and especially thanks to the GNOME foundation for providing a very well-organized mentorship. Also, it’s been great to work with my Mentor and the Banshee community in General, they are all very open minded and friendly people!

Obligatory Appbundle

Since it is the last report in the GSoC row and lots of bugs have been fixed, I rolled a new app-Bundle for downloading. As I am afraid of calling it stable (see above)  and is based on the banshee git master (instead of the 2.4 branch), I am just calling it “gsoc-final-bundle” instead of doing any version numbers or alpha/beta keys. The feature freeze date for 2.5 is next wednesday, and only after then I will provide “OS X 2.5″ bundles

Download Banshee-gsoc-final.dmg here.

UPDATE: Now that 2.5.0 is out, here is an updated appbundle for Download (Lion required!). All my patches made it to 2.5.0, so there are no more external patches applied in this build!

GSoc Report #5 – Bughunting days

I am a little late with my biweekly report, but here it is. Since it’s not too long till the end of the GSoC period and I’ve accomplished all tasks I wanted to do, I focused on bug hunting and polishing the code I’ve written so far.

Volume-WIDGET BUG, GSTREAMER BUG

This is a nasty one and has always bugged me, and I looked into that even before I started the GSoC assignment. When starting an OS X bundle (up to alpha4) you will notice the volume slider is always muted, and stays that way the entire time. Although, you will be able to hear music. Then, when you try to click the volume button, the popup-slider behaves very strangely: It disappears immediately after you release the mouse button, and sometimes won’t even pop up. This “volume button” issue are actually multiple bugs:

• I found out, banshee uses it’s own custom volume button widget written in C#. I blamed the misbehaving to the custom implementation, and rewrote it to reuse Gtk.VolumeButton provided by Gtk# (which wraps the Gtk+ volume button widget). To my great surprise, using the Gtk.VolumeButton resulted in exactly the same strange behavior when popping up. That was back in the days before I applied for GSoC, and I then wrote a C-based minimal test-case for plain Gtk+ that could reproduce this error using Gtk’s Quartz backend. I reported it to the gnome bugzilla along with the test-case, and sadly it still lurks there unprocessed tagged as unconfirmed. However, I’ve now taken the time to polish and submit the code that replaces banshee’s custom volume widget and throws away lots of code, that is then replaced by only few code since we can reuse Gtk.VolumeButton.
• The fact that the volume is muted on startup also manifests itself on the linux platform, as I could test with my ubuntu system. However, it’s not that big a deal on Linux since the volume gets set from the GStreamer backend right when starting to playback a track – but not on OS X (thus it remains muted). It took me hours spending in gdb and reading banshee’s C code that handles the GStreamer pipeline construction to find out why that happens. I finally could come up with a small fix that works around that, and also fixes the volume being at level zero on startup on linux (and OS X), which made it already to git master.
• Since my fix is just a mere workaround, I wanted to really find out what was causing the inconsistent behavior between the Linux and OS X platform. After again spending even more hours in gdb and reading GStreamer docs, I finally could pinpoint it and come up with a cross platform minimal C test-case, to reproduce the issue using GStreamer only. I’ve reported it back and hope it will get fixed soon by the GStreamer guys. One of the biggest problems I had when coming up with the minimal test-case, was that ubuntu’s gcc always failed to link the code, while using the exact command as I did on OS X. After (some more) hours I finally found out ubuntu changed something in gcc that makes it sensitive to the order of the option flags passed to gcc – that’s why I’ve added the compile commands for each platform as a comment in the test-case. Screw you, Ubuntu devs, for not adding a runtime hint when using the wrong order!!

Open media files from finder

I’ve also added support for opening media files directly in Finder. That means, if you install Banshee.app to /Applications, you can simply right click onto any .mp3 file, select “Open with -> Banshee.app”. This works with multiple selected files, too. I’ve opted not to default any file extension to banshee, so installing banshee will not overwrite your associated default media player.

Simplifying the build, include BCE

I’ve also worked to improve bockbuild, so it can be easier used to create application bundles from scratch on a virgin OS X system. I therefore merged the package definition files from banshee-git and banshee into a single package. Whether to build from the stable-2.4 branch or latest git master is now determined by the -r (release) flag upon build. Additionally, I’ve created a separate package banshee-bce, which is responsible for fetching and compiling latest banshee-community-extensions and is build right after banshee. I also fixed a bug that caused the bundle process to fail, because solitary was not yet compiled. Additionally, I modified the bootstrap-bundle script to write the needed environment variables into the MonoDevelop project files. The functions have been there since the early weeks of my GSoC, but until now had to be executed manually.

So, I’ve wrapped up bockbuild that much and added all sorts of auto stuff, that now it should be very easy to compile yourself your own build of banshee:

git clone git@github.com:Dynalon/bockbuild.git
cd bockbuild/profiles/banshee/
./darwin.py -bvdz

Provided you have installed latest XCode along with command line tools and git for OS X (!!!), these three commands are sufficient on a virgin OS X system to fetch and build all sources, and you will end up with a self-sustained Banshee.app residing in the bockbuild/profiles/banshee/bundle.darwin/ folder, including the BCE plugins. Use ./darwin.py  without any arguments to find out what the parameters are for. In the next report, I am going to tell you how you could then use MonoDevelop to start hacking on the banshee source code.

That is pretty much all to report for now. I know, nothing very spectacular added this time, but fixing bugs and lowering the bar for future developers by simplifying build process is important sometimes.

Mountain Lion support

Sadly, my 2008 Macbook white is too old to be officially supported by Mountain Lion, so I can’t test against it. David reported an older version (I think alpha3) to be working on the ML-RC1 out of the box, and in my github repo I reuse the banshee dependency packages from the xamarin repo, so maybe you can just fetch the latest build and run it on ML. Feedback on that would be greatly appreciated from any ML user!

Alpha5 bundle

Here is the obligatory alpha5 bundle, which is the same as alpha4 but includes the above mentioned patches.

GSoC Progress Report #4 – MidTerm passed

This is the 4th post in the series of my biweekly gsoc status reports. Last week was MidTerm deadline, and I am happy to say that I passed my MidTerm evaluation. I am a little ahead of time, actually. I proposed to have a Banshee.app ready with no working Mass Storage yet, but preview USB plug-in detection. The last alpha3 build already had full usb mass storage support working. So I had the time in the last two weeks to polish the hardware support a little.

Sync your music to network shares with Banshee

I’ve modified the OS X specifc hardware detection code a little, so that not only USB devices get recognized, but all kind of mountable volumes that have an .is_audio_player file on them. This allows one to create a fake mass storage device to sync your media to, but it’s actually just a folder on a network share. This can be any kind of network share, like smb or afp mounts, or as I am going to show step-by-step, a remote SSH location mounted by sshfs/MacFusion.

Howto: Syncing to a remote machine over SSH

I want to be able to sync my media collection and selected playlists from Banshee to my linux powered File-/DLNA-/ Server where I store all my music. I’ll only want to carry a subset of tracks on my Macbook, which has only limited disk space. This is how we can enable syncing via SSH (provided you use at least alpha4 build):

1. Login via ssh to the machine to want to sync to, and create a folder for your Music and put an .is_audio_player into it:
   mkdir ~/SharedMedia/
   mkdir ~/SharedMedia/Music
   mkdir ~/SharedMedia/Playlists
   touch ~/SharedMedia/.is_audio_player
2. Edit the .is_audio_player file with your favorite editor and have it look like this:
   audio_folders=Music/
   folder_depth=2
   output_formats=audio/ogg,audio/x-ms-wma,audio/mpeg,audio/wav,audio/x-flac
   playlist_path=Playlists/
3. On your OS X system, install MacFusion (successor of MacFuse)
4. From the MacFusion tray icon, select “Connect to Server” and enter the ssh:// address of your remote machine. In my case, this is “ssh://timo@orion/home/timo/SharedMedia/”. Replace “timo” with your username and “orion” with your  hostname of your machine. You will need to have setup ssh public key authentication for the connection to succeed!
5. Banshee will detect the network share as a generic mass storage device. You can now sync your whole media collection or single playlists between the network share and your banshee media database!

Other improvements in the hardware backend include stability fixes as well as making the “Eject” button working for the external devices.

Native file dialogs

Also new in alpha4 is the support for native file dialogs. Right now, the Gtk file open dialog is just broken on OS X. If you own a system with more than one harddrive (no matter if internal or external through usb/firewire/thunderbolt) the Gtk Filechooser will not display it. Same holds true for any network shares. Only way to access external storage is, if you know that OS X mounts those volumes in your /Volumes path – a fact that is not known to lots of OS X users (and there is no need to, as you got your external Volumes always in your sidebar). Currently only the Import Media->From File/Folder uses the OS X native file open dialog. At other points, its more complicated due to Gtk Widgets being embedded in the chooser. But those dialogs are usually less often used (export playlist dialog, choose cover art dialog).

As usual, with this bi-weekly report I’ve rolled a new alpha bundle which included my latest development progress and all features talked about in this post, as well as all previous posts. Get it here or follow this direct download link to test it yourself!

Source Code

Since the source for the hardware support and the native file choosers will likely change within the GSoC period, I’ve not submitted patches to upstream, yet. However, I’ve setup a mirror on GitHub including latest banshee master branch from gnome, and a branch with my changes (nicely formatted and rebased). A pull request against myself can be used to retrieve a single patch including all my changes for this GSoC assignment. I’ve also updated the banshee-git package in bockbuild to auto-retrieve this patch and build when doing a release build.