Tag Archives: mac

Rainy’s first release, Tomboy sync server deployment made easy

** UPDATE: All Rainy related stuff can be found in a Wiki at http://dynalon.github.io/Rainy – See the NEWS section there for latest release and feature information **

It’s been a while since I blogged about me hacking on Rainy, a note synchronisation/cloud server for using with Tomboy and drop-in replacement for the somewhat unmaintained Snowy. But though not blogging, development  went further and I came up with a releasable first version, that I invite everyone to try out (source & binary download links are at the bottom of this post).

Attention: Canonical has announced removal of the public note synchronization service offered with UbuntuOne for around end of February, so now would be the perfect time to help testing and report bugs, so that Rainy can be shipped as a stable and mature, free alternative for all Tomboy (and Tomdroid) users!

Single-file deployment

Notes are usually very personal, and not everyone feels comfortable storing them in the cloud, which is IMHO why Tomboy is still a good competitor to Evernote and alike. This is why I want the deployment/installation of Rainy to be easy and hassle-free as much as possible, so that everyone can setup his own instance on his own private home or cloud server.

That is why I am so happy I found the IL-Repack project on GitHub, which enables me to link all required assemblies (including those from the System.* namespace) together with my own code into a single, dependency-free CIL executable (“Rainy.exe”), which is less than 900kb of size.  You will still need the mono runtime installed, but that is nothing difficult to do, a simple “apt-get install mono-complete” or “zypper install mono-complete” should do the trick on Ubuntu/Debian or openSUSE. If you are running Tomboy, you are already running mono, so no need to install anything!

This Rainy.exe is the preferred deployment target, as it runs on all system with mono available. The intermediate code it is compiled to, makes a binary deployment quite easy, and there is not much need for a user to compile from source. I’ve tested it on Linux & Mac, and it also runs on Windows if  you (a) use mono (since we need Mono.Data.Sqlite) and (b) install a feasible native sqlite3.dll.

Building the packed executable is already part of the Makefile (“make pack” ), so feel free to roll your own blobs from the source code.

What else is new?

Compared to the initial state that I had come up within the first two weeks, there are some major changes:

  • Introduction of sqlite3 as backend, which replaces the basic JSON plain-file which lacked scalability and safe-transactions
  • Using ServiceStack.OrmLite as an OR mapper, so that sqlite can be easily replaced by mysql/postgres/ms-sql in the near future
  • Two backends can be configured to use:
    • full sqlite, storing all notes and OAuth metadata into a single “rainy.db” sqlite3 file. Makes backups very easy, as only the “rainy.db” file needs to be backuped.
    • the XML backend that comes with tomboy-library, which stores each note in readable, plain XML files. This is perfect for small installations, as notes stay readable in any text editor and it is easy to write 3rd party Apps. However the OAuth metadata is still stored in a sqlite database when using this backend.
  • Lots of additional unit tests, modelling different synchronization scenarios

How stable is it?

This is actually the very first release of Rainy, as I’ve only maintained a source tree on GitHub. I’ve tagged it “0.1”, and to sum up the stability and maturity: I’ve tested it on my private machines, using Tomboy on two different Ubuntu 12.04 and latest Tomdroid (0.7.2). Tomboy works good, I did not lose any notes in my testing, nor did I encounter any crashes. Tomdroid has some bugs, but if you start with a note and hit refresh a few times upon setup, you will get to a point where syncing works without any noticeable flaws, at least to the extend I’ve tested (which isn’t that big, I have to admit). But nevertheless: Be warned that this is beta quality software, and always do backups before testing.

Please report any bugs or issues you may find into the GitHub issue tracker. And of course, if you are a C# coder, contributions are always welcome! Find the code at http://github.com/Dynalon/Rainy

Public open beta server

I own a small virtual server instance I installed Rainy upon, which I am hereby offering as a testing playground to the public. While Rainy still lacks HTTPS support, (though this can be worked around through setting up a secure nginx reverse proxy, which I will blog about if there is demand), Tomboy does not yet have client-side note encryption, so I do not even start to pretend that this public instance is a secure cloud space. It is just not. I provide it for testing and experimentation only. Since it is not meant to be private, I have created 50 public username / password combinations that can be used by everyone. If you randomly pick one for testing, there is a good chance you won’t be disturbed by anybody else while testing 🙂

Click here for more info about the public server

Downloads & Docs

The Documentation reside in the source code as Markdown files in the GitHub repo and are also viewable in a wiki-style layout on http://dynalon.github.io/Rainy

The single Rainy.exe binary releases can be found at http://rainy.notesync.org/release/

Advertisements

GSoC Progress Report #4 – MidTerm passed

This is the 4th post in the series of my biweekly gsoc status reports. Last week was MidTerm deadline, and I am happy to say that I passed my MidTerm evaluation. I am a little ahead of time, actually. I proposed to have a Banshee.app ready with no working Mass Storage yet, but preview USB plug-in detection. The last alpha3 build already had full usb mass storage support working. So I had the time in the last two weeks to polish the hardware support a little.

Sync your music to network shares with Banshee

I’ve modified the OS X specifc hardware detection code a little, so that not only USB devices get recognized, but all kind of mountable volumes that have an .is_audio_player file on them. This allows one to create a fake mass storage device to sync your media to, but it’s actually just a folder on a network share. This can be any kind of network share, like smb or afp mounts, or as I am going to show step-by-step, a remote SSH location mounted by sshfs/MacFusion.

Howto: Syncing to a remote machine over SSH

I want to be able to sync my media collection and selected playlists from Banshee to my linux powered File-/DLNA-/ Server where I store all my music. I’ll only want to carry a subset of tracks on my Macbook, which has only limited disk space. This is how we can enable syncing via SSH (provided you use at least alpha4 build):

  1. Login via ssh to the machine to want to sync to, and create a folder for your Music and put an .is_audio_player into it:
    mkdir ~/SharedMedia/
    mkdir ~/SharedMedia/Music
    mkdir ~/SharedMedia/Playlists
    touch ~/SharedMedia/.is_audio_player
  2. Edit the .is_audio_player file with your favorite editor and have it look like this:
    audio_folders=Music/
    folder_depth=2
    output_formats=audio/ogg,audio/x-ms-wma,audio/mpeg,audio/wav,audio/x-flac
    playlist_path=Playlists/
  3. On your OS X system, install MacFusion (successor of MacFuse)
  4. From the MacFusion tray icon, select “Connect to Server” and enter the ssh:// address of your remote machine. In my case, this is “ssh://timo@orion/home/timo/SharedMedia/”. Replace “timo” with your username and “orion” with your  hostname of your machine. You will need to have setup ssh public key authentication for the connection to succeed!
  5. Banshee will detect the network share as a generic mass storage device. You can now sync your whole media collection or single playlists between the network share and your banshee media database! 🙂

Other improvements in the hardware backend include stability fixes as well as making the “Eject” button working for the external devices.

Native file dialogs

Also new in alpha4 is the support for native file dialogs. Right now, the Gtk file open dialog is just broken on OS X. If you own a system with more than one harddrive (no matter if internal or external through usb/firewire/thunderbolt) the Gtk Filechooser will not display it. Same holds true for any network shares. Only way to access external storage is, if you know that OS X mounts those volumes in your /Volumes path – a fact that is not known to lots of OS X users (and there is no need to, as you got your external Volumes always in your sidebar). Currently only the Import Media->From File/Folder uses the OS X native file open dialog. At other points, its more complicated due to Gtk Widgets being embedded in the chooser. But those dialogs are usually less often used (export playlist dialog, choose cover art dialog).

As usual, with this bi-weekly report I’ve rolled a new alpha bundle which included my latest development progress and all features talked about in this post, as well as all previous posts. Get it here or follow this direct download link to test it yourself!

Source Code

Since the source for the hardware support and the native file choosers will likely change within the GSoC period, I’ve not submitted patches to upstream, yet. However, I’ve setup a mirror on GitHub including latest banshee master branch from gnome, and a branch with my changes (nicely formatted and rebased). A pull request against myself can be used to retrieve a single patch including all my changes for this GSoC assignment. I’ve also updated the banshee-git package in bockbuild to auto-retrieve this patch and build when doing a release build.

GSoc progress report #3

This is my 3rd post in the series of my bi-weekly status reports for my GSoC banshee OS X assignment. This two weeks period was actually quite more fun than the weeks before, as I shifted from doing packaging and bugfixing work (which is sometimes very time consuming and frustrating when dealing autotools and  recompiling packages over and over again to check for regressions) to actual coding work.

Hardware support

Part of my self-set midterm goal is bringing basic support for hardware devices in OSX banshee, in particular to USB mass storage devices. Mass storage devices are very common: From the first generation of mp3 players, the mass storage usb  driver was used to put mp3 files into the players flash memory and it is still widely used today: All recent Android phones support it, as well as low-cost consumer mp3 player sticks and more sophisticated “feature phones” are also often loadable via usb mass storage. That makes having mass storage support in OS X banshee very important.

OS X disk arbitration library

After little research, I found there are several methods to access usb devices on OS X. There is the IOKit framework, that can be used to handle everything usb related, as far as low level device driver programming. Luckily, there is another Framework that is a little higher on the device layer while not being bound too strictly to USB and fits perfectly for our purposes: the DiskArbitration framework. You can use it to hook callbacks whenever a disk device (being a USB/Firewire mountable volume like a usb stick or harddrive, network shares or .dmg images) appears or disappears to the system. So I chose to go for it and started research.

Unfortunately, neither the IOKit nor the DiskArbitration framework are yet bound in MonoMac. That meant I had a number of choices:

  • writing myself a C based glue layer to interface with the frameworks
  • bind the needed function calls myself with DllImport P/Invokes and wrap the datatypes

The first way is usually an easier one from a programmers perspective, as I could do everything in C and just write a very basic public interface that is marshaled by the mono runtime. Future changes in the framework’s public ABI can be spotted very easily, as the C compiler will complain about missing functions or changes datatypes.

However, I chose the second way as usually prefer keeping one language in a project. Banshee uses quite some C gluelayers at the moment, bug there are efforts to remove them and someday have a completely C# based codebase.

Although MonoMac did not bind the IOKit/DiskArbitration framework, there is limited support for the CoreFoundation framework which helped doing a managed-only solution. I did never work on low-level OS X APIs before, neither did I work with MonoMac, so I spent nearly a full-time week digging into the Header files (which turned out to be the best documentation for the API) and finding out about MonoMac, and how I could handle the native CFObject types and marhsal them into C#.

If I had more knowledge about MonoMac and OS X internals, I’d have tried to contribute them to MonoMac by binding the missing frameworks. I am however glad I didn’t chose that in the first place, as it turned out I only needed a handful of funtions wrapped, and especially IOKit is a huge beast with far more complexity needed for my purposes. So right now, there are two small static IOKit and DiskArbitation classes that expose the needed library calls via the public static extern DllImport mechanism.

Hooking into disk arbitration wasn’t the difficult part. Turned out, for device recognition I needed the usb vendorId/productId pairs, and those were not provided by disk arbtitration (as its not bound to the USB bus). It took me days to figure out how to do that, and I had a chance to catch up with my rusty C abilities, as I did the prototype completely in C, and then rebuild it in C#.

Outcome

I managed to get USB device recognition working quite well (as far as I could test). In the latest alpha build (see below), you can plugin usb mass storage devices at will, before or during banshee runs, and we hook into all necessary events (device unmounting, hard usb removal because life is to short to remove usb safely) and banshee will refresh the devices list.

The usb vendor/product ID pairs are exposed, so lots of device abilities are detected automatically. My Samsung Galaxy S (with CM9) is correctly detected as an android device, and I tested a simple USB stick with an .is_audio_player file to pretend it being a MP3 player. There might be glitches with devices having more than one partition on them, I have yet to test that.

Syncing works (again as far as I could test) although I had to apply two non-hardware backend related patches (1,2) to make the process work seamlessly.

As I only have a limited number of devices, I invite everyone to test with your devices, and give me feedback so I can further improve the mass storage support.

Minor improvements

The alpha3 build I put together includes, besides the very large hardware-support patch (that is not yet in upstream as it will likely change in the next weeks), enabled  UPnP/DLNA client support . There was not much to do, just create a bockbuild package dependency and it worked out of the box, same way as it does on Linux.

As the folks from xamarin pointed out to me in IRC, there are known problems with Glib 2.32 on OS X which severely impact performance. The (quite old) gcc Apple ships seems to have problem with gcc interlocked/atomic intrinsic (compare-and-swap and alike). I downgraded to Glib 2.30 by reusing the xamarin package and to my great surprise I could actually feel it. The UI is a little more responsive, but the great change is the playback: While I often had sound stuttering when the system was under load, that is completely gone now. That change is also included in the latest alpha build.

New alpha build release

As I had done with the two progress reports before, I’ve bundled a new Banshee.app (this time alpha3) ready for everyone to download and test. Get it from here and do not forget to read the instructions provided. Feedback is greatly welcome, you could use the banshee mailing list for that.

GSoC progress report #1

GSoC coding phase has run for nearly two weeks now and I have quite some things to report on what I was working on during that time.

BCE build system

One core goal of my GSoC assignment is to enable the BCE build system to work an mac. This was actually very easy task to do, as I could just recycle the bootstrap-bundle and profile-configure scripts from banshee, with just some minor modifications. So for now, a ./bootstrap-bundle will prepare the build environment (mainly scan through your setup and set all necessary environment variables, like LD_LIBRARY_PATH and so on), and “./profile-configure darwin” will run the configure script with the suggested enable/disable flags.

BCE compatibility check

After the basic  build system setup was done, I was curious on what modules would compile out of the box – I thought about my previously contributed FolderSync which does not rely on any unmanaged code – and just tried to enable more and more extensions by adding an according “–enable-<extensionname>” to the profile-configure file. For a handful of extensions this worked seamlessly – but for some others I had to learn there were dependencies that had to be build first and thus I created package files suitable for bockbuild.

A very nasty bug in the gnome-doc-utils m4 macros wasted nearly a day of my time – which was mostly due that I am not that familiar with the whole autotools stack. I finally managed to fix it by replacing the m4 macros with the ones shipped with banshee.

OpenVP investigation

The OpenVP extension gave me a very hard time. OpenvP  creates stylish visualisation in banshee on linux, and it’ll be great to have it on OS X. After nearly 2 days of research, I came to the conclusion that its not easily portable, and here is why: The OpenVP extension relies on the nowadays abandoned/unmaintained  Tao.OpenGL stack. I then found that OpenTK offers a compatibility layer to that, so I created a bockbuild package and added OpenTK as a dependency. This allowed to build the OpenVP .dll and enable them in banshee. However, I quickly found that trying to start it would result in EntryPointNotFound exception: The GLWidget that is used by OpenVP tries to call native gdk_x11_drawable_get_xid() which is unavailable – as we use the gtk-quartz backend, not the x11 one. There is no similar function in the gtk-quartz backend, so I was screwed. After a while I found an alternate GLWidget implementation, which is also based on OpenTK and does not reference native code, but leaves that to the OpenTK backend. I could get everything to compile after some fixes and even be able to run, but upon activation banshee crashed completely from within OpenTK. After searching the OpenTK source, I found that the drawable context is setup using System.Windows.Forms constructs – and that didn’t work out on  OS X using gtk-quartz. So after 2 days I decided that OpenVP will not easily be portable, unless there is a more reliable GtkGLWidget that works with quartz. Bummer.

Webkit-based extensions

There are a bunch of extension that require Banshee.WebBrowser which is currently not available on the OS X plattform. I dug into what would be necessary to get Banshee.WebBrowser enabled, which would be very nice because that would allow Amazon’s WebStore to work. Banshee uses at the moment a glue library (libossifer) to embed WebKit within Gtk#. That native glue code links against WebKit directly. I had some failed attemps to get the WebKit source from svn/git (its just huge!),  and the tarball failed to build. I’ll have to investigate this further, especially since there is a GtkWebKit project which could replace libossifer and a banshee branch with some code to replace libossifer with webkit#. I will definitely get my hands on the WebKit-part in the next weeks, but found it too time consuming for the first weeks as I wanted to show some early results 🙂

BCE extensions running right now

After some dependency research, package creation & compilation, I have finally now running and working the following extensions:

  • FolderSync
  • AlarmClock
  • AlbumArtWriter
  • DuplicateSongDetector
  • RadioStationFetcher
  • LiveRadio
  • LastFMFingerprint (required native fftw and libsamplerate packages)
  • StreamRecorder (requires fixed dllmap .config, lame package plus some Makefile fixes)

Discarded extensions

Some extensions just don’t make sense on the OS X plattform, as they interface with software that is usually not present there:

  • Zeitgeist-data-provider
  • Awn integration
  • Telepathy integration
  • Appindicator

I’ve disabled these extension in the profile-configure file so they don’t get build when on OS X.

 The remaining extensions

As with the remaining extension, I do not know yet whether I should port them or there are some problems. I.e., I could get the RandomByLastFM plugin to compile but it throws some exception while running. As mentioned before, all Banshee.WebBrowser dependentand extensions require further investigation. With the clutterflow extension I am not yet sure whether its possible to integrate cutter into gtk-quartz.

Bockbuild changes

Besides some new packages added that were needed by the BCE, I’ve bumped some packages to newer versions, and merged David’s and Xamarin’s branch into my local branch at github. I try to stay aligned with the xamarin branch of bockbuild as close as possible, because most of the gtk-quartz/mono fixes they integrate directly apply to banshee.  However, due to a bug in pango’s CoreText implementation, we are still stuck on an older pango version.

I also rearranged some of the bockbuild util* stuff and added a python script that  allows to write the necessary environment variables directly into a MonoDevelop .csproj file in XML format using the “–csproj-instert=<filename.csproj>” parameter. Check the build howto for more information.

Banshee changes

I’ve also prepared a patch to banshee which switches to the new gtk_mac_* API and drops the old ige_mac_* API . Before I sent this patch upstream, I’ll check whether its possible to completely switch to the new GtkOSXIntegration API, which would use Cocoa instead of Carbon. The patch is included in my bockbuild repo and automatically applied when building.

Build instructions

I’ve created an extensive HOWTO on how to build current banshee 2.4 with the currently running extensions from BCE on the OS X plattform. So if you want to build banshee from source, or get your hands on the code, now is a good time for it 🙂

Unfortunately, there are still some glitches when creating an .app Bundle, so I can’t offer an early alpha .app right now – but stay tuned!

GSoC preparations – part 1

The official GSoC coding phase hast not started yet (will start on May 21th) but of course, I already did some research and preparations (as is also suggested by Google) in order to be ready when it does. In this post, I put together some thoughts and findings that occured to me while prep’ing. This is part 1, there will be a part 2 soon ^^

Enter bockbuild

First step when working on a project is setup a build environment. I did this already before my actual GSoC application, so I can get a better understanding  and estimate the effort necessary to integrate the BCE into banshee’s current build system. When building software that originates on Linux for Mac, you inherently face the problem that there is no package repository tool available as you would expect from a linux system (apt, zypper, etc.). You can’t just do a “zypper si –build-deps-only” to install the required dependencies  as I would do on my openSUSE box. And if you go for fetching and compiling every little dependency yourself, you will soon realize that it comes to more than 60 different software packages (for banshee), each in their own tarball, git repo or whatever, and each possibly dependent on another package(s). Eventually, you will be lost in compilation.

This is where bockbuild comes into play. Created by Aaron Bockover, bockbuild takes care of fetching and compiling the sources from scratch in the correct order and installs them all together into a single $prefix. This includes building build tools (like autotools and cmake), gtk and its dependencies (cairo, freetype, glib etc.), gstreamer, a full mono environment and finally the bunch of banshee dependencies and most of the C# wrappers to the aforementioned libraries.

On my development machine (a 2008 Macbook white with a Core2Duo and SSD) the whole process of fetching and building the environment takes about 2,5 hrs. Because one would not rebuild the entire world after each tiny package upgrade  (bumping a tarball to a newer version or moving to a newer git commit), one can simply recompile the modified packages. This is where it gets messy. While the recompilation of a single package may succeed on the pre-setup world, a completely new build of the world from scratch could fail. This is one of the reasons I spent almost two days before my application for GSoC to patch bockbuild for my system to successfully compile the world – just in order so I could compile a recent banshee version on my machine. The whole process is very painful and I wouldn’t recommend it to anybody – maybe a reason why to this date there are not that much banshee developers based on OS X. I decided to change that, or at least lower the bar for other developers to get their hands on banshee on the Mac.

Better testing of bockbuild

As the build results of bockbuild when building the world are dependent on dozens of factors, I needed a clean-room environment where I can test the builds and later on the .app bundle. Now that I am sponsored by Google and knowing that I  will (hopefully) soon receive my initial funding, I did not hesitate to spend some of Google’s money to back my development process: I acquired a copy of Parallels 7 (which comes as a full-version educational edition for 39,99 €) which allows me to run virtualized versions of Lion and – with a little hackery  – my old Snow Leopard DVD non-server version.

I am really no big fan of Parallels, they don’t seem to treat their customers well (pop-up ads for their own products as well as crapware bundles like Kaspersky subscriptions etc), but I’ll maybe blog on that later. For now,  Parallels 7 will give me a playground to accomplish my next goal:

Stable bockbuild releases

In order to allow other developers (and users!) to compile banshee from source and roll their own AppBundle, I’ll try to maintain two branches in my GitHub repo:

  • -stable branch, which is tested to build completely from scratch on different OS X installations with different settings. It is alligned with the banshee releases – for now I created a banshee-2.4-stable branch which could be used to build banshee 2.4 and roll your own AppBundles
  • a –testing The master branch which someday will eventually turn into banshee-2.6-stable branch. Patches against this branch might cause the build world to fail, as it is not tested everytime a change is made. Use at your own risk!

Now the branches are setup, but documentation on building the world and banshee on OS X are a little outdated – I will give more detailed instructions in a future blogpost, where I will also tell more about how to easily setup MonoDevelop to compile and run banshee for your development purposes.

Hello GNOME, Hello GSoC!

This is my first post in a series of posts about my Google Summer of Code project. I was accepted by the GNOME Project to spend this summer coding on the famous banshee media player. I’ll focus on the Mac OS X port of Banshee, which needs quite some love and polish to shine as much as it’s linux port.

But first, a brief little introduction of myself: My Name is Timo Dörr, I am currently enrolled as a student at the Karlsruhe Institute of Technology (fomer University of Karlsruhe) and  will soon finish my studies in the field of computer science. I’ve used linux since more than a decade as my one-and-only operating system until I bought myself a Macbook and played a little with Mac OS X. Since then, I would call myself a “dual-user”  hopping between both universes. I therefore am really excited about the GSoC assignment, which not only allows me to work with the banshee team and improve one of the best media players out there, but also enables me to get a closer insight into developing in C# with Mono on OS X – which I’ve already learned to love in the Linux world.

I’ll work on different parts to (hopefully) bring banshee’s OS X up on pace to its linux port, but to shortly sum up my proposal (which eventually got accepted by GNOME & Google), here is my TODO list:

  •  Integrate the Banshee Community Extensions (BCE)  into bockbuild (the OS X buildsystem currently used to build banshee on OS X) so OS X users can enjoy existing plugins, too
  • Select and propably adjust native code parts within the plugins to work with OS X
  • Work on simple hardware detection support of mobile devices, which is currently not working at all on OS X
  • Make lots of smaller improvements that make banshee feel more “native” on OS X, like usage of the media keys, native file dialogs and more.

I will regularly post about my work and the problems I faced, and am always open for tips and suggestions. Though I’ve made a reasonable schedule of what to complete when to successfully finish the assignmet, I think there is quite some room for further small tasks. So if you ever wanted to use banshee on OS X (or other platform) but found anything weird, quirky or missing feel free to email me: timo (at) latecrew (dot) de – I might end up working on it during or after the GSoC period 😉