Support for iPhone-like devices in Amarok 2

[Nico]

Binary package hint: amarok

Hello,

I have started the move from Intrepid to Jaunty on a couple of machine.

And I have discovered the available Amarok is now 2.0.2, the 1.4 branch is not there anymore.

This is annoying me because I have lost the ability to sync my iPod Touch (like an iPhone) with an advanced music manager.

Amarok 2 tries to be as automatic and as cross-platform as possible.
It relies on Solid for media player detection.
Under Linux Solid relies on HAL to detect media players
HAL only sees real hardware, like disks or USB mass storage devices.

IPhones or iPod Touches are used through FUSE mounts, using SSH (WiFi) or iFuse (USB cables).
FUSE mounts are not hardware mounts, really, so they are not reported back to HAL.
So Solid does not see the iPhone.
So Amarok does not see the iPhone.

I am trying to stir some ideas with the Amarok devs on the Amarok Mailing list. It looks like that whatever outcome rises out of this will not be in time for this Ubuntu release cycle (and I am worried that my bug is anyway coming way too late into the party anyway...).

GTKPOD relies on Gnome VFS to see mounts. Gnome VFS sees FUSE mounts. So Gnome VFS sees the iPhone. So GTKPOD sees the iPhone.
GTKPOD manages the iPhone properly using the database and libgpod.

But GTKPOD, however, is not really comparable to Amarok in terms of music management.

Rhythmbox does not see the iPhone.
Banshee does not see the iPhone.

So in Jaunty, GTKPOD is the sole working, if not excellent, solution for iPhone-like device management at this time.

As a stop-gap solution, would it be possible to bring back Amarok 1.4.10 as an alternative/parallel install in Jaunty?

[Jonathan Thomas (echidnaman)]

A fix has been made upstream in trunk. I'll see if I can't get a stable release update for Kubuntu 9.04 put together once Kubuntu 9.10 opens for development.

[Nico (nico-rdo)]

Hi Jonathan,

Thanks for this.

Is it possible to have more details on the fix?

[Nico (nico-rdo)]

ignore my previous comment, please. I just saw the upstream bug....

[Nico (nico-rdo)]

hmmm

I did a compile from SVN, including that fix.

And the problem remains the same.

The ipod-convenience package does the fuse/sshfs mount, but that mount is not declared/seen by HAL, so Amarok still does not check it, so no iPod Touch/iPhone is seen by Amarok.

Amarok logs with the latest SVN including the fix:
http://www.youplala.net/~will/amarok2/amarok_2.1_svn_20090424.log

Historical stuff:
http://www.youplala.net/~will/amarok2/

[Jonathan Thomas (echidnaman)]

Would it be possible for you to test the Amarok 2.1 beta packages from https://edge.launchpad.net/~kubuntu-experimental/+archive/ppa ?

[Nico (nico-rdo)]

OK, I have setup the ppa and upgraded the Amarok package.

It is complaining heavily about a few modules at startup (Librivox, Lyricwiki), probably not related to our case.

And the result is the same, negative.

The sshfs mount is still not declared in HAL, Amarok uses Solid to detect devices, Solid only uses HAL to detect devices, amarok sees no iPhone/iPod Touch.

So the situation with the 2.1 package from the PPA is the same as with my SVN checkout os 2.1 done this morning.

[Jonathan Thomas (echidnaman)]

Thanks, I've reopened the upstream bug and also posted a link to here.

[c.h. (wryfi)]

I mostly use arch these days, but I've learned a little about this problem recently, and thought I'd share (since the kde guys were kind of rude about it in their upstream bug tracker).

The recent commit to amarok should allow it to properly detect iphones/touches -- *if* the device is recognized by solid, which means it must also be recognized by hal. sshfs mounts don't show up in hal because no hardware is actually involved, and hal is for managing hardware. So, something in amarok or solid, or (less likely) hal needs to change for sshfs to become viable in amarok2.

An option which may be viable, which I'm experimenting with, is iFuse[1]. iFuse attempts to communicate with iphones/touches via native Apple protocols over USB. Since the connection is via USB, the device is recognized by hal and solid, at least in theory.

For me, on arch, theory breaks down rather quickly. The hal portion of iFuse is dependent on an fdi file included with iFuse, which doesn't exactly work (at least not on arch). I've been hacking on it, but I'm not quite there yet.

There is a deb package for debian experimental that might be a good place for ubuntu users to start. It might even work out of the box for you.

[Nico (nico-rdo)]

On Tue, 2009-04-28 at 02:26 +0000, chris wrote:
> I mostly use arch these days, but I've learned a little about this
> problem recently, and thought I'd share (since the kde guys were kind of
> rude about it in their upstream bug tracker).
>
> The recent commit to amarok should allow it to properly detect
> iphones/touches -- *if* the device is recognized by solid, which means
> it must also be recognized by hal. sshfs mounts don't show up in hal
> because no hardware is actually involved, and hal is for managing
> hardware. So, something in amarok or solid, or (less likely) hal needs
> to change for sshfs to become viable in amarok2.

This is exactly what I descibe in the bug.

>
> An option which may be viable, which I'm experimenting with, is
> iFuse[1]. iFuse attempts to communicate with iphones/touches via native
> Apple protocols over USB. Since the connection is via USB, the device
> is recognized by hal and solid, at least in theory.

Well, no.

iFuse is just like SSHFS. Both use FUSE, and fuse does not declare
volumes in HAL.

>
> For me, on arch, theory breaks down rather quickly. The hal portion of
> iFuse is dependent on an fdi file included with iFuse, which doesn't
> exactly work (at least not on arch). I've been hacking on it, but I'm
> not quite there yet.

This fdi file is for when HAL detects the USB device. It is no
declaration of the FUSE volume.

>
> There is a deb package for debian experimental that might be a good
> place for ubuntu users to start. It might even work out of the box for
> you.
>

I even tried compiling it from source, just in case I was wrong. I was
not.

Nico

[c.h. (wryfi)]

On 04/27/2009 10:54 PM, Nico wrote:
> This is exactly what I descibe in the bug.
>

The upstream bug was very limited in scope. Look at its title. It was
not intended to address the solid/hal issue, but to replace an internal
recognition function that had been removed in amarok2 ("Probably a good
first step is to get isIpod to return true for iPhone."). They fixed
the bug, as it was reported to them (getting the amarok function isIpod
to return true for iPhone), though they did not fix the underlying
problem you reported here (amarok2 doesn't recognize an sshfs mounted
iphone).

> iFuse is just like SSHFS. Both use FUSE, and fuse does not declare
> volumes in HAL.
>

ifuse is different because it is over a hardwired USB connection.

Note that ntfs-3g also uses fuse and its volumes are reported perfectly
by HAL. HAL recognition of fuse volumes was implemented circa 2006
(search the bug database at freedesktop.org).

The reason sshfs volumes are not detected by HAL has nothing to do with
fuse. It is because sshfs volumes are not hardware. HAL is the
HARDWARE abstraction layer. If there is no hardware involved, HAL is
not involved. You wouldn't expect an NFS or SMB volume to show up in
HAL, either. ;-)

Moreover, HAL most certainly *does* detect USB iphone connections. You
can prove this to yourself by running `lshal -m` and watching while you
plug in your iphone.

> This fdi file is for when HAL detects the USB device. It is no
> declaration of the FUSE volume.
>

You are correct that the fdi file is for when HAL detects the iphone
over USB; however, the fdi file most certainly does contain hal
declarations for the ifuse volume. The debian helper script (which was
adapted from fedora) even assigns major and minor numbers to the device
to facilitate this functionality. I've gotten it partially working, but
in my case hal is still slightly confused between the ifuse volume and
the ptp-class camera device. (The fdi file contains some <remove>
directives related to camera keys, but they don't seem to be working for
me. Or they're not applying to the right usb device. It's a little
hard to tell.)

The main problem I'm seeing is that the <match> directives in the ifuse
fdi file don't directly match the attributes reported from my iphone-3g
to the kernel to hal. The fdi file is out-of-date relative to some
combination of (1) my kernel (2.6.29.1), (2) my device (iphone 3g),
and/or (3) my hal version (0.5.12git20090406).

It's worth noting that the ifuse home page implies that iphone 3g
support is uncertain. I am able to mount my 3g via ifuse, however, so
hal support should be just a matter of tweaking the fdi directives to
properly match the hardware info the 3g is sending to hal via the kernel.

> I even tried compiling it from source, just in case I was wrong. I was
> not.

The problem isn't with compilation. The fdi directives are provided by
upstream or the package maintainer. Play more with tweaking the fdi
file and watching the results (after restarting hald) in the full output
of lshal.

[Nico (nico-rdo)]

On Tue, 2009-04-28 at 11:58 +0000, chris wrote:
> On 04/27/2009 10:54 PM, Nico wrote:
> > This is exactly what I descibe in the bug.
> >
>
> The upstream bug was very limited in scope. Look at its title. It was
> not intended to address the solid/hal issue, but to replace an internal
> recognition function that had been removed in amarok2 ("Probably a good
> first step is to get isIpod to return true for iPhone."). They fixed
> the bug, as it was reported to them (getting the amarok function isIpod
> to return true for iPhone), though they did not fix the underlying
> problem you reported here (amarok2 doesn't recognize an sshfs mounted
> iphone).

True.

I've been on the Amarok 2 ML.

The target they have at this time is to expose more media device to the
scripting API so that manual scans of mounted volumes can be made, not
using Solid, then iphones can be mounted using whatever mean is
available (iFuse, sshfs, etc...).

>
> > iFuse is just like SSHFS. Both use FUSE, and fuse does not declare
> > volumes in HAL.
> >
>
> ifuse is different because it is over a hardwired USB connection.

No it is not.

iFuse shows a network object multiplexed into USB as a disk.

iPhones do not show themselves as disks outside of the camera stuff. The
music stuff is through a network protocol.

So, once more, iFuse will not put anything in HAL.

>
> Note that ntfs-3g also uses fuse and its volumes are reported perfectly
> by HAL. HAL recognition of fuse volumes was implemented circa 2006
> (search the bug database at freedesktop.org).

Yes, sure, because it mounts real disks/partitions.

>
> The reason sshfs volumes are not detected by HAL has nothing to do with
> fuse. It is because sshfs volumes are not hardware.

It is the same for iFuse. iFuse deals with USBmux (network protocol) and
AFC (network service), not with disks.

> HAL is the
> HARDWARE abstraction layer. If there is no hardware involved, HAL is
> not involved. You wouldn't expect an NFS or SMB volume to show up in
> HAL, either. ;-)

So do not expect iFuse to do either.

>
> Moreover, HAL most certainly *does* detect USB iphone connections. You
> can prove this to yourself by running `lshal -m` and watching while you
> plug in your iphone.

Sure, it detects the only part that looks like a disk, and linux
supports it very well to give you acces to the pictures on the device.

>
>
> > This fdi file is for when HAL detects the USB device. It is no
> > declaration of the FUSE volume.

True, it detects the picture device and assumes that if that part is
there, then the rest will be too.

> >
>
>
> You are correct that the fdi file is for when HAL detects the iphone
> over USB

The pictures storing part of the iPhone.

> ; however, the fdi file most certainly does contain hal
> declarations for the ifuse volume. The debian helper script (which was
> adapted from fedora) even assigns major and minor numbers to the device
> to facilitate this functionality. I've gotten it partially working, but
> in my case hal is still slightly confused between the ifuse volume and
> the ptp-class camera device. (The fdi file contains some <remove>
> dir...

[Tiago Cogumbreiro (cogumbreiro)]

 Bug #368706 asks for the creation of a package for Amarok 1.4.

[Ookadoo (jbrown0)]

Just want to add that I too have experienced this when upgrading for 9.04 and as a result have had to switch to gtkpod and ifuse.

But I still have problems (slow, directory syncing doesn't work well) so I would love to see whatever needs it get support for these non-physical fuse/sshfs/etc filesystems.

[Andrew Ash (ash211)]

The discussion for not including Amarok 1.4 packages has occurred several times [1][2][3], so I'm changing this bug's direction towards better support for iPhones.

[1] https://bugs.edge.launchpad.net/ubuntu/+source/amarok/+bug/368706
[2] https://bugs.edge.launchpad.net/ubuntu/+source/amarok/+bug/331449
[3] https://bugs.edge.launchpad.net/ubuntu/+source/amarok/+bug/316889

[Andrew Ash (ash211)]

Nico, could you please comment on the current status of iPhones/iPods working in Amarok 2 with the latest version?

Please include your findings and the version of Amarok you're using (''apt-cache policy amarok''). Thanks!

[manuw2009 (manu-wagner)]

Hi,

Thanx to the HUGE efforts of the community, libgpod is now starting to handle
properly the iphone & ipod touch 3G without jailbreaking, through fuse.
gnome's rhythmbox is seeing the ipod, thanx to gvfs 1.5 enhancements (gvfs afc
backend).
I believe solid should enable to do the same thing for kde & amarok.
Has anyone a clue on how to do it (even through an amarok script) or who could
be of any help on the subject ?

[Zorael (zorael)]

GNOME apparently does a very nice job of this now. Even in KDE with gvfs packages removed, gtkpod reacts and opens up the iPhone automatically after mounting the device with ifuse. Solid only reacts when the device itself is connected - at which point Amarok correctly deduces that it's not a media device - but stays blissfully unaware when it gets mounted and the media device directory structure becomes available.

Either Solid, HAL/udev or Amarok needs to be supplemented with extra fuse-awareness. GNOME's choice was apparently to place their solution in gvfs instead of in HAL, so perhaps KDE should do the same and place theirs in Solid. Or in Amarok.

[manuw2009 (manu-wagner)]

Hi,

Any ideas on how to implement this through scripting/command line ?

Cheers.

[Mackenzie Morgan (maco.m)]

Re-targeting to kde4libs since the issue exists down in solid, not in Amarok itself.

[Jonathan Thomas (echidnaman)]

Solid uses HAL for detecting this, so it's not really solid's fault.

[Benjamin von Engelhardt (bve)]

How far are we here? I'm using ubuntu maverick which is supposed to support iphone. However, its not working for me. My Iphone makes the same noice as when the usb-cable is plugged in in the moment amarok starts, but I can't find it in amarok. amarok --debug gives me:

amarok: [MediaDeviceCache] Odd, got a deviceRemoved at udi "/org/freedesktop/Hal/devices/usb_device_5ac_1290_596371b2203ea228c9e0dfbb408e468ca3fe18dd" but it did not seem to exist in the first place...
amarok: BEGIN: void MountPointManager::deviceRemoved(const QString&)
amarok: END__: void MountPointManager::deviceRemoved(const QString&) - Took 6.7e-05s
amarok: BEGIN: void MediaDeviceMonitor::slotDeviceRemoved(const QString&)
amarok: END__: void MediaDeviceMonitor::slotDeviceRemoved(const QString&) - Took 5.9e-05s
amarok: END__: void MediaDeviceCache::slotRemoveSolidDevice(const QString&) - Took 0.00046s
amarok: BEGIN: virtual void ScanManager::startIncrementalScan(const QString&)
amarok: BEGIN: void ScanManager::checkTables(bool)
amarok: END__: void ScanManager::checkTables(bool) - Took 0.00014s
amarok: BEGIN: QStringList ScanManager::getDirsToScan()
amarok: END__: QStringList ScanManager::getDirsToScan() - Took 0.00088s
amarok: [ScanManager] GOING TO SCAN:
amarok: [ScanManager] "/home/bve/Musik/_AMAROKMTIME_1273053984"
amarok: BEGIN: void ScanManager::writeBatchIncrementalInfoFile()
amarok: END__: void ScanManager::writeBatchIncrementalInfoFile() - Took 0.00072s
amarok: BEGIN: XmlParseJob::XmlParseJob(ScanManager*, Collections::SqlCollection*)
amarok: END__: XmlParseJob::XmlParseJob(ScanManager*, Collections::SqlCollection*) - Took 0.0033s
amarok: BEGIN: virtual void XmlParseJob::run()
amarok: END__: virtual void ScanManager::startIncrementalScan(const QString&) - Took 0.016s
amarok: [ScanManager] Success. Committing result to database.
amarok: [ScanResultProcessor] Database temporary table setup did not complete due to no directories needing to be processed.
amarok: END__: virtual void XmlParseJob::run() - Took 0.036s
amarok: BEGIN: virtual XmlParseJob::~XmlParseJob()
amarok: BEGIN: void StatusBar::hideProgress()
amarok: END__: void StatusBar::hideProgress() - Took 0.00063s
amarok: END__: virtual XmlParseJob::~XmlParseJob() - Took 0.0014s
amarok: BEGIN: void ScanManager::slotFinished()
amarok: END__: void ScanManager::slotFinished() - Took 0.00023s
amarok: BEGIN: void MediaDeviceCache::slotAddSolidDevice(const QString&)
amarok: [MediaDeviceCache] Found new Solid device with udi = "/org/freedesktop/Hal/devices/usb_device_5ac_1290_596371b2203ea228c9e0dfbb408e468ca3fe18dd"
amarok: [MediaDeviceCache] Device name is = "iPhone" and was made by "Apple, Inc."
amarok: [MediaDeviceCache] udi "/org/freedesktop/Hal/devices/usb_device_5ac_1290_596371b2203ea228c9e0dfbb408e468ca3fe18dd" does not describe a portable media player or storage volume
amarok: END__: void MediaDeviceCache::slotAddSolidDevice(const QString&) - Took 0.069s
amarok: BEGIN: void MediaDeviceCache::slotAddSolidDevice(const QString&)
amarok: [MediaDeviceCache] Found new Solid device with udi = "/org/freedesktop/Hal/devices/...

[manuw2009 (manu-wagner)]

There must be a regression in kde 4.5.1 as it's broken again in maverick :
my itouch appears as an iPod with zero track !
I'm using maverick rc and amarok 2.3.2
it eventually worked in lucid using p mcenery ppa
:(

[Benjamin von Engelhardt (bve)]

ideviceinfo sees the iphone, but only until I start amarok - than my iphone somehow has disappeared (ideviceinfo now saying: No device found, is it plugged in?)

[manuw2009 (manu-wagner)]

Hi again,
As reported here https://bugs.launchpad.net/bugs/655908
there seems to be a regression in Maverick on the amarok+iphone front.
As I mentionned there https://bugs.kde.org/show_bug.cgi?id=245648, my guess is that it comes from a libmtp improvement preventing amarok from handling the iphone.

Workaround :
1/ Revert back to Lucid's packages libmtp8 & libmtp-dev
2/ Install libimobiledevice0 from Lucid

[Julien Lavergne (gilir)]

I think the bug is in libimobiledevice, since it doesn't install a HAL fdi file since Maverick. I re-added it in a testing package, can someone with a maverick system test with the version libimobiledevice_1.0.1-1ubuntu0.1~ppa1 in my PPA (https://launchpad.net/~gilir/+archive/unstable) ? Please note that the fdi is not really up-to-date, so if you have a recent Ipod/Iphone/Ipad etc ..., probably, it will not be detected even with this fix.

I can't find any information about a plan for Amarok to migrate to something else than HAL for hardware detection. I'm not an KDE user (only libimobildevice maintainer), so if you have information about this, I am interesting :)

[Launchpad Janitor (janitor)]

This bug was fixed in the package libimobiledevice - 1.1.0-3

---------------
libimobiledevice (1.1.0-3) experimental; urgency=low

  * debian/control:
   - Add Conlicts / Replaces on libimobiledevice1 and libimobiledevice0, since
     they provide the same fdi file. LP: #753015, Closes: #620065
   - Update Maintainer field, and add me to Uploaders.
   - Update Vcs-*.
   - Remove duplicate section.
  * debian/gbp.conf:
   - Add specific gdb.conf.

libimobiledevice (1.1.0-2) experimental; urgency=low

  * debian/control:
    - Add Conflicts/Replaces for libimobiledevice2-dbg with previous version.
  * debian/31-apple-mobile-device.fdi
   - Re-add HAL fdi file, until Amarok don't rely on it.
     Closes: #615107, LP: #362182
  * debian/libimobiledevice2.install:
   - Install the HAL fdi file.
  * debian/patches:
   - 90_fix_4.3_iOS.patch: From upstream, fix corrupted bytes in iOS 4.3.
 -- Julien Lavergne <email address hidden> Tue, 19 Apr 2011 10:05:05 +0000