mytharchive fails with python error

Bug #118700 reported by David Morris
4
Affects Status Importance Assigned to Milestone
MythTV
Invalid
Undecided
Unassigned
Mythbuntu
Won't Fix
Wishlist
Unassigned
mythplugins (Ubuntu)
Won't Fix
Wishlist
Unassigned

Bug Description

Binary package hint: mytharchive

Hi,

I'm using the 0.20svn20070122ubuntu3 package from ubuntu of mytharchive. In my setup I've got a dedicated backend and a dedicated frontend. Since my backend is under powered I wanted to run the archiving on the frontend.

The python error is here.

2007-02-14 18:13:03 mythburn.py (0.1.20060910-1) starting up... 2007-02-14 18:13:03 Obtaining MythTV settings from MySQL database for hostname pacifica 2007-02-14 18:13:04 Obtaining date and time settings from MySQL database for hostname pacifica 2007-02-14 18:13:04 Processing Mythburn job number 1. 2007-02-14 18:13:04 Options - mediatype = 0, doburn = 0, createiso = 1, erasedvdrw = 0 2007-02-14 18:13:04 savefilename = 2007-02-14 18:13:04 Looking for: /usr/share/mythtv/mytharchive/themes/Compact/theme.xml 2007-02-14 18:13:04 Loading font 0, /usr/share/mythtv/FreeSans.ttf size 23 2007-02-14 18:13:04 Loading font 1, /usr/share/mythtv/FreeSans.ttf size 18 2007-02-14 18:13:04 Loading font 2, /usr/share/mythtv/FreeSans.ttf size 16 2007-02-14 18:13:04 wantIntro: 1, wantMainMenu: 1, wantChapterMenu:0, wantDetailsPage: 0 2007-02-14 18:13:04 Final DVD Video format will be pal 2007-02-14 18:13:04 There are 1 files to process 2007-02-14 18:13:04 Pre-processing file '1035_20070213210000.mpg' of type 'recording' 2007-02-14 18:13:04 ------------------------------------------------------------ Traceback (most recent call last):

    File "/usr/share/mythtv/mytharchive/scripts/mythburn.py", line 3393, in <module>

        processJob(job)

    File "/usr/share/mythtv/mytharchive/scripts/mythburn.py", line 3169, in processJob

        preProcessFile(node,folder)

    File "/usr/share/mythtv/mytharchive/scripts/mythburn.py", line 1045, in preProcessFile

        mediafile = os.path.join(recordingpath, file.attributesfilename?.value)

    File "posixpath.py", line 62, in join

        elif path == or path.endswith('/'):

AttributeError?: 'NoneType?' object has no attribute 'endswith' 2007-02-14 18:13:04 ------------------------------------------------------------

I'm guessing that either

* mytharchive looks on the localhost for the mythtv database to grab the information about the recordings. However the database is actually on the backend. In which case it should really connect to which ever backend is set in the frontends config files to grab the info.

or

* the backend database hasn't got permission/a hostname entry for mytharchive to dip into the database

Changed in mythplugins:
assignee: nobody → ubuntu-mythtv
status: Unconfirmed → Confirmed
Revision history for this message
Dave Walker (davewalker) wrote :

I'm not satisfied that this is a database issue.

Initially I thought it was a permission problem with /usr/share/mythtv/mytharchive/scripts/tmp

However, i need to investigate this further with a vanilla machine.

Revision history for this message
David Morris (dave-greenacre) wrote :

ok, I solved the problem, I needed to nfs mount the recordings dir, which I did as /recordings then I needed to update the database

insert into settings (value, data, hostname) VALUES ('RecordFilePrefix', '/recordings', 'pacifica');

Please close this if its not a bug, but why did I manually need to edit the database?

Revision history for this message
David Morris (dave-greenacre) wrote :

Rejected upstream with the following comment

Currently MythArchive will only work properly on combined FE/BE machines.

Changed in mythtv:
status: Unconfirmed → Rejected
Revision history for this message
David Morris (dave-greenacre) wrote :

I feel this should be working on mythbuntu though and I suggest the work around listed above if we can automate it somehow

Revision history for this message
Bob K Mertz (online-bibleboy) wrote :

I had the same exact issue and the same resolution worked fine for me. I am using MythBuntu Alpha 4 and I have done apt-get upgrade on both machines just yesterday so it appears it hasn't been resolved.

I would really like to see this fixed in mythbuntu.... I can handle issuing those commands but I imagine many people that download mythbuntu won't be able to (or be too scared to).

Revision history for this message
Mario Limonciello (superm1) wrote :

Well the problem here is that this is dependent upon the assumption that you are mounting your recordings in the same place on all your backends and frontends. This is fine if you only have one backend and one frontend, but really makes for a troublesome situation otherwise.

Let me present two situations that automating this breaks entirely:

(1)
Machine A, master backend mounted at /var/lib/mythtv/recordings locally
Machine B, slave backend mounted at /var/lib/mythtv/recordings locally

Neither machine A or machine B clearly have each others recordings mounted.

Now you add Machine C, a frontend. Well so what do you do about mounting these recordings? You can clearly only mount one machine's recordings.

(2)
Machine X, master backend mounted at /var/lib/mythtv/recordings/X
Machine Y, slave backend mounted at /var/lib/mythtv/recordings/Y
Machine Z, frontend

Now on Machine Z, you can go and make the directories /var/lib/mythtv/recordings/X and /var/lib/mythtv/recordings/Y, and then mount them there no problem. The issue then is what do you put into the settings database? You can't just say /var/lib/mythtv/recordings, because it's expecting to find the recordings at wherever you define the RecordFilePrefix to be.

----
The only possible solution that I forsee would involve a patch upstream that would change the behavior of the RecordFilePrefix, as well as how these recordings are then accessed.

Revision history for this message
Bob K Mertz (online-bibleboy) wrote :

I didn't think about that angle. Finding a way to have this work would be a great thing since I think many people are starting to go the seperate machine route now that hardware is becomming more accessible. On the other hand, if it isn't possible to make it work, it would be very beneficial to have an error message to indicate that the front end and back end need to be on the same machine and give the options for making it work.

I can see it not working 100% streamlined "out of the box" but it would be nice to have scripts or options that could be changed from the default. Also, I think the most common configuration would be a single backend with multiple front ends - but that's just my opinion :)

Changed in mythbuntu:
status: New → Confirmed
laga (laga)
Changed in mythbuntu:
importance: Undecided → Wishlist
Changed in mythbuntu:
milestone: none → 8.04-alpha1
Changed in mythplugins:
status: Confirmed → Triaged
Changed in mythplugins:
importance: Undecided → Wishlist
Revision history for this message
Axel Pospischil (apos) wrote :

I know this is an older bug, but according to http://www.mythtv.org/wiki/index.php/MythArchive#MythArchive_reports_that_files_are_not_available_locally this problem is not longer an issue with mythtv > 0.21 which uses "Storage Groups":

[...]
MythTV usually streams video if you have a separate frontend and backend. MythArchive cannot use streamed video: it needs to be able to access the file locally. This can be accomplished by exporting the directory containing your recordings via NFS and then adding the mountpoint to the frontend's Storage Group.
[...]

Revision history for this message
Mario Limonciello (superm1) wrote : Re: [Bug 118700] Re: mytharchive fails with python error

Even if you are using storage groups, you still need to have some sort of
prefix set and as you said, use NFS. - So this particular problem persists.

On Tue, Jul 15, 2008 at 13:36, Axel Pospischil <email address hidden> wrote:

> I know this is an older bug, but according to
>
> http://www.mythtv.org/wiki/index.php/MythArchive#MythArchive_reports_that_files_are_not_available_locally
> this problem is not longer an issue with mythtv > 0.21 which uses
> "Storage Groups":
>
> [...]
> MythTV usually streams video if you have a separate frontend and backend.
> MythArchive cannot use streamed video: it needs to be able to access the
> file locally. This can be accomplished by exporting the directory containing
> your recordings via NFS and then adding the mountpoint to the frontend's
> Storage Group.
> [...]
>
> --
> mytharchive fails with python error
> https://bugs.launchpad.net/bugs/118700
> You received this bug notification because you are a member of
> Mythbuntu, which is subscribed to Mythbuntu.
>

--
Mario Limonciello
<email address hidden>

Revision history for this message
Axel Pospischil (apos) wrote :

You are right ;) But that does not leave me manually tweaking the database. It leaves me
 1. configuring NFS on either backend/frontend machine by hand and
 2. adding the exported directory as a Storage Group at the frontend machine
 3. taking care for a stable and performant network connection.

Due to the complexity of possible installation/mount scenarios this might be a thread for the "mythbuntu-control-centre" which is able/build to handle package management or other system configuration tasks - already containing nfs server under "system-services".

But that does not offer a unique procedure of (automatically) including different Storage Groups that are avaiable from other backends via NFS. However: the problem using different exported directories from different backends you (Mario) decribed above is now solved with "Storage Groups".

It might be a solution to add a mytharchive option "share (for backends) / use (for frontends) directories via fileserver" - and let the user choose between nfs/smbfs/cifs. The implementation of that task would be indeed not be a trivial one, but possible ;)

Automatically solve this task assumes that every user of mytharchive on a client with "only frontend installation" will need an configured file server on the backend. And - cross installation issues over system boarders will probably not be realistic ever. So this task will has to be done on the backend manually anyway - but could somehow be supported through mytharchive/control-centre.

Also not considered is the fact that some users might not know how network performance and stability issues can leave such a system nearly unusable (e.g. some might try a sloppy wireless connection).

To come to an end:
------------------------
For now there should be a warning message when using mytharchive on a "pure" frontend machine. Pointing to a help page how to configure a "networked" frontend/backend system the right way might be a good idea though.

Revision history for this message
laga (laga) wrote :

Axel Pospischil schrieb:
>
> Also not considered is the fact that some users might not know how
> network performance and stability issues can leave such a system nearly
> unusable (e.g. some might try a sloppy wireless connection).
>
>

I guess they'll find out quickly that it's not a good idea ;)

> To come to an end:
> ------------------------
> For now there should be a warning message when using mytharchive on a "pure" frontend machine. Pointing to a help page how to configure a "networked" frontend/backend system the right way might be a good idea though.
>

You can download recordings from the backend using an URL like
http://host:6544/Myth/GetRecording?ChanId=12345&StartTime=2008-07-15T00:00:00.
I plan on adding support for that to Mytharchive very soon - I just need
to get it working...

Revision history for this message
Axel Pospischil (apos) wrote :

> You can download recordings from the backend using an URL like
> http://host:6544/Myth/GetRecording?ChanId=12345&StartTime=2008-07-15T00:00:00.
This would be a good solution. However the best it would be for the frontend using the _ftp_ transfer protocol, because this really is the fastest way to get big files over the network without causing additionally traffic/loss/... . But you have to deal with what we have ;)

Just a few comments because I see you are going deeper into the problem: There are two possible scenarios:
 1. Download to mytharchive frontend -> Process** locally
 2. Process** on (fast) NFS mount

** mythreplex, ffmpeg, dvdauthor & Co. ...

(1.) If I understand this right, you first download the mpg file from the backend to e.g. the temporary mytharchive dir on the frontend and then process it? This would use all resources at best and avoid problems mentioned in (2.). Probably you could force mytharchive to use a "Storage Group" that points to this directory after copying. I don't know if mytharchive already distinguishes between local/distant file systems and can handle the existence of two identical recording files in different locations?

(2.) Due to the fact, that even on fast gigabit networks with fast backend servers (and this is not the fact here, since people like to use their faster frontend due to a weak backend server) the throughput would reach at best (!) approximately 20-30MByte/s on a mounted NFS directory. To avoid any flame wars here: I am talking about real life speeds, not theoretical speed. Compare that to possible disk speeds of around 80-100 MByte/s with direct disk access. Additionally using a NFS mount on the backend server that way will cause heave CPU usage on this machine! On the other hand this would lead into less disk usage on the frontend machine (however heavily using the PCI bus for the network connection).

So laga I wish you good inspirations getting this (1.) to work ;) ... Someday ;)

Revision history for this message
laga (laga) wrote :

Axel Pospischil wrote:
>> You can download recordings from the backend using an URL like
>> http://host:6544/Myth/GetRecording?ChanId=12345&StartTime=2008-07-15T00:00:00.
>>
> This would be a good solution. However the best it would be for the frontend using the _ftp_ transfer protocol, because this really is the fastest way to get big files over the network without causing additionally traffic/loss/... . But you have to deal with what we have ;)
>
When downloading from mythbackend, I get 11.7MByte/s over fast ethernet.
That should be good enough :) Besides, FTP is a horrible protocol.

> Just a few comments because I see you are going deeper into the problem: There are two possible scenarios:
> 1. Download to mytharchive frontend -> Process** locally
>

Yup, that's what happens. Mytharchive can already download files
automatically if it determines that they are on a remote file system, so
your other concern has already been addressed by the MythArchive developers.

>
> So laga I wish you good inspirations getting this (1.) to work ;) ...
> Someday ;)
>
>
Oh, it's working. ;) I just need to polish it a bit.

Do you want to test the patch?

Thanks for your input!

Revision history for this message
Axel Pospischil (apos) wrote :

> Do you want to test the patch?
Shure, If you will attach it here. I will have time to practise next weekend ;)

Revision history for this message
Axel Pospischil (apos) wrote :

Just to mention: no stress - take all the time you need ;)

If you are going to give the patch upstream sometimes or you know, where this will be discussed it would be kind to just post it here.

Greets Axel

Revision history for this message
Axel Pospischil (apos) wrote :

I have to add something to my post #10
(https://bugs.launchpad.net/mythbuntu/+bug/118700/comments/10)

> 2. adding the exported directory as a Storage Group at the frontend machine
This is only possible, if the frontend machine acts as a secondary backend (slave backend) due to the fact that "storage groups" are only manageable through the "mythtv-setup" program. The latter is only part of the mythtv-(master)-backend package.

However, if you are running "just frontend machines" it is not necessary to change the start as long as

2a. The storage group is a shared nfs/smbfs/cifs directory among all backend/frontend machines (use the same paths)
2b. "archive directory" is such a shared dir too (configuration->media->archive file settings)
2c. "temporary mytharchive directory" is set to a local directory (configuration->media->archive file settings)
2d. "Copy far files" is checked (configuration->media->archive file settings)

Then mytharchive - as laga wrote - copies the selected video files automagically ;) to the local mytharchive directory and processes them there.

Greets

Revision history for this message
Mario Limonciello (superm1) wrote :

Thank you for helping to improve Mythbuntu by opening this ticket. Unfortunately the Mythbuntu team does not foresee a time when they will be able to tackle this issue, so I am closing this ticket. With that being said, if someone else wants to write a patch for this, we will be happy to help push it upstream. Please do not let that action deter you from opening tickets on other issues that you find!

Changed in mythbuntu:
status: Confirmed → Won't Fix
Changed in mythplugins (Ubuntu):
status: Triaged → Won't Fix
Curtis Hovey (sinzui)
Changed in mythplugins (Ubuntu):
assignee: Registry Administrators (registry) → nobody
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.