Adding scanning (and ScanTo) support for HP LaserJet M2727nf MFP

Bug #256541 reported by Kyle Moffett
20
This bug affects 2 people
Affects Status Importance Assigned to Milestone
HPLIP
Fix Released
High
Unassigned

Bug Description

My parents recently went out to the store and bought themselves a network-connected HP LaserJet M2727nf MFP multifunction device. They carted it home, all proud of themselves, and promptly called me over to set it up on their windows desktops. Naturally I set them up to print to it via Samba+CUPS instead of trying to install the printer drivers on each system, but when it came to scanning support I ran into a wee problem... there was none.

Being an enterprising Linux geek, I decided I'd reverse-engineer or improve the drivers to the point where it would work. After lots of trial and error and more than a few tears, I was able to get minimal "scanimage" functionality (although the page boundaries are occasionally a bit off) by editing the models.dat file for [hp_laserjet_m2727nf_mfp]:

* changed "io-mfp-mode" from "1" to "6"
* changed "scan-type" from "-1" to "5"
* changed "status-type" from "6" to "9"

All three changes seem to be required for it to work. The top of the generated /usr/share/hplip/data/models/models.dat file mentions a "model_editor_dat.py" script that I should edit, but I can't seem to find it to change it.

After tinkering a bit and using wireshark from a linux box between my parents' Windows desktop and the M2727nf MFP, I determined that the hplip driver also doesn't seem to be willing to turn on the auto-contrast adjustment, and as a result the images are a bit bright and low-contrast.

Once I had minimal functionality working, I tried to get the "Scan To" button on the front of the system to work. With more wireshark-ing and some help from this URL:

  http://rende.se/index.php?n=Main.ScanToFolder

I was able to reverse-engineer the HTTP+XML-based ScanTo protocol for my printer. I ended up going quite a bit further than the guy at that URL in that I wrote a fully-featured Perl daemon to monitor and control the scanner. Said Perl daemon (hpscantod) is attached to this bugreport as a plain text file.

Unfortunately, while the "scanimage" command(s) in the script work just fine when the scanner is idle, they don't work at all when it's waiting for a computer connection in ScanTo mode. In that case I get an IO error during communication with the scanner:

[hpaio] scan/sane/soapht.c 1986: unable to start dime document:
hp:/net/HP_LaserJet_M2727nf_MFP?ip=10.0.0.11
scanimage: sane_start: Error during device I/O

Comparing the traffic dumps from the windows client with what SANE is sending, I seem to be missing the value for the <ScanIdentifier/> field in the <wscn:CreateScanJobRequest/> SOAP message and the value for the <JobToken/> field in the <wscn:RetrieveImageRequest/> and <wscn:CancelJobRequest> SOAP messages. The former looks like it should be the same as the ScanIdentifier for that particular ScanTo option (IE: The text displayed on the LCD screen), and the latter seems to be returned from the SOAP CreateScanJobRequest call in the JobToken field of the reply.

Since I couldn't get my head around the soapht.c option-handling code, I did the next best thing and added a horribly ugly dirty hack by way of (getenv("SANE_SCANID")?getenv("SANE_SCANID"):""). I added "%s" to a couple places in the XML #defines and inserted said dirty hack in the sprintf() argument lists. After a quick rebuild I was able to successfully initiate scans from the LCD+buttons on the front of the scanner.

The "hpscantod" daemon I wrote still needs some proper support for a config file (currently all the configuration is directly in the script itself), but other than that it seems to coexist peaceably with windows clients doing SendTo on the same scanner.

I look forward to your replies and comments. If I am unresponsive for some reason, please send me an email at <email address hidden>; this is my first bug reported through this tracking system and I'm not entirely familiar with how it works yet.

Cheers,
Kyle Moffett

Revision history for this message
Kyle Moffett (kyle-moffetthome) wrote :
Revision history for this message
Kyle Moffett (kyle-moffetthome) wrote :

I found and fixed several miscellaneous bugs in the ScanTo daemon, attached is the updated version.

Cheers,
Kyle Moffett

Revision history for this message
dwelch91 (dwelch91) wrote : Re: [Bug 256541] Re: Adding scanning (and ScanTo) support for HP LaserJet M2727nf MFP

Thanks for your submission, Kyle. We will take a look at your code and
determine if we can incorporate it in a future release. We would most likely
port it to C or Python, as Perl is not one of the languages we use in HPLIP,
but it looks like you have done the hard part of figuring out how to make it
work. It looks like you are monitoring the scan button every 5 seconds?
Does that seem like an acceptable rate? (Responsive enough to the user, but
doesn't slow down the PC?)

Thanks, Don

On Sun, Aug 10, 2008 at 8:22 AM, Kyle Moffett <email address hidden> wrote:

> I found and fixed several miscellaneous bugs in the ScanTo daemon,
> attached is the updated version.
>
> Cheers,
> Kyle Moffett
>
> ** Attachment added: "Updated HP ScanTo perl daemon"
> http://launchpadlibrarian.net/16690850/hpscantod.txt
>
> --
> Adding scanning (and ScanTo) support for HP LaserJet M2727nf MFP
> https://bugs.launchpad.net/bugs/256541
> You received this bug notification because you are a member of HP Linux
> Imaging and Printing, which is subscribed to HPLIP.
>

Revision history for this message
Kyle Moffett (kyle-moffetthome) wrote :
Download full text (3.7 KiB)

On Sun, Aug 10, 2008 at 12:17 PM, dwelch91 <email address hidden> wrote:
> Thanks for your submission, Kyle. We will take a look at your code and
> determine if we can incorporate it in a future release. We would most likely
> port it to C or Python, as Perl is not one of the languages we use in HPLIP,

Heh, I understand entirely. I tend to use C for a lot of things, but
when trying to hack something together quick using
HTTP+XML+random-text-goop-of-the-week I resort to Perl for sheer
prototyping speed. Just as reference, I did notice one misfeature
with the last version of the Perl script, I'd turned off tempfile
cleanup for debugging and forgot to turn it back on again; to fix just
change the "CLEANUP => 0" bit on line 412 or so to be "CLEANUP => 1".

> but it looks like you have done the hard part of figuring out how to make it
> work. It looks like you are monitoring the scan button every 5 seconds?
> Does that seem like an acceptable rate? (Responsive enough to the user, but
> doesn't slow down the PC?)

5 seconds seems to be roughly what the HP windows-only scanner
software uses and that seems about right in my testing. Any slower
and it starts to feel laggy and if you don't poll at least every 30
seconds or so then the scanner will sometimes time out waiting for a
connection.

I'm sure that there is some measurable load on the system from the
frequent polling, but that seems entirely due to the crappy
notification protocol designed into the printer. I certainly wouldn't
want to install such software on my laptop, it would be yet another
factor in killing the battery life; sitting there waking up the CPU
and polling the network every 5 seconds or so. Good thing I don't
much care about repetitive tasks running on their samba server :-D.

I did notice one other problem with the script the way it stands;
duplex scanning doesn't seem to work although I thought it did before.
 The scanner scans the front of the first page (from the ADF), then
flips it over while scanning the back. Then when it goes to flip it
over the second time to drop it in the out-bin, it doesn't quite clear
the last inch of the page from some part of the machinery before
reversing to re-feed the page. The result is that it tries to
back-feed the page the wrong way across the glass in front of the scan
head and gets all jammed up.

I double-checked and this *doesn't* happen while doing a duplex copy
from the keypad on the front of the machine, so it's not a hardware
problem. I seem to recall having successfully done a duplex scan
using "scanimage" before, so perhaps the corrected paper size settings
or something are confusing the microcontroller into screwing up the
duplex mechanism. When I get a chance I'll try to munge the options
to see if I can get it to work again.

The only other problem (I don't remember if I mentioned this before)
is that the contrast seems very low; the images are washed out and
faded towards white. I tried patching the source to send
<AutoExposure>true</AutoExposure> in the <wscn:CreateScanJobRequest/>
SOAP call the same way the Windows driver does, but that didn't seem
to have any effect. Neither does manually specifying "--contrast ...

Read more...

Revision history for this message
Matthew Marshall (matthew-matthewmarshall) wrote :

I tried editing models.dat as described, but I'm getting this error:

scanimage: open of device hpaio:/net/HP_LaserJet_M2727nf_MFP?ip=192.168.1.111 failed: Error during device I/O

I captured the network traffic, and it ends with the printer responding with "noSuchName"

Any clue at to what's going on? I've tried using both 2.8.2 and 2.8.7.

(I've attached a plain-text dump of the network traffic.)

Revision history for this message
Kyle Moffett (kyle-moffetthome) wrote :

On Fri, Aug 15, 2008 at 4:11 PM, Matthew Marshall <
<email address hidden>> wrote:

> I tried editing models.dat as described, but I'm getting this error:
>
> scanimage: open of device
> hpaio:/net/HP_LaserJet_M2727nf_MFP?ip=192.168.1.111 failed: Error during
> device I/O
>
> I captured the network traffic, and it ends with the printer responding
> with "noSuchName"
>
> Any clue at to what's going on? I've tried using both 2.8.2 and 2.8.7.

I determined that for the Windows installer and management tools to work
properly, the scanner needs to have SNMP enabled for full read/write access
from the community string "public". I left it that way when setting up the
Linux tools as well. It's a really stupid security design, but that's the
way they decided to implement it. You'd think that if they're going to
require SNMP to query/operate the scanner they'd at least use SNMPv3 or
something to provide a minimal semblance of security. If that doesn't help,
you might try resetting the scanner/printer to default settings and trying
again, that helped me tremendously.

Cheers,
Kyle Moffett

Revision history for this message
Shane R. Spencer (whardier) wrote :

The daemon just dies on me.. I'm using the following:

Product Name: HP LaserJet M2727nf MFP
...
Firmware datecode: 20080612

I finally got my tmpdir sorted out.. perl modules installed.. finally getting the error:

scanimage: open of device hpaio:/net/HP_LaserJet_M2727nf_MFP?ip=10.0.0.39 failed: Error during device I/O

Yay!.. It took quite a bit to get that far.. is there a resolution? I used ADF 300 PDF

Revision history for this message
Shane R. Spencer (whardier) wrote :

Ohyeh.. Forgot to add this (Hardy 8.04 + backports)

ii hplip 2.8.2-0ubuntu8 HP Linux Printing and Imaging System (HPLIP)
ii hplip-data 2.8.2-0ubuntu8 HP Linux Printing and Imaging - data files
ii libsane 1.0.19-1ubuntu3 API library for scanners
ii sane-utils 1.0.19-1ubuntu3 API library for scanners -- utilities
ii xsane 0.995-1ubuntu1 featureful graphical frontend for SANE (Scan
ii xsane-common 0.995-1ubuntu1 featureful graphical frontend for SANE (Scan

Revision history for this message
Kyle Moffett (kyle-moffetthome) wrote :

On Tue, Oct 21, 2008 at 3:07 PM, whardier <email address hidden> wrote:
> The daemon just dies on me.. I'm using the following:
>
> Product Name: HP LaserJet M2727nf MFP
> ...
> Firmware datecode: 20080612
>
> I finally got my tmpdir sorted out.. perl modules installed.. finally
> getting the error:
>
> scanimage: open of device
> hpaio:/net/HP_LaserJet_M2727nf_MFP?ip=10.0.0.39 failed: Error during
> device I/O
>
> Yay!.. It took quite a bit to get that far.. is there a resolution? I
> used ADF 300 PDF

Unfortunately it seems that the partial support for this scanner
(first showed up in 2.8.6) was "not intended to be released" according
to HP:
  http://sourceforge.net/forum/forum.php?forum_id=849275

I had originally tried this using hpijs 2.8.6 (where it worked), and a
little while afterwards HP released 2.8.6b which removed all the beta
code I had managed to get working. To be honest it's pretty un-cool
for HP to actively remove support for devices that were now possible
to get working, even if the support for said devices is pretty flawed.
 The open-source motto *is* "Release Early, Release Often" after all.
I had worked around the flaws well enough for it to be significantly
useful and was keeping a bug and patch list to forward upstream. Now
I'm intentionally staying on 2.6.8 to keep the scanner working until
HP get around to re-releasing that code. Since they did publish it
under GPL there's nothing to keep you from installing the original
2.8.6, as long as you accept that there is no warranty and it may kill
your dog, steal your firstborn, or nuke your disk drives.

Good luck!

Cheers,
Kyle Moffett

Revision history for this message
SebastienBarthelemy (barthelemy) wrote :

Hello,

> Unfortunately it seems that the partial support for this scanner
> (first showed up in 2.8.6) was "not intended to be released" according
> to HP:
> http://sourceforge.net/forum/forum.php?forum_id=849275

Anybody has information about the future plans for this printer scan support ? The only information I found is from Aaron Albright (https://answers.launchpad.net/hplip/+question/41817):
   "The problem is not a technical problem. We are working on provide support in the "near" future however.."

Is it still worked on ? Any information about how much "near" will last ? (btw, I fully understand that this is not an easy answer to give)

cheers

Revision history for this message
Mark Crawford (mark-crawford-deactivatedaccount) wrote :

The release of HPLIP ver. 2.8.10 is imminent and contains scanning support for this and a few other MFP devices that have been lacking it. It should be posted by the end of the day (U.S. time). Please monitor http://hplipopensource.com for the release of HPLIP 2.8.10.

Revision history for this message
Mark Crawford (mark-crawford-deactivatedaccount) wrote :

Quick update...HPLIP 2.8.10 has been released. However, we have had some website-related issues that will not be resolved until next week. So, you can go directly to SourceForge.net to download this latest version of HPLIP which has support for scanning on the following HP printers that have been inquired about of late:

 - HP LaserJet m1522 MFP (Postscript) (REQURIES plug-in for scanning)
 - HP LaserJet m1522n MFP (Postscript) (REQURIES plug-in for scanning)
 - HP LaserJet m1522nfs MFP (LJMono, Postscript) (REQURIES plug-in for scanning)
 - HP LaserJet m2727 MFP (LJMono, Postscript) (REQURIES plug-in for scanning)
 - HP LaserJet m2727nf MFP (Postscript) (REQURIES plug-in for scanning)
 - HP LaserJet m2727nfs MFP (Postscript) (REQURIES plug-in for scanning)
 - HP Color LaserJet cm1015 MFP (Postscript) (REQURIES plug-in for scanning)
 - HP Color LaserJet cm1017 MFP (Postscript) (REQURIES plug-in for scanning)
 - HP Color LaserJet cm1312 MFP (LJColor, Postscript) (REQURIES plug-in for scanning)
 - HP Color LaserJet cm1312nfi MFP (LJColor, Postscript) (REQURIES plug-in for scanning)
 - HP Color LaserJet cm1312 MFP (LJColor, Postscript) (REQURIES plug-in for scanning)
 - HP Color LaserJet cm1312nfi MFP (LJColor, Postscript) (REQURIES plug-in for scanning)

Revision history for this message
Aaron Albright (albrigha-deactivatedaccount) wrote :

Done.

Changed in hplip:
assignee: nobody → kalosaurusrex
importance: Undecided → High
status: New → Fix Released
Revision history for this message
Andrew Ruthven (andrew-etc) wrote :

Matthew,

I suspect your problem was not having a patched hplip...

Kyle,

Do you have the patch for hplip available? I started writing a Perl daemon to drive the scanner on my M1522nf, but using the hp-scan stuff would probably be better...

Cheers!

Revision history for this message
Aleksandr (amurchick) wrote :

Problem is still active!
See bug#1049502

Revision history for this message
mCoRN (metall-corn) wrote :

Hello.
I try to modify script from http://rende.se/index.php?n=Main.ScanToFolder
to work with HP2727MFP

Scan on button pressed - SOLVED. Only for one destination.
Set 2 or more destination - NOT WORKING

Comments on russian, sory.
http://ubuntuone.com/2kgVwL7Iv1SeUT2fv0898C

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.