Wednesday, December 1, 2010

Ruby on Rails and inconsistent database results

I started stressing a RoR project that has grown pretty big. Some serious hands on testing was showing that functionality was working well, performance was fine, but sometimes I would just get weird results from the database or ActiveRecord. I would create a new AR object, save it, use it a few times, update it, then it would suddenly just disappear. I would start getting ActiveRecord::RecordNotFound exceptions doing a Thingy.find(1234), when thingy#1234 definitely existed in the database. It would take a restart of Phusion Passenger or for one of the workers to timeout before I would start seeing the object again, and if I refreshed a page with Thingy.all(:conditions=>c) the results would change, then change back. I'm using MySQL so its not exactly what I was expecting to see.

I had issues in the past with some forking of processes that could just run through to conclusion in the background - they were removed. I made sure that there were good Thingy.transaction do end blocks covering my updates. Still, things were getting worse, not better.

Eventually I ended up hunting around the code from the dim and distant past. That stuff I don't touch because it "just works". Well, I roll up to an interesting section in a class :


 sql = ActiveRecord::Base.connection();
      sql.execute "SET autocommit=0";
      sql.begin_db_transaction
      sql.delete 'delete from a_table where some_conditions' 
      sql.update sqlstring
      sql.commit_db_transaction


This was valid, as the SQL going on in the sqlstring was complex to say the least. But since I've removed this from the main flow of the application things seem to have settled down considerably.

I'm guessing that my standard transactions were getting caught up in my attempt to borrow a connection from the pool explicitly and who knows what was happening. Or maybe Passenger was losing its connection and recreating a connection. I don't know, but I'm not doing it again!

Thursday, November 11, 2010

Ajax request keeps Google Chrome pointer spinning

I have a wonderful instant messaging app running using AJAX to maintain a long term connection to a simple HTTP server. How I did that is a discussion for another time, but the problem was that I needed to create the connection when the user first hit a web page. So I just simply put a Javascript call in body onload, something like:

<body onload="start_listener();">

Plain and simple, it worked. But on Google Chrome, the page continued to say 'Waiting for servername' and the mouse pointer was continuously spinning. I suffered this for a while, until I realized that the same call to start_listener() didn't exhibit the same endless spinning if I fired it from clicking a link.

I must be the only person on the planet with this problem, as Google yielded no results. I tried moving the call to a script at the end of the page, putting it into a script as window.onload, all to no avail. I double checked the request really was asynchronous. Chrome worked fine, but the cursor was disconcerting to say the least.

So I decided to search similar projects: node.js has nice event driven messaging demo, frequented by many observers and a handful of weirdos. They don't have the spinner problem, as they have you click on a link to get started. But something in there made me thing that I should attack this from another direction. How about I triggered an event that called the Javascript function, just as if I had clicked a link with the mouse?

Well, brainwave finally kicked in. Use a timer on the page, to start the AJAX request soon after the page loads. The Javascript must be event based, right?

So I converted my onload to:

<body onload="setTimeout('start_listener()', 1000);">


effectively triggering the call to AJAX as an event. And at last, Chrome stops spinning.

What's the deal? I guess that if you put an AJAX call directly into the main page load, Chrome treats it as an extension of the original request (maybe more through the design of the page request handling than on purpose?). But pull it out so that the AJAX request falls into a distinct event, and all works well. Timeout lets you do that without needing a user to click anything.

Finally I can look at my application without getting freaked out that it is failing to load something...

Wednesday, October 27, 2010

Counting pages

I've been working on a Ruby on Rails project for a while. One area of it has morphed into a bit of document management, and for some users it is important to know how many pages a specific document has in it. At least for PDFs and TIFFs.

Well, ImageMagick is one approach, letting you load the document then review its properties. But as anybody who has used it will know, unless you are careful, this can be a huge memory sink. In fact I use ImageMagick 'convert' as a way to force my machine to run out memory during testing, to see if it fails gracefully.

So, I hunted around a bit and came up with these programs: tiffdump and pdfinfo. I also considered tiffinfo, although the 'rawness' of tiffdump just seemed more appealing when parsing out the data I needed.

To install them (on Ubuntu):

sudo apt-get install libtiff-tools poppler-utils

Then use the command line programs from Ruby, something like this:


path = '/home/someone/somewhere/somefile.xxx'
mime_type = WEBrick::HTTPUtils.mime_type(path, WEBrick::HTTPUtils::DefaultMimeTypes)
if mime_type=='image/tiff'      
  return `tiffdump '#{path}' | grep 'Directory'`.count('\n')
elsif mime_type=='application/pdf'      
  return `pdfinfo '#{path}' | grep 'Pages'`.split(':')[1].chomp.to_i
else
  # whatever
end


Not pretty, not clever, but a lot faster than RMagick, and a lot easier than the Ghostscript approaches I've seen discussed but never actually working.

Tuesday, October 12, 2010

Chroot - ooh now I can run OpenOffice

I've been struggling with OpenOffice crashes since I've been running Ubuntu 10.04 (Lucid). I've tried everything. I've added horrible red-herrings to one of the many seemingly relevant bug reports on Launchpad. And in the process, I've tried debugging (debug symbols seem to be inadequate) and then I saw a discussion about recreating a bug from a previous Ubuntu version in a chroot based basic installation. So I followed the instructions for creating a chroot with a basic Ubuntu installation, installed a few basic packages (nano for example), set up the en_US UTF-8 locale, following Andrew Beacock's blog (necessary to install Java). I also had to add some archives for apt to pick up OpenOffice. Now I have Lucid running chroot'd inside Lucid.

I don't know chroot well and heard some issues around mounting disks, and I'm using ext4 with ecryptfs encryption for my home which kinda gets in the way, so I went the roundabout route and mounted ssh using sshfs (yes I had to install both of these first).

Finally I installed openoffice.org-ubuntu and openoffice.org-human-style. And I finally ran ooffice and edited a document all day long with no crashes. I don't know if that's it, allowing me to avoid some strange library conflict, or whether tomorrow is another day and another crash. But currently I like the chroot method for testing a clean install without making a whole clean install, or making it difficult to get at my documents, which I find a VM image tends to. And it took me about half an hour total time to get it to work, without chewing up half my disk or half my memory. I like chroot for this. Hopefully it will keep me productive for a while.

Tuesday, July 27, 2010

A lot going on in Ubuntu-land

I just ran the Ubuntu update for the day. A new release of the kernel and another new version of Firefox.

Frankly, I've moved to Google Chrome, since Ubuntu seemed to be getting slower and slower. I've not had that complaint with Chrome, though I did add the FlashBlock extension which prevents a lot of unnecessary advertising from chewing up CPU in the background.

As for the Kernel, there are some fixes to the ext4 file system I committed to when I installed Lucid, and the ecryptfs encryption module that I have decided might be a safe bet for encrypting my personal, business and backup data. The challenge now is to see if I can recompile the ecryptfs module on the Rackspace cloud server I've been using. I did it once, now let's see if I can do it again.

Oh and if you've been reading this block in the past, you'll see that I've been struggling with OpenOffice crashing. It still does, regularly if a document is showing a graphic on a page when I swap between applications. Not nice, as some of the documents I've been doing are pretty graphic intensive. I will continue to try and resolve this...

Tuesday, July 6, 2010

Canon PIXMA MX700 - print and scanning with Ubuntu

I own the nice-enough little all-in-one printer, scanner, fax unit, the Canon MX700. It was cheap, and I desperately needed a scanner at short notice. It worked nicely with my wife's WinXP laptop, but I've always struggled with it on Linux. An upgrade to Ubuntu 10.04 (Lucid) left me without a working printer again.

So here I go: when you set up your new printer, you won't find the MX700 in the list. You won't find an MX anything in fact. So pick the Canon PIXMA MP520. This apparently was released the same time and probably apart from network printing options or something benign, offers the same printing firmware. At least with USB, this works nicely for me.

As for scanning, I previously had to mess around to get SANE to understand the device. But then I ran across this: the SANE backend that is the standard scanning support for Linux claims to support the MX700: http://www.sane-project.org/sane-mfgs.html#Z-CANON


PIXMA MX700USB Ethernet0x04a9/0x1729CompleteFlatbed and ADF scan. All resolutions supported (up to 2400DPI)pixma
(0.16.1)
sane-pixma


Well, I just tried it with my favorite app, gscan2pdf and it just worked.

Finally, this stuff seems to be coming together nicely. Great work SANE team...

Wednesday, June 30, 2010

Dimdim - the free WebEx - and sharing your desktop on Linux

Dimdim is a fairly functional online meeting tool, similar to WebEx. The two advantages over WebEx (and GoToMeeting) are that Dimdim provides a free option, and it works with Linux.

Very recently Dimdim pushed out an Alpha version of a plugin for sharing your Linux desktop, and starting a meeting directly from an icon on your desktop panel. I downloaded the plugin, extracted the files, and ran the installer in a terminal session (~/Downloads/WebMeeting). This did most of what was required, except when I tried to run the WebMeeting plugin (in the Gnome Internet menu) it didn't work. Running the program from the terminal session told me that libssl.so.6 did not exist. So I simply created a couple of links:

cd /usr/lib
sudo ln -s libcrypto.so.0.9.8 libcrypto.so.6
sudo ln -s libssl.so.0.9.8 libssl.so.6

Hey, it works! Thanks Dimdim for providing a usable meeting tool on Linux!

Friday, June 25, 2010

VirtualBox not saving the VM state

I thought I was going crazy. I would close a guest virtual machine in VirtualBox, saving its state so that I could come back to it later. The next day I'd come back and the guest would reboot from a powered down state. The suspended state was gone, with an unknown amount of data lost in the session (maybe none, but I don't know - a couple of times I got weird errors when Windows came back up).

Anyway, it wasn't me. It seems that there is an error in the current version of VirtualBox (I'm running 3.2.4) that loses the state. I've been following the discussion around the fix for this on VirtualBox.org:
http://www.virtualbox.org/ticket/5656#comment:25

Anyway, it seems that the issue has been identified and we just need to wait for version 3.2.6 to arrive. Which is all fine and dandy, but 3.2.5 is not available for download yet, so its going to be a little time for us mere mortals that aren't in a position to run beta versions of the software.

[Edit: July 6th, 2010]

Well, version 3.2.6 arrived quickly it seems. I tried it out last week, and my sessions are being saved ok. Good work vbox team.

Wednesday, June 23, 2010

apache2: Could not reliably determine the server's fully qualified domain name

I've been getting this warning on starting up Apache for a while. I have the domain name in the site configuration (under /etc/apache2/sites-enabled), with the line:



ServerName consected.net

appearing right after the line, but the error still appears. 

Its easy to fix though. Just 

nano /etc/apache2/httpd.conf 

Then add the following line to the file (my file was empty to start with, so don't panic)

ServerName consected.net

Now restart the apache server (service apache2 restart) and the error has gone. Cool!


Monday, June 21, 2010

Big update to Ubuntu 10.04 Lucid - will it fix stuff?

I ran the Update Manager this morning to see what would appear. There was quite a list. Evolution featured pretty highly in the bugs fixed. But more importantly in fact were a range of fixes to Xorg related libraries that may help Ubuntu and Debian users who have suffered recently from crashes to their desktop environment with certain machines and some new desktop appearances. Oh, and for PDFs, if you've ever struggled selecting text that spans more than one line, Poppler has some fixes to make text selection (and therefore copy and paste) much better.

I don't claim to know if this is going to help any of my issues (or yours), but it seems that there are some nice bug fixes in there.

Thursday, June 17, 2010

OpenOffice 3.2 crashes (on close, save, random)

I'm a Linux user. I'm running Ubuntu (yes, because I'm not smart enough to install Debian, I know it). I like the idea of having most of my hardware issues sorted out up front. Hey, I installed Debian once, a couple of years ago, and just trying to work out my display resolution was enough for me.

When Ubuntu 10.04 Lucid was released I took the plunge to move to a new(ish) PC with a clean install. With that came OpenOffice 3.2. Which on my machine appears to be the buggiest piece of non-productivity software I've ever used. Now I know it must be my machine causing problems, because I'm not a software tester and I can get it to crash about 10 times per day (no exaggeration). So, what's the problem?

Well, I'll be editing something in the Writer and Presentation applications and suddenly OpenOffice will just die. No sign of nothing. Anywhere. Look at the System Monitor or run a terminal with htop and I see its still there. I fixed this (in the same way that I stopped Gnome crashing on this new-ish PC). I disabled Compiz. I just went to System | Preferences | Appearance and set the Visual Effects to None. No more Gnome crashing. It sounds like the Intel graphics device on my cheapie Nettop MSI Wind PC is not well liked by the Linux display drivers.

So now OpenOffice doesn't just go into hiding, but we actually get the proper 'recovery' dialog. Basically tells you that OpenOffice dies, sorry, and we'll try and recover your documents. Of course, I had the autosave set for 15 minutes, which is way too long when the program dies every 10. Changed that.

Now what? I'm trying two things:


  1. In the OpenOffice Options I deleted the Evolution connection from OpenOffice.org Base | Connections - don't know why, but I don't need these two programs being inextricably linked
  2. Deselect Use Hardware Acceleration from the Option OpenOffice.org | View menu
My guess is that #2 might help a bit. Maybe if my graphic card is not well supported, turning off Compiz helped to not crash Gnome, perhaps turning off funky hardware acceleration in OpenOffice will help not crash OpenOffice. That's my theory. I'll let you know pretty soon if its going to work!

[Edit: June 17th, 2010]
Yeah, nice try amigo. I got through most of the day, then I flipped back from browsing (in Chrome) to OpenOffice (using an Alt-Tab I think), and BLAM! - OpenOffice crashed. I hadn't touched it in ages. I think the hardware acceleration helped, a bit. But this crashed when the memory usage was high (maybe even paging) due to running a virtual machine in VirtualBox, and also running RhythmBox to listen to some tunes. There have been hints on the Ubuntu forums that playing music can be an issue. I wonder,  just wonder if OpenOffice tries to send some sort of alert sound, and this conflicts with the other PulseAudio stuff going on, and OOo just says 'no way'. Either way, back to the drawing board on fixing this problem.

[Edit: June 21st, 2010]
I've been playing around to see what the issue is here. I'm running with Sun Java "1.6.0_18 with accessibility support" according to the OOo options. I will try switching to the Free Software Foundation 1.5.0 JRE in a while. But first... I installed an alternative window manager: OpenBox. I've just run
> openbox --replace &
to see if there is some weird issue with Metacity, though I highly doubt it. I'll try for a while anyway.

[Edit: June 22nd, 2010]
Nope, not a Metacity issue, but I think I have made progress on this one. At least, I've not seen a crash in hours. I think the problem was self-induced. I stupidly wanted to make OpenOffice run as fast as possible, since it can be a bit sluggish sometimes. So I went with the option to run as native code using GIJ / GCJ. I installed the package openoffice.org-gcj that describes itself as:

This package contains the .jars from openoffice.org-java-common compiled to native to make the Java features of OOo faster when using gij.

Plus I installed GIJ and some other GCJ stuff. Well, I dumped it, removing the OOo component and anything with GCJ or GIJ in the package name. Restarted OOo and I've not had a crash since. I'm not quite ready to claim it really is solved, but I'm happy enough so far.

[Edit: June 25th, 2010]
It is better, but still goes splat. Next step is a complete uninstall, purge of configurations and reinstall. Maybe I'll get lucky.


[Edit : June 30th, 2010]
Done with it. I completely removed OpenOffice as packaged by Ubuntu, and downloaded the most recent release from openoffice.org
This is the 3.2.1 release, where Ubuntu was back at 3.2.0-ubuntu7 or something ridiculous.
The install is more painful in this way. I downloaded the 32 bit DEB version. Extracted everything to a directory, then installed with some nasty setting using dpkg.

cd DEBS
sudo dpkg -i --force-overwrite --force-depends openoffice.org*.deb desktop-integration/openoffice.org-debian-menus*.deb
# now, because the package manager is still telling me there are issues:
sudo dpkg -i ooobasis3.2-core06_3.2.1-18_i386.deb

Maybe this will work!

[Edit : July 6th, 2010]
OK. So I'm an idiot. All my dates for my edits said July, when I was really in June. So now its really July, its probably important for me to update this a little more.

Since I've transitioned to the 'vanilla' version of OpenOffice, it seems to be crashing less. I still have some nasty Word format DOC files that seem to cause problems (especially those with Word drawing stuff in them), but its better. My wonder is if there is an extension or may a skin that is Ubuntu specific that is causing some trouble.

Since I have not managed to be smart enough to download debug versions of the software (and frankly the OpenOffice website hasn't been particularly helpful in this regard), I'm going to have to assume that I'm not really going to be able to help, and it will just fix itself.

[Edit : August 10th, 2010]
I think I may just have the answer. I was getting this rather horrible message related to UNO in the crash reports that were being sent. Anyway, something was bugging me about the way OpenOffice was doing a spell check or something nasty in the background. I disabled Spellcheck, Hyphenation and Thesaurus from the Options. Then re-enabled the spellcheck, and nothing has crashed in a while. I hope this is the fix I needed.

[Edit : September 1st, 2010]
None of my efforts have succeeded. But I have just got through a day without a crash...


Yesterday I removed a slow Intel 5100 wifi card from my PC (this was an ill advised add-on). I had been struggling with the iwlagn wifi driver being unreliable for ages, and finally got round to rewiring to get a direct ethernet connection from my router. Anyway, for the first time in months I have run all day long without an OpenOffice crash. With multiple documents open, with all the activities I do still being done.

Really could this be related to a piece of network hardware? This might explain why people are seeing different results on different machines - a different wifi card, or different driver version perhaps. Maybe OOo or one of its components tries to chat to the network unnecessarily during certain events and it gets confused. Either way, I hope my luck holds out.


[Edit : October 13th, 2010]
In case anybody was wondering, no, my last 'fix' did not fix the problem. But if you check my post Chroot - ooh now I can run OpenOffice I think I have finally found a way to run the software without a crash. I've gone two days, with intensive editing, and some memory hogging VMs running in the background, and hey presto, no crash. So what? Well, a comparison with apt-show-versions showed me a couple of differences in OOo that appear worrying. Worrying that I shouldn't have to remove them, but worrying that if I don't I'm going to be battling this thing forever. Lets see if a quick sudo apt-get remove openoffice.org-gtk openoffice.org-gnome solves my problems. There were lots of discussions about this back in an earlier Ubuntu version,  Feisty or Gutsy I think. I'll keep you posted on this new revelation.


Wednesday, June 16, 2010

The stupid stuff I did

I've been messing around with software for years. Messing around professionally, and for fun. And there are many times where I've struggled with stuff for hours or days, wondering why something won't work, until 'Eureka!' I do something magical and everything is wonderful in the world. By the next day (well, hour often) I've forgotten what I did and how to fix it the next time it happens. And to be honest, I'm sure there are other people like me that also struggled, but had to give up because their day job just didn't allow constant tail-chasing.

Now, I'm not a software expert. I write it to be functional. I can see the beauty in the concept, but I don't code like those other guys. I fix stuff to work, well, and I stumble across issues with a good background technical knowledge and often a lot of luck. So, don't expect coding wonders, just some occasional thoughts distilled from many sources that might help other people who are also wondering what the hell is going on.

I'll credit my sources of info, and I'll happily share ideas and thoughts others have. Virtually nothing written here will be original, just hopefully the way it is presented.

I hope you enjoy, or at least find something useful if you stumbled here by accident. My other blog is far more 'professional' and targeted at getting me some consulting work in business process improvement and generally fixing business issues with technology, so this blog gives me an outlet for all the other stuff.

Happy reading!
Phil