How not to write documentation.


I've grown accustomed to ruby having poorly accessible documentation. What I mean by 'poorly accessible' is the fact that ri Array gives me a list of things Array can do (which is nice), but to actually find out about Array.delete I have to run ri Array.delete. I got used to Python's often-helpful ways of pydoc showing you what appears to be "as much as possible" when you pydoc a class, module, or method.

Maybe I'm doing it wrong. Either way, the following is annoying and unhelpful. While it tells me the arguments that should be passed, and what is returned, it doesn't help me really know more about the function. Luckly, I'm already familiar with select from other languages.

snack(~) % ri IO.select
------------------------------------------------------------- IO::select
     IO.select(read_array 
     [, write_array 
     [, error_array 
     [, timeout]]] ) =>  array  or  nil
------------------------------------------------------------------------
     See +Kernel#select+.
Ok, fine... Let's look at Kernel#select.
snack(~) % ri Kernel#select
---------------------------------------------------------- Kernel#select
     IO.select(read_array 
     [, write_array 
     [, error_array 
     [, timeout]]] ) =>  array  or  nil
------------------------------------------------------------------------
     See +Kernel#select+.
*sigh*

Site design refresher / CSS overflow property on IE 7


I updated some of this site's design. The most obvious changes are cosmetic (new banner logo at the top, slight color/layout differences). The least obvious change (hopefully) is that this site is now layed out purely with CSS, not tables.

After refreshing my rotting CSS braincells, I got a layout working properly and was quite happy. Then I tested in IE, and saw my <pre> tags being displayed incorrectly (according to my desire, not necessarily the code, which don't always align):

The above should have been two lines of text with a horizontal scroll bar. I want to say use scroll-bar for text that's too wide but expand vertically without a scroll bar: Never a vertical scroll bar, only ever a horizontal one.

I've spent a bit today scouring the web with not much help. I've randomly permuted css values for overflow, overflow-x, overflow-y, min-height, etc. Having failed that, I read everything I could find from randomly permuted search queries, one of which lead me to a depressingly long detail about IE6's expanding box problems. The page claims (and several others do, also) that IE7 fixes several box expansion problems.

I created a very small demo with minimal CSS to show the problem here: Click here to view the demo. It includes the solution I found, detailed below.

After some other random permutation, I gave up and tried wrapping the pre in a div and applying the overflow properties to the div. It worked. It's 2009 and I still have to deal with weird and obscure browser rendering inconsistencies. I came up with this:

<style>
  .scroll-wide {
    height: auto;
    overflow-x: auto;
    overflow-y: hidden;
    max-width: 500px;
  }

  /* On firefox, pre tags have a top and bottom margin > 0, which makes your 
   * scrolling div have a blank top line, which isn't what
   * we want. Fix one weirdness to find another? I didn't fully investigate.
   * Here's the fix: */
  div pre {
    margin-top: 0;
    margin-bottom: auto;
  }
</style>

...

<div class="scroll-wide">
  <pre>
    stuff
  </pre>
</div>

Solving this with wrap-pre-in-a-div can be automated with jQuery and a CSS definition:

// javascript
$("pre").wrap("<div class='pre-wrap'></div>");

// In CSS
div.pre-wrap {
  /* overflow/height/whatever options */
}
It's still possible I was doing something wrong and that this hack isn't necessary, but I don't know. I'm just glad to have it working now.

PS: If you use Meyer's reset.css, you'll want to include pre { margin-bottom: auto; } , or IE will again clip the bottom of the pre contents with the scrollbar.

RAID is not a backup techology.

I spent some time yesterday making backups of things I care about (code, content here, etc) in two remote places, in case anything should happen. Now two places are copying down data for me every few days. Backups are easy to ignore, but critical when you lose data.

Today, I heard about journalspace going down because it lost data and didn't have backups. While I don't use the service, so it doesn't affect me, the failure they experienced makes for a great case study in false data security and backups. From the front page of journalspace.com:

Here is what happened: the server which held the journalspace data had two large drives in a RAID configuration. As data is written (such as saving an item to the database), it's automatically copied to both drives, as a backup mechanism.

The value of such a setup is that if one drive fails, the server keeps running, using the remaining drive. Since the remaining drive has a copy of the data on the other drive, the data is intact. The administrator simply replaces the drive that's gone bad, and the server is back to operating with two redundant drives.
RAID is not a backup solution. RAID can get you, depending on the configuration, better throughput and/or better data reliability. If you lose a drive in some raid configurations, the system can continue working normally without that drive. Backups should copy data somewhere other than the machine hosting the original data. The page goes on:
So, after nearly six years, journalspace is no more.
After almost 6 years nobody had a cron job that backed up data to somewhere offsite (or a more complex backup system)? Ouch! My condolences to journalspace and its users on the loss.

Losing important data unexpectedly will sting you bad if you don't have appropriate backups. The only thing to do is learn from this mistake and move on, accepting the consequences of the loss.

This isn't the first website I've heard of having to shutdown because they permanently lost data. Learn from their mistakes, keep backups of your stuff!

Found some old presentations

While doing backups of my important personal stuff (this website, my code, etc), I found some presentations I'd done in school: I also removed a pile of comments that were spam or people testing the comment system. Some comments had unicode characters in them causing python to barf (I have a python cgi script that helps me search/delete comments), but doing .encode("latin-1") fixed it.

There were only a few spams that got past my scripts. I might add word filtering to the pyblosxom comment plugin so crap mentioning viagra and cialis won't even be permitted. Still seeing these happily in my logs:

REJECT: Comment attempt by 193.203.99.31 rejected. Reason: expired secret, referer: http://www.semicomplete.com/blog/articles/week-of-unix-tools/day-1-sed.html
REJECT: Comment attempt by 78.110.175.18 rejected. Reason: javascript test failed (expiration ok), referer: http://www.semicomplete.com/projects/xdotool/

Goodbye, 2008!

Another year later, and much has changed.

Basic life summary: Left Google for a startup company. Got married.

This year was a great year of learning and exploration. The learning started by evaluating different regular expression systems: from Oniguruma and C, to Boost and C++, to PCRE and C - all in trying to find a non-perl platform for grok. Learning flex and bison for grok's config parsing was fun. I also learned how to write C modules for Ruby and Python.

Greater expansion of my X11 knowledge occured in an effort to improve xdotool and keynav; other X11 work was in toying with the idea of writing my own window manager. The window manager was never completed, but I learned a great deal more about how things in X work. I also learned ruby, ruby on rails, more automation skills and more networking knowledge. Being one to always use the tool that most easily and appropriately solves a problem, I went into the realm of Windows. This journey has so far taught me VBScript, PowerShell, WMI, sysprep, and a host of random windows management tricks.

My new job brought me exactly what I needed - a fresh pool of things to learn and problems to solve. Since leaving Google, I never found myself wishing I had stayed. It doesn't hurt that I haven't had to carry a pager in the past eight months (something we'll be changing in the future as SLAs will demand it). The various projects I've been working on at the office are pretty cool, and some day I may write about them here.

In consumerland, I signed up for Rhapsody and haven't regretted it. I tried the iTunes store, but I don't fit iTunes. I love Rhapsody. It would cost me more than $3500 (at $0.99/song) to replicate my growing Rhapsody library in iTunes, but it's cost me less than $100 (6 months of service, so far). For Christmas, I got a Sansa Fuze which I can use with Rhapsody (For a $2/mo increase in service to upgrade to 'Rhapsody To Go') and most any music I want. Rhapsody is awesome, and iTunes can suck it. As a bonus, I don't have to deal with any retarded headphone adapters to listen to music on my iPhone anymore, thanks to my new Sansa Fuze.

Event happenings this year seemed less frequent than past years. Still, there were good ones. I did a cool hack at Yahoo! Open Hack Day with Delicious and Yahoo! Pipes. I again helped run the Hack or Halo competition at Shmoocon 2008, which went smoothly. The annual trip to Vegas for Defcon occured, too. Lastly, but not leastly, I flew out to Rochester for BarCamp Rochester 3 and hung out with lots of college friend folk.

It feels like I coded less this year. Reviewing this year's posts, I see less significant project work than previous years. Grok got ported to C++, then distaste for that lead to porting it to C. After months of barely hacking on it, the C version finally got a beta release. Tabsearch, keynav, and xdotool received some love and attention in the form of updates, features, and bug fixes.

I started giving up my habit of running things myself (AKA, love the cloud). One change I made this year was to start using Google code hosting. Despite my odd usage (multiple projects under the same name "semicomplete"), it's been awesome. Publishing and tagging releases is trivial, subversion change logs are easily viewable, and the source code viewer is easy to read. There are other features, but I don't use them.

I also switched from using mh as my mail reader to using google hosted apps. My familiarity with using gmail at Google made the transition easy. This switch has let me stop caring about training spam assassin and stop wonderng why so much spam gets through, because my spam experience with google hosted mail is much much nicer. Gmail good.

December was spent writing articles for sysadvent - a systems administration advent calendar. Despite the stress of forcing myself to write so much, it was one of the most valuable things I've done online this year. Sysadvent is part of a response to my realization that I don't really have any community ties. I'm a community of one. I work on my own tools and projects. I'd like to help elsewhere, outside of the office, but nothing has so far attracted me.

Lastly, some of this year's notable hacks included an attempt at speeding up build systems with caching, vim function to make C++ Template error messages readable, and a few gnu screen scripts to make life easier. I also keep sample code in revision control in an effort to provide sample code to help searching.

Here's to a fantastic 2009 :)

Grok beta 20081228 available

The new C version of grok is ready for beta testing.

The requirements are listed in the INSTALL file. There are piles of differences between the new C version and the old perl version, including a different config file syntax to let you more easily batch common input sets through the same set of matches. I'll publish a complete feature list when I get around to it, which isn't right now.

The tarball comes with a sample grok.conf that shows you a a few different things you can do with the new version.

To run it, once you've built it, you must have a 'grok.conf' in the same directory from which you are running the 'grok' binary.

Please send any questions you have to grok-users@googlegroups.com.

Download: grok-beta-20081228.tar.gz

Sysadvent 2008 finishes

Sysadvent 2008 came to a close tonight as I clicked the publish button.

I honestly thought I was going to burn out after 4 or 5 days of writing. Somehow I maintained sanity and was able to write 23 of the 25 articles. Ben Rockwood and Matt Simmons tossed an article each my way, which was a great help in letting me relax. Much thanks to them. I also had help from a few friends who suggested ideas or gave pointers when I had researcher's block - when I couldn't decide what to research and write about.

My wife reminded me every night around 8PM that I should start writing. Without her helping me remember my goal (an article a day), I probably would've burnt out or slacked off playing games or coding, so the successful completion of the first sysadvent is due greatly to her caring for my own projects. <3

Each article took a varying amount of time. Some took longer to research, some took longer to write, and some took longer to rewrite. On average, each article probably took on the range of 2 to 6 hours to complete (research, write, rewrite). Some probably took longer, but after a few hours I lost track of time. When I got stuck hard, I would play TF2 and Left 4 Dead for a bit to distract and help me refocus. I learned that taking breaks is critical for breaking through walls of unproductivity.

The whole research-and-fiddle-with-a-tool experience was pretty sweet, except for the stress of a self-imposed deadline meaning I had to stop researching and start writing.

For those that read it, I hope you enjoyed it. If you have comments, please email me (jls-sa@semicomplete.com).

Does Linux have the potential to be as good as Windows for gaming?

I know you can download Wine to run Windows games on Linux, but I want to make sure that everything in Linux can be just as good as it is in Windows XP. I'm wondering if networking will screw up considering the rest of my family runs on Windows… Does that mean no LAN? Stuff like that. I'll post more details if people need more so check back.

Most games are compatible with Windows, and nothing else. Some of them have Mac versions, and some of them have Linux versions (Quake 4, Neverwinter Nights, etc.), but most of them just run on Windows.

So if you spend a lot of time playing recent games, you should stick to Windows. But you can still install Linux, keep Windows (see the install section), and use both of them, depending on your needs.