Be Freely

MacBook revisited

2011-06-30T00:40:00.000-07:00

A few more hours of trying to make the Mac do my bidding, I've decided to set up a Debian virtual machine to act as a "test server", partly due to the limited availability of open-source software pre-built for OS X. (Yes, I know about Homebrew. It broke the first time I tried it... moving on.)

Using VirtualBox, the Debian installer (brilliantly) detected that I was installing it in a virtual machine, and thus also installed the package virtualbox-ose-guest-utils, among others.

Unfortunately the OSE version of VirtualBox does not include the ability to share filesystems between host and guests, so I elected to replace it. For a number of reasons, a minimal Debian installation doesn't play very nice with the standard (non-OSE) guest additions installer, so I had to find another way...

The short and sweet, for Debian Squeeze...

Add this repository to your /etc/apt/sources.list, if you don't already have it:

deb http://backports.debian.org/debian-backports squeeze-backports main contrib non-free

Then `apt-get update` and `apt-get install virtualbox-guest-utils`, and reboot the VM.

In my case, the shared filesystem was mounted automatically on boot, in /media/.

Additionally, the VM was given 3 network interfaces:

Host-only
Bridge to wired ethernet
Bridge to wireless network

This has the effect of getting the VM internet access whenever possible, while still making it accessible as though it's a separate host, even when I'm not actually on a real network - useful for testing web applications. The bridged interfaces also make it available to other machines on my network (in my office), which may not always be preferable, but for now I can see it as a convenience.

OK, now to get something done...

MacBook

2011-06-29T20:50:00.001-07:00

My boss recently replaced his old, dying Windows laptop. I'd observed his usage patterns a bit previously, and noted that he keeps a lot of stuff open for very long periods; he'd hibernate multiple times daily, not rebooting for several months on end. He'd regularly have his Windows task-bar 3-rows deep, completely full of running applications, which he avoided closing at all costs.

On my recommendation, he bought a MacBook. I figured it would get him the stability of a *nix (backed by Darwin, after all), and the UI would provide him a generally more efficient way to manage his work.

That worked so well for him that he decided the company would standardize on MacBooks. As a regular Linux user, I wasn't quite sure what to expect for my own workflow, but I was optimistic.

So far I've reinstalled the OS 3 times, in 2 days, due to various mistakes I've made in setting up the system to my liking. Learning a new system, for engineers (i.e. tinkerers) like me anyway, is bound to include a few mis-steps and whatever work required to recover from them. Fortunately the core OS X installation is reasonably comprehensive, far more complete than Windows could even hope to be.

After today's most recent re-installation, I decided it'd be nice to have these things documented for newcomers to the OS X environment. So here's what I've learned...

If/when you reinstall, OS X must be installed on a case-INsensitive filesystem, since some applications will use different cases for the same paths (e.g. /Users/ vs /USERS/).
Once you enable FileVault encryption, you should not change your user's "short name", since it prevents you from logging in again. Change your short name first, if necessary.

Customizations to make the transition from Ubuntu (etc) less shocking:

In Finder, enable "Show Path Bar" under the View menu
In Terminal, create a customized color profile with a transparent background, and set it as default.
Install MacVim, The Unarchiver, Cyberduck, and Git

Other points of interest...

Enabling Spaces (virtual desktops) is done in the System Preferences, and then switching between then with keyboard is a matter of Control + Left (right, up, down, 1, 2, etc)
Copy and paste works in the terminal due to the Edit functions being accessed via the Command key, rather than Control
There are no programs to let you view the contents of an archive - every utility I've found, including the built-in ones, require that you extract it entirely first, as far as I can tell... (except `tar` on command-line)
Filesystem support on the Mac is limited - it's pretty strictly Mac-format (HFS+) volumes and FAT32 or exFAT. Open-source filesystems like ext3/4 are not supported, from what I see, which is quite disappointing... and requires that I set up a Linux virtual machine in VirtualBox, in order to work on the external HDDs I usually use.

So far the learning curve has been notable, but not disastrous, and I think I like the Mac experience, in spite of its shortcomings. We'll see how this pans out...

Google Chrome Password Export

2011-05-21T20:07:00.000-07:00

Occasionally Google Chrome crashes. (Surprise!)

A few times now, it's blown up my profile data, which includes bookmarks and passwords that I do not want to have to look up, reset, etc.

Passwords are not what I want to burn mental energy on. It's best when my browser can manage them properly, and hopefully the same for bookmarks, etc. Fortunately Google Chrome's sync feature is reasonably reliable, but (being paranoid) I can't afford to have it "accidentally" delete all my saved passwords.

So I found a way to export them, just in case. Thankfully Chrome uses SQLite databases (extensively), so the data is fairly easy to access.

On Ubuntu Linux, very simple step-by-step:

sudo apt-get install sqlite3
sqlite3 -csv ~/google-chrome/Default/Login\ Data 'select * from logins' > chrome-passwd.csv

Et voila! Your passwords are now in a format that you can read in any sensible spreadsheet program, should the need arise.

The passwords were not stored encrypted, as far as I could see... I believe Chrome's encryption model for personal data is applied strictly to transport between Chrome and Google's servers. (Please let me know if I'm missing something.)

You should keep that file safe, given that it contains plaintext passwords. Ubuntu's standard encryption (ecryptfs) is probably fine, but there are lots of options.

If you're using Windows, you'll have to install SQLite (version 3) and figure out the rest of this process on your own... and if you need encrypted storage, I recommend TrueCrypt. (I'd also recommend ditching Windows, but that's another story...)

Paranoid filesystem access

2011-05-04T19:02:00.000-07:00

In some of my work projects, it's necessary to grant filesystem access to our clients and contractors, so they can upload and manipulate data on our servers.

I'm paranoid. The systems I manage are a tight ship - if you want on the ship, you will conform to our security requirements. Resistance is futile.

OpenSSH offers a handful of very useful features in this regard. The following configuration provides some nice features:

Users in the sftp group are chroot-ed into their home directories
Users in the rsync group can run rsync only within their home directories.
Multiple users in a client's organization can share a home directory
I can switch a user from sftp-mode to rsync-mode by changing their groups.

Relevant parts of SSH config: http://pastebin.com/SUQFLVmH

Rsync shell wrapper: http://pastebin.com/HGSfn18H

User setup in this environment is slightly trickier, so I've scripted it... highlights:

mkdir -p /path/to/$GROUP/.ssh
touch /path/to/$GROUP/.ssh/authorized_keys.$USER
groupadd -f $GROUP
chgrp $GROUP /path/to/$GROUP/
useradd -g users -G sftp -d /path/to/$GROUP/ -s /bin/bash -c $realname $USER
chown $USER:root /path/to/$GROUP/.ssh/authorized_keys.$USER
chmod 0444 /path/to/$GROUP/.ssh/authorized_keys.$USER

Some blanks to fill in there for your SSH public keys, etc. Use group 'rsync' to switch to rsync-mode, rather than 'sftp'.

So now external users can push data to us, edit files we're hosting for them, etc... and I can still sleep at night, paranoia and sanity still intact.

Synergy

2011-03-29T10:08:00.001-07:00

I do a fair bit of web development, in various senses. I spend a lot of time testing the same code in various common browsers, and thus have to use multiple operating systems. (This is one of the more annoying parts of such work.)

I have a netbook that dual-boots Ubuntu and Windows; my main workstation is exclusively Ubuntu. Switching back and forth between the two takes more time and ends up being more frustrating than I like... but last night I found a solution.

Open-source software saving my sanity once again.

Enter <synergy-foss.org>, a lightweight tool for sharing keyboard and mouse between multiple systems, simulating a multi-monitor setup across multiple physical computers.

The client/server model of it struck me as counter-intuitive at first, but one adapt's... the "server" is the main machine with the keyboard and mouse; "clients" are other machines that behave as extensions on the primary desktop. When a client connects, the server's mouse/keyboard can move between displays as though it's a single (multi-monitor) desktop.

Ubuntu's community documentation provides a nice introduction.

Very nice. I may even donate to Synergy's developers.

Debian Desktop - take 2

2011-02-15T12:35:00.000-08:00

So I was given this old 1.4ghz Celeron eMachine a few years ago. At the time it has Windows XP, and wasn't worth booting - just inconceivably slow. At some point since then, I installed Debian Lenny on it, and used it for a side project here and there. That worked well.

In prelude, just as "white is the black" or whatever, "Squeeze is the new stable" (har har har).

Recently Debian Squeeze was rolled out, and my Lenny installation was aging anyway. So I burned the recent XFCE install CD, booted it, and began installation.

A few notes along the way...

Something about the graphical installer environment doesn't like this hardware. Screen goes all flashy, initiates seizures. In the ISOLINUX boot menu, go to 'Help', and then at the 'boot:' prompt, enter "install vga=791" ... not always the ideal setting, but lets you get to the less-pretty installer anyway.
Squeeze supports EXT 4. Yay!
Disable the Desktop Environment item in the 'tasksel' stage - leave only 'Standard...' option selected. This gives you a viable shell environment to build up a minimal desktop from, if you don't need all the features of a full Debian-style desktop.

Next up, installing packages... login as root locally, after you reboot.

apt-get --no-install-recommends openssh-server
apt-get install sudo vim-nox autofs
apt-get install xfce4 xfce4-terminal gdm
apt-get install xfce4-places-plugin xfce4-goodies thunar-archive-plugin thunar-media-tags-plugin thunar-volman thunar-thumbnailers
apt-get install chromium-browser chromium-browser-inspector
apt-get install xrdp vnc4server

In my case I'm using this box as a music manager, so it's useful to have remote desktop options... so I've enabled XDMCP for GDM. On my Ubuntu box, I'm using Xnest to connect to it...

Xnest :10 -query -sss -geometry 1280x1000

Alternatively, I've also set up XRDP, so I can access the same desktop from a Windows-based client.

I'm not 100% decided on which music management software I find most useful, so I'll save that for a later post.

Overall, I'm quite pleased with Squeeze. The steps outlined here required very minimal manual configuration, so getting a viable desktop with access to local network resources was quite convenient. Good work Debian!

Updates...

Added Thunar plugins and other XFCE conveniences
Added XRDP alternative for Windows-compatible Remote Desktop

Installing WordPress in a Subdirectory

2010-08-12T09:47:00.000-07:00

If others have posted about this already, I couldn't find it last night.

We're converting a domain for a client from hosting some demo content to hosting a WordPress site. In the process, we need to keep the previous URLs valid, but let WP provide the document-root. I also like to keep things organized, so WP will live in its own directory (/WP).

First, extract WordPress to the /WP directory, and update its configuration per the manual. Your life will be easier if you STOP before step 6 (install.php), and setup these rewrite rules first...

For this situation we have two .htaccess files.

One at the document root...

# /var/www/default/.htaccess
RewriteEngine On
RewriteBase /

# by default, /wp-admin redirects to the full
# path to /wp-admin/ - in this case /WP/wp-admin/
# which is BAD, and breaks login.
# this rewrite preempts that by adding the
# trailing slash, so that wp-admin goes directly
# to login without the path
RewriteRule ^(.*)wp-admin$ $1wp-admin/ [R=302,L]

# Shortcut for directory index
RewriteRule ^$ /WP/ [PT,L]
RewriteRule ^/$ /WP/ [PT,L]

# Pass all non-file queries to WordPress
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*)$ /WP/$1 [PT]

And one for WordPress...

# /var/www/default/WP/.htaccess
# managed by WP, essentially its default
# BEGIN WordPress
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]

# END WordPress

OK, you can now proceed with step 6 on the manual - install, without using /WP in the URL.

(2010-08-13: update for directory index shortcuts)

Debian Desktop

2010-07-26T15:47:00.000-07:00

Thus begins a series on how to do what I do to set up a Debian desktop.

In this case, I'm setting up my IBM ThinkPad R51 (1830). This is a 5-6yr old laptop that's been running Ubuntu for a long time. Ubuntu served me well, but I'm exploring some alternatives of late since I've got a workstation that better suits my professional needs.

This will address the procedures at a fairly high level, assuming you know plenty about managing a Debian system. Other people have documented the myriad relevant subjects, so I'll skip most of it here. Read instructions where links are provided.

So, for today...

Start with the Debian 5.0.5 XFCE installer. Backup your data first, then boot from this CD and proceed with installation. Partition it like so:

256mb for /boot, ext3
1.5gb for swap
12gb for /, ext3
remainder (~25g) for /home, ext3

Proceed with installation as instructed.

When the system reboots, log in, configure the desktop to your liking.

For networking, this ThinkPad uses the ipw2200 firmware. Install it and relevant configuration tools.

Add non-free to your standard debian sources.
Add backports repository. You probably don't need to worry about pinning.
echo 'options ipw2200 led=1' | sudo tee -a /etc/modprobe.d/ipw2200.conf
apt-get update; apt-get install firmware-ipw2x00 wicd
reboot or modprobe ipw2200

A few more desktop niceties:

apt-get install human-icon-theme xfce4-{places-plugin,utils} thunar-{archive,media-tags}-plugin thunar-{thumbnailers,volman}

Tools for synchronizing my work on multiple systems:

apt-get install rsync unison git-core

Tools for web development:

apt-get install iceweasel-{firebug,webdeveloper} flashplugin-nonfree

And... that's it for now. That much gets me to a point where I can surf and post blog updates like this ;) More will be forthcoming as I figure out what else I want to do on this system.

ISP Linux Mirrors

2010-07-26T14:51:00.000-07:00

Ubuntu has been my stand-by for a few years, since my laptop was my primary workstation as well. Recently, due to some hardware upgrades, I've begun using a proper workstation with dual-monitors (and a resolution higher than 1024x768, finally!!!).

I tried a couple of distro hops, trying Source Mage and Gentoo (as I do on my servers) but today realized that the amount of work needed to make a source-based distribution as featureful as Ubuntu on my laptop would take an insane amount of work. It'd be possible, just not feasible with my currently very limited spare time.

In the process I stumbled across a familiar name in the list of Debian mirrors: frontiernet. I've also recently switched to Verizon FIOS, which sold me to Frontier communications - my new ISP.

After a few minutes research, I discovered that my ISP hosts CentOS, Debian, Ubuntu, and OpenBSD. Brilliant! And a little testing confirms that it's a bit faster than my previous preference: OSU OSL.

Good stuff. In my opinion, ISPs should do more of this - as the customer's use of FOSS would reduce malware traffic (including Windows Update) on their networks and improve overall security. Seems like an ethical move to me.

Data Extraction (packed EXE)

2010-06-17T22:01:00.001-07:00

I don't often blog about Windows stuff. I try to avoid Windows. This is an exception.

Some time ago I received a slideshow in an EXE. It was meant for previewing images before printing. Since then I've occasionally wanted digital copies of the images in it, but never had the time/patience to dig into it.

Tonight I found the gusto and made it happen. Here's how...

Try opening the file in 7-zip or Resource Hacker. If both complain, it's probably packed.

Determine the packing method used on the EXE. A common packing method is UPX, which has a free tool to pack/unpack the data.

Mine was packed using aspack. I discovered this by peering through the binary data of it. (Not for the weak of heart.) A quick way to check this: on a Linux machine, 'grep aspack yourfile.exe'.

Unpack your file. UPX makes it easy, as stated. For me, I found a tool called 'aspackdie' (version 1.3d) which seemed to like the file. It reported that the file was packed using ASPack 2.12, and reported that it appended "extra data" at address 046DD740h - to a destination file, unpacked.ExE.

I then opened it in WinHex, moved to that address (measured from the end of the file!), and specified that ending section of the file as a "block". I instructed WinHex to look for audio and photo files in various formats, within that range, and to look at every byte in the file.

About 10 minutes later, I had 1200+ photos waiting for me to inspect... which I promptly archived, along with the original presentation, to a CD.

Xbox360 USB Storage

2010-06-09T15:03:00.000-07:00

Anyone who bought an Xbox 360 with the original "20gb" release of harddrives knows that they're restrictively low on capacity. This is made worse by the Xbox's system caching procedures (etc), so ultimately you're left with about 14gb of user space on the drive. If you play more than a dozen games, each of which have updates downloaded from Xbox Live, your drive fills up very quickly.

I'd been getting nervous about it, thinking I'd have to go drop some exorbitant amount of money on an over-priced, Microsoft-branded harddrive, when they decided to enable storage of Xbox data on USB drives. Since I've got a couple spares lying around, I opted to try it out. Here's what I've learned...

The Xbox 360 supports only one partition per USB drive, and it should be FAT32.
Extra partitions can still exist on the drive, for use on other computers, but only the first is used/recognized by the Xbox.
If you provide an un-partitioned drive, the Xbox 360 will create a filesystem filling the entire drive - which breaks on some systems. I recommend pre-partitioning it.
When the Xbox 360 "configures" the drive, it creates a directory and occupies/reserves 16gb (or however much you select) for itself - in chunks, since FAT32 doesn't support more than 2gb per file very well.
The remainder of the space on that partition is available for you to store music, videos, etc - but you'll have to do that from your PC.

While I think Microsoft is (per usual) playing the tyrant with it's 16gb limit, this feature is at least a marginal improvement. It's significant enough for me that I appreciate it.

The technical (filesystem) requirements are specific, but not terribly unreasonable, since the practical benefit of using such a harddrive with an Xbox 360 pretty much ties the drive to the Xbox. Most users shouldn't be too hassled with it.

Improvements I wish Microsoft would make:

Remove the hokey 16gb limit - it's not a technical limitation, it's a marketing decision.
Allow games to be installed on the drive completely, not requiring the disc to be present. I don't mind tying the installation to my hardware and the disc (jointly), but I want the convenience of playing without inserting the disc at all!

That's it for now. I'll be playing with my new expanded Xbox 360 storage if anyone needs me...

NAS Setup

2010-04-09T16:00:00.000-07:00

Long story short, I'm transitioning a small company away from a full-featured server. (The old server is showing its age, and the domain authentication introduced too many complications for this client.)

Instead, we're using two NAS devices for two different sites, which will synchronize nightly.

Each one is composed of:

1x Synology DS410j NAS unit
4x Seagate Barracuda 1TB drives (firmware CC38)

I ordered them from Amazon. I used to be a NewEgg fan, but these days Amazon tends to have slightly lower prices, and NewEgg has goofed up enough orders to frustrate me.

So a quick summary of the set-up process...

Install the drives in the Synology unit
Download the latest software from Synology (for your OS)
Install the Synology Assistant, which should find your device on the network
Upload the DSM "firmware" to the device, configure as needed
Log into the device via your browser (port 5000 on mine)
Configure it further, and set up (initialize) the drives
Start pushing data to it

The "firmware" upload took about ten minutes, and the drive initialization is taking... something on 7 hours so far. While waiting for the drives initialization, I decided to investigate the device a bit.

As far as I can tell, the "firmware" is actually a 110mb tarball of the embedded Linux installation that drives this device, which gets extracted to the installed drives.

Port scan results (after some configuration) with notable items emphasized:

PORT STATE SERVICE VERSION

22/tcp open ssh OpenSSH 5.2 (protocol 2.0)

80/tcp open http Apache httpd 2.2.13 ((Unix) mod_ssl/2.2.13 OpenSSL/0.9.8k PHP/5.2.12)

111/tcp open rpcbind

139/tcp open netbios-ssn Samba smbd 3.X (workgroup: WORKGROUP)

445/tcp open netbios-ssn Samba smbd 3.X (workgroup: WORKGROUP)

2049/tcp open rpcbind

5000/tcp open http Apache httpd 2.2.13 ((Unix) mod_ssl/2.2.13 OpenSSL/0.9.8k)

5432/tcp open postgresql PostgreSQL DB

MAC Address: 00:11:32:02:B8:XX (Synology Incorporated)

Via SSH, some other (technical) observations:

uname: Linux Synology 2.6.24 #1141 Sat Mar 13 00:51:39 CST 2010 armv5tejl unknown
RAID and volume management: Linux mdadm and lvm
Web servers: Apache and Lighttpd
Databases: PostgreSQL and MySQL
Media services: mt-daap, vlc

Root password seems to be the same as the admin user's, so you can SSH in as root as soon as the service is enabled.

It looks like the system allocates about 3gb on each drive for its "firmware" and user settings, mirrored across each drive (2x RAID 1 arrays). The device blocks on volume creation (initialization) while mdadm is syncing them, and lots of configuration options require the volume to exist. This results in a total setup time of about 8 hours.

The primary volume that it automatically allocates consumes the entire remaining drive space, which is likely OK for most users. In my situation it automatically set up a RAID 5 array with (the remaining space on) all 4 drives. I like that. Advanced Linux geeks can adjust the volume allocation, since the system provides resize2fs and the usual suite of LVM tools - though at this point I don't see myself needing that, since the device uses ext3 - very high limits on the size and number of files in the filesystem.

The setup process took something like 10 hours after all. I don't know exactly, because I decided to go to bed around 80% completion. Best to do the next one overnight.

Data transfer to this device is a little bit disappointing. I have it on a 100MB ethernet line, which usually gives me 10-12mb/s of real transfer on other machines. On this device, using rsync (regardless of compression) I get around 7mb/s; piped through SSH, only 3-4. As far as I can tell this is due to the rather light processing capacity of the ARM processor it uses.

Still, I'm satisfied with this device and the functionality it provides - especially rsync and NFS.

An eventual update - after a few days of working with the system...

I'm still pleased with these devices. They do their jobs pretty darn well.

Network backup in our environment has proven to be a little bit tricky though. On one end of the backup flow, we have an existing infrastructure requiring standard rsync and ssh ports.

The set up is a little bit convoluted though, so I'm documenting it elsewhere.

I'd also recommend the use of the optional software packages maintained by NSLU2-Linux. For the Synology devices I've used, this script is the right one. Most useful to me is the 'fdupes' tool, which finds duplicate files and lets you delete them - nice for saving space.

One thing I miss is 'unison' - a 2-way synchronization tool... like rsync, only smarter about changes occuring in multiple sources. The optional packages don't include it, sadly, so I'm pursuing another route: from one primary workstation, rsync to the Synology device - and have all the other workstations/laptops sync with the workstation via Unison.

Another day...

So here's where the rubber hits the road. I brought the on-site device to the business location for the initial deployment. On arrival, I discovered the wireless network wasn't operating properly, and on further investigation, that the router was basically toast.

The next morning, I went to pick up a replacement, only to discover then that it didn't support any sensible DNS services (or DHCP reservation for that matter) - I don't recommend the Linksys BEFRS81.

I'm on a tight schedule though, so I move ahead and let that one play in the back of my mind. After setting up some other hardware, I realized I could set up DNSMasq using the optional software kit. I'm familiar with DNSMasq in my home network, and it serves me well there - so I deployed it here, and disabled the DHCP features of the router.

Good stuff. Right now I'm very thankful for open-source software and platforms that remain open to customization.

Network Archival

2010-04-01T13:41:00.001-07:00

Today's wonderful Linux utilities: SquashFS, NFS, and automount.

If you use Linux, you probably know about tarballs: archived collections of files, usually compressed using Gzip or Bzip2. SquashFS takes those to the next level.

The highlights for SquashFS:

Threaded, so it can take advantage of modern multi-processor hardware.
Duplicate file detection, optimizing net compression.
Replicates all filesystem metadata (by default)
Mountable archives, so they can integrate directly into your filesystem
NFS-exportable. (Oh, and NFS rocks too!)

In combination with NFS and automount, SquashFS just makes my day.

I archive all kinds of stuff, and SquashFS generates near-optimal results, ensuring that the result consumes the minimum space on disc. (The result is read-only, mind you, but that's great for backups.)

With NFS and automount, I can list specific directories in my automount pool, over the network (NFS), and the archives are automatically visible.

Some configuration samples...

# /etc/auto.squashfs.loop
* -fstype=squashfs,ro,loop :/srv/autofs/&.squashfs

# /etc/auto.master
/media/loop/sq /etc/auto.squashfs.loop --timeout=30 --ghost

# /etc/exports
/media *.local(rw,async,crossmnt,no_subtree_check)

The result: from other workstations:

$ mount storage:/media /mnt/storage
$ ls /mnt/storage/loop/sq/archive
[ directory listing within the archive.squashfs ]

When I want to add a new directory, I just drop it in /srv/autofs on my storage machine - et voila, everyone in my network can peer into it, while disc consumption remains minimum.

Linux making my day as usual...

Ode to SSH

2010-02-07T23:35:00.001-08:00

Dearest SSH,

"Have I told you lately, that I [SSH] you?"

You encrypt my shell sessions.
You forward my TCP ports.
You proxy my HTTP sessions.
You compress my various tunneled applications.

...

Now is one of those moments when I reflect on how much benefit I get from a particular open-source project. Sometimes it's awk, Linux in general, etc... this time it's SSH.

Some of the things I use SSH for:

Secure terminal access to my servers (obviously)
Accessing my music archives from afar (DAAP over SSH)
Remote Desktop, only much safer (RDP over SSH)
Safely administering remote databases (MySQL over SSH)
Safer browsing from other networks - hotels, coffee shops, etc (SOCKS proxy via SSH)

My working environment, with the benefit of SSH:

Using GNU Screen, I can resume my work anywhere.
With vim, I can have a sensible programming environment on very low bandwidth
Git lets me push code around between several machines efficiently
Rsync (or unison) lets me synchronize and maintain file systems very conveniently
I issue updates and test code (mostly web apps) from anywhere, as though I were on site

Very rarely do I have to think about security - my SSH configuration lets me focus on business, rather than logistics. (Mind you, I had to think about it in order to set it up the first time, but once it's done, it's wonderful.)

Dearest SSH, Thank you.

Gentoo won't boot!

2009-12-22T08:25:00.000-08:00

I recently moved from Spokane to Portland. In all the chaos of finishing some things for my work, disassembling my home-office, and putting it all in a moving truck, etc, I neglected one critical element...

I had installed an update to sys-fs/udev which required kernel 2.6.27 or later, while the most-recent kernel I've got installed/compiled/etc is 2.6.25 (plus a couple others which were fubar). I had neglected to keep a recent kernel in working condition.

Gentoo users/admins will see the problem immediately (and probably shouldn't read the rest of this post, since it's not terribly magical).

Old kernel: udev won't run, so it won't see devices, especially my RAID array.
Newer kernels don't work at the moment

So I'm pretty much stuck. (If anyone can imagine a Windows machine in this situation and suggest a way to fix such a problem on Windows, I'd love to hear it...)

Fortunately I already had the sources for 2.6.31 on my server's filesystem, so I dug out a copy of System Rescue CD (somewhat aged), booted the server, mounted the RAID devices in their appropriate locations, and chroot-ed into my server's filesystem.

Then I:

Took a snapshot of /boot and /lib/modules
Copied the System Rescue CD's kernel config (/proc/config.gz) into my new kernel directory (/usr/src/linux/)
Used 'make oldconfig' and 'make menuconfig' to bring it up-to-date and fine-tune it a bit.
Use genkernel to build the new kernel and initramfs (including RAID/LVM tools) and install them
Adjust my GRUB configuration, reboot, voila!

Mind you, at this point the kernel config wasn't optimal or terribly awesome, just functional. I'm currently in the process of tuning it further.

So... that's how Linux saved my sanity today.

Time Management Matrix

2009-12-04T14:26:00.000-08:00

So, in the last year I picked up use of Covey's Time Management Matrix (thanks to the recommendation of my previous boss).

It works great on a whiteboard, but I've always wanted a web app to do it.

So I set up a Google Spreadsheet to help me out. Here's how...

Create a new spreadsheet.
Rename the first sheet 'Entries'
Create a new sheet called 'Matrix'
On the 'Entries' sheet, label three columns:
- Name
- Importance
- Urgency
Create some example entries (see screenshots below)
Go back to the 'Matrix' sheet
Fill the cells in like so:

A2: =SORT(FILTER(Entries!$A$2:$C$100, Entries!$B$2:$B$100 >= 5, Entries!$C$2:$C$100 >= 5); 2; 0; 3; 0 )
D2: =SORT(FILTER(Entries!$A$2:$C$100, Entries!$B$2:$B$100 >= 5, Entries!$C$2:$C$100 < 5); 2; 0; 3; 0)
A14: =SORT(FILTER(Entries!$A$2:$C$100, Entries!$B$2:$B$100 < 5, Entries!$C$2:$C$100 >= 5); 2; 0; 3; 0)
D14: =SORT(FILTER(Entries!$A$2:$C$100, Entries!$B$2:$B$100 < 5, Entries!$C$2:$C$100 < 5); 2; 0; 3; 0)
Cells A1:C1, D1:F1, A13:C13, and D13:F13 are left for labeling each section - and can be merged sensibly.
I recommend marking borders along the right-hand side of column C, and the top of row 13 - for clarity.
Also, columns B, C, E, and F can be shrunk to improve usability, since they contain integers 1-10.

And... that should be it! Create task entries, assign importance and urgency, and it should automatically appear in the appropriate quadrant on the Matrix sheet.

Screenshots:

Compact Nautilus in Ubuntu

2009-11-19T10:55:00.000-08:00

So, I occasionally note the compact Nautilus view that Fedora and similar distros use by default, because it's neat and tidy.

I prefer Ubuntu on my desktops though, which uses the more expanded view, something like this...

Unfortunately, the Nautilus developers don't make it terribly intuitive to switch between them.

The good news, there are fairly few adjustments to make to get to the compact view (at least in Ubuntu 9.10, for which this is written).

First, open Nautilus, go to the Edit menu, select Preferences.

Under the Behavior tab, deactivate "Always open in browser windows". This change does the bulk of the work.

Then under the Views tab, change the default view to "Compact View", and activate "Text beside icons".

As far as I can tell, the order of these changes doesn't really matter.

That's pretty much it!

The result:

Off the Shelf

2009-10-02T10:13:00.000-07:00

Among systems analysts it's commonly understood that if an off-the-shelf product meets the business need, it's faster to develop around that. Start-ups and small operations, in desperate need of revenue in this economy, hazard the failure of their operations if they don't consider development (or deployment) time when deciding technical strategy.

Being very technically minded, a student of both Computer Science and Management Information Systems (a business degree), I see and experience both sides of this problem. The systems analyst and systems administrator in me wants it fast, easy, reliable, and well-supported. The programmer and entrepreneur in me want the pride of having developed something on my own, intellectual property.

Open-source products have become particularly interesting to many firms in the last year, as they've had to seek cost savings in IT. (As any of my readers - if I have any - would know, I like that. FOSS is good.) This brings to mind two very important aspects:

They're seeking "off-the-shelf" products that work, not writing their own
They're trying to save money

Writing their own tools usually doesn't save money, since they have to pay the wages of whoever is developing them, and it usually takes longer. (The only exception here is when the off-the-shelf product is a bad fit and requires excessive integration effort.)

That's a problem for the little guy (start-up), because he (a) doesn't have time, and (b) doesn't have money. He needs to take advantage of available systems where it's fast and cheap/free.

"A 50%-good solution that people actually have solves more problems and survives longer than a 99% solution that nobody has because it’s in your lab where you’re endlessly polishing the damn thing." [Joel on Software]

(This relates to a situation I'm currently in, where I've repeatedly recommended this off-the-shelf attitude, but it's fallen on deaf ears. I'm quite frustrated by it, but hesitant to say anything directly since it's very slowly turning around.)

Google Apps FTW

2009-06-25T11:08:00.000-07:00

So I'm too busy to set up my own mail server on my home network. Not that it's particularly arduous, just that I have way too much to do anyway.

A good friend introduced me to Google Apps some time ago. At that time I was too busy to look into it, but recently I got tired of not seeing the output of my cronjobs at home. Thanks to Google, I now get a much better view of my home network's routine operations, with very little work, and no significant maintenance burden.

(You'll notice a theme there - I'm busy. That's why I rarely post here, I guess.)

So I've long maintained a dyndns.org account so that I can refer to my home server(s) by name. I looked into the configuration requirements of Google Apps the other day, and discovered that it's really quite simple:

Register with Google Apps
Add Google's mail exchangers as the MX record on my dyndns account(s)
Post a very small file at the root of my public-facing web server so Google knows its me.

et voila! My system logs, cronjobs, and other administrative info are reported via email in a much friendlier fashion.

Mind you, I've had my servers configured for quite some time to route mail through Google's SMTP system, since I was originally just passing it to my regular email account. I was missing a lot that way, but not anymore! I've now got accounts (and nicknames) for every system account I use (postmaster, cron, etc), and all collecting in one sensible location.

I look forward to using Google App's per-domain Docs, as well as Google Sites for this domain, to take notes on what I do for my home network.

This feels like a major improvement, and pretty light on the work-load.

ATI hardware support

2009-05-19T12:26:00.000-07:00

So recently we got some new hardware at the office. Some Dell workstations with dual dual-head video cards. Each workstation was to have 4 monitors.

The video cards had ATI RV630 [FireGL V3600] processors/etc.

The workstations were to run Ubuntu 9.04.

Using the proprietary driver, we had no trouble setting them up for single-monitor usage. We even managed to get 4 separate desktops running on each display, but there was absolutely no way to run them in a single-desktop arrangement across all displays. The open-source driver wouldn't generate a real image.

So that really sucked. I spent a couple days fiddling with them, trying all possible combinations I could think of or could find online. Nothing worked.

We then pulled the quad-head Nvidia Quadro 440 from my boss's workstation, plugged it in, and in less than 20 minutes had exactly the configuration I was hoping for.

That was the deciding point right there. I will never buy ATI for my Linux systems again.

What cannot be accomplished on ATI in days of investigation and trial/error, can be done simply in 20 minutes on Nvidia.

Upon further investigation, it turns out the ATI drivers for Windows aren't that much better. They just barely work, from what I hear (and have experienced, though some time ago). I can't help but wonder what prevents ATI from producing quality drivers - lack of funding? lack of talent? I can only speculate.

I guess ATI/AMD doesn't want my money (or that of anyone I advise). Oh well.

Thanks Nvidia for making my life easier!

Source Control

2009-02-27T08:56:00.000-08:00

I recently posted my kit of shell scripts to a publicly visible repository, github.com.

These scripts are mostly conveniences, wrapping more complex operations into simple commands, saving session data, automatically cleaning out old junk data, etc. They're not all fully implemented - some are just stubs for future projects - but as it is my life is made easier by them.

So far I'm quite pleased with GitHub. Its free, friendly, secure, and they have a good backup system with Amazon S3 - excellent for open-source projects. They also offer paid accounts for private repositories, which is great for contributing to friends' closed-source business projects.

I've recommended to a friend for his business, and it's turning out really well so far.

That friend isn't so much the technical type - he's certainly adept and has done his share of coding, but he's not an uber-geek like $(yours truly). He's more interested in running the business, as far as I can tell. In discussing his company's engineering needs, the value of a good source control system became apparent to me.

From a developer's point of view, it gives me a means of undoing mistakes or reviewing changes that introduce bugs, and provides a communication medium to identify and discuss parts of the code - this saves time! From a management point of view, the source control provides a means of securing and preserving intellectual property, gives some idea of a developer's productivity, and (on the basis of time saved) lowers development costs.

As always, a good backup system (such as GitHub has) also provides disaster recovery options. At the prices GitHub charges for their plans, I can't really justify not using such a service - it costs substantially less than doing it myself, and the benefits are outstanding!

I really wish they would have discussed this in my business and IS classes, and would have made more emphasis in my CS classes. Could likely have saved me some gray hairs through college too...

It Just Works

2009-01-14T21:13:00.000-08:00

I'm a software engineer. I make stuff work, it's my job. I get paid for it too, which is nice.

When I get home, I don't want to muck about with my laptop or most of the other systems on my network. I want them to "just work". I don't want to fix them every few months. They need to be reliable, simple, and low-maintenance.

That's why I use Ubuntu (among other linuxes). Simple, easy to use, low-maintenance.

My Windows machines rarely remain stable or responsive for more than 9 months, I find. My most recent installation is no exception - I installed it around August, and it's now showing its age. About half the time, it hangs somewhere in the boot process and never gets to a Windows login screen.

All of my hardware is a couple years old, cuz I've got more important things to pay for. I've got a Debian box that runs a 1ghz Celeron with 256mb of RAM. My Windows box has 1.7gb of RAM and a 2.4ghz hyperthreading Pentium IV. When Windows successfully boots, my Debian box consistently reaches its login screen a couple minutes sooner than my Windows box does.

After logging in, my Windows box takes (at best) 3.5 minutes to let me actually start any applications. When I try to start an application, it often doesn't ever render a window - its process appears to be running in Task Manager, but it's not listed in the Applications tab, and it's nonexistent in any other form. After two or three tries starting it, it eventually comes up... but only because I'm a persistent little prick.

Linux hasn't ever done that to me without an error message of some kind. No mysteries with Linux. It's reliable, it does what I want and need it to do, and usually without complaining. When it does complain, it's an easy fix - install this update, move this file - et voila.

That's why I use it, and that's why my work is to build Linux-based systems to serve the simpler desktop needs of the common user - email, web, media, productivity - all free, easy, and reliable.

Amen.

ShellQL

2008-10-22T15:39:00.000-07:00

This is a concept I've been working on, for a few hours per day, for about two days. Not a lot yet.

In writing a lot of BASH scripting over the last couple months, I've discovered that some of my code could be better expressed in SQL than pipes of sed, awk, cut, etc. Not that these tools aren't great... they're just not always the most effective solution.

With the classic UNIX tools, string parsing is awfully sensitive, and it can be awkward to throw data around between these tools and shell loops and such.

So I'm playing with a new idea. Shell Query Language.

It will simply be an SQL-like interface to common system tasks. At this time, my intent is to implement SELECT-like behaviors first, simply reading stuff from the system, not making changes. Preliminarily I'll also implement an 'exec' feature which simply passes commands to user's $SHELL.

It'll read files, directories, and the output of programs, and that will be the supported datasets for querying. Directories will be recursive by default, and their queries can be limited using 'depth', 'mindepth', and 'maxdepth' features.

Am I re-implementing 'ls', 'find', and so many other tools? Yeah it seems I am. Unfortunately I haven't work out a convenient way to translate SQL-like expressions into the the respective calls to these and other common utilities. Ideally I'd rely on them, but I'm not sure yet how.

Originally, 'bashql' was a syntax for polling data from 'ls' and other programs that produced tabular output, as well as file sources, and pushing them through 'awk', 'sed', 'join', 'cut' and so forth, extracting only the desired information. This was to emulate SQL:

select name from /home/user where size=2k #implicit depth=0, only select files/directories in the specified dir
select fullpath from /home where depth=5; # all files and directories under /home where maxdepth=5 and mindepth=5 (exactly 5 steps down from /home)

As of this morning, I've realized the most effective route here is to implement a shell wrapper at a lower level. BASH simply isn't effective for parsing and manipulating commands at the degree I have in mind - while BASH itself is quite effective for its own purposes, writing a parser in BASH is proving quite difficult.

This next attempt will be to build a high-level 'SQL-like' parser in Flex and Bison.

A "table" in this context can be one of the following:

The output of a program, e.g. 'ps'
A filesystem
File content (of one or multiple files)

The data must be interpretable as some sort of table structure. Text-based tabular data is commonly deliniated via tabs, spaces, and commas. Some data types have complex records, which are expressed over multiple lines and deliniated via other means.

An example of complex data would be a set of Debian packages, *.deb: each one represents a record, its fields expressed in the control file (accessible via dpkg-deb).

Shellql will support abstract tabular interpretation functions. Built-in interpreters will support the base filesystem through system calls, and interpretation of text-file content (tab-, space-, and comma-separated).

To see all available interpreters, SHOW INTERPRETERS would produce something like:

TblType(dtype arg [,...]) Description
Directory(path rootdir) Recursively examine all file and directory nodes beneath the specified rootdir
File(path file, char** IFS) Read the specified file, use strings/characters from IFS to split the fields
Command(string command, char** IFS, path shell = $SHELL) Executes the provided command in a shell, defaulting to $SHELL from the user's environment

TblType(dtype arg [,...])	Description
Directory(path rootdir)	Recursively examine all file and directory nodes beneath the specified rootdir
File(path file, char** IFS)	Read the specified file, use strings/characters from IFS to split the fields
Command(string command, char** IFS, path shell = $SHELL)	Executes the provided command in a shell, defaulting to $SHELL from the user's environment

* IFS represents a set of strings to use to separate fields

To see all available tables, SHOW TABLES would produce something like:

Alias Type Arguments
/home/user Directory path="/home/user"
Processes Command command="ps";IFS=" ";shell="/usr/bin/bash"

Alias	Type	Arguments
/home/user	Directory	path="/home/user"
Processes	Command	command="ps";IFS=" ";shell="/usr/bin/bash"

Install Windows XP to USB

2008-09-22T22:39:00.000-07:00

I'm replicating another person's work in this post, as the original draft seemed to be offline at time of posting. (It later became available again.)

Linked above, <http://www.ngine.de/article/id/8> is the original source of the work by Emanuel Schleussinger.

I've revised it somewhat, hoping to improve its clarity and usability. My draft: <http://docs.google.com/Doc?id=ddv4s552_152c78p44dz>

Moved

2008-07-15T16:19:00.000-07:00

So a few weeks ago I quit my job and moved to Spokane.

The place we got is great. 2000 square feet, 2 bedrooms, one bath, and lots of living space. A covered garage, and a nice yard with lots of trees. The streets are lined with trees too, an element of which I'm especially fond. Oh, and it's not a dump like our last place - this one is basically fully carpeted, and it's in good condition, and the paint isn't hanging off the walls!

Got a new job too, which I'm totally appreciating. My last one had some serious drawbacks. This new one is definitely a learning curve though - the subject matter isn't foreign, but the office dynamics and approach to developing solutions is significantly different.

The subject matter is what's most interesting though - developing management tools for Linux - right up my alley. ... I'll update this later, gotta get back to work though ;)

iPod tools

2008-07-10T09:22:00.000-07:00

So I finally caught up with the times: thanks to the generosity of my in-laws, I now have an iPod.

Only I don't like that iTunes restricts my capacity to share music with my wife. I can't copy music from my iPod to my computer and then to her iPod. Oh, and iTunes doesn't play nice with converting my collection of videos for some reason.

So, here are two tools to help with that:

SharePod - lets me manage my music as I please, regardless of iTunes restrictions
Handbrake - lets me convert DVDs and other video formats to iPod-compatible format.

Handbrake is open-source and runs nicely on Linux, Windows, and Mac OS X.

SharePod is a propietary Windows-based product. I would use `gtkpod` in Linux for this functionality, but its support for 5th generation iPods is limited. SharePod is marginally better, though I've observe a few quirks I'm not entirely fond of.

I'll keep this updated as I find new tools that better address the problem(s).

Switching to Debian

2008-06-13T10:36:00.000-07:00

Eh, so Ubuntu is great on my laptop.

Not so great on a lower-end workstation I have at home. It's a 1ghz Celeron with 384mb of RAM. Xubuntu ran pretty well on it, but I got tired of upgrading it every 6 months. So I switched it to Debian Lenny. Seems it was a pretty good move so far.

I still run XFCE on it. XFCE is great on this system. It's light enough that the system runs reasonably smooth, and yet featureful enough that my wife can navigate it fine. She uses the "guest" account on it though, which automatically logs in after 30 seconds, because it's easy.

I, of course, customize the interface extensively. These are screenshots of my preferred desktop.

First, the desktop itself. The top panel: XFCE menu, task list (web browser minimized), desktop switcher, the drive mounter, the volume control, the system tray, the clock, and window list (for all desktops).

The icon dock with a launcher menu, the notepad, icon box, and system meters (network, CPU, RAM, swap).

And the file manager, Thunar. Nothing too special here. Thunar + associated plugins is a very functional file manager.

The GIMP, which I was using to make these screenshots. (Hadn't discovered the screenshot applet yet.) I also switch to using the small icons theme in GIMP, and group it into a single panel, which I haven't done in this screenshot yet.

The XFCE menu expanded, notepad with transparency.

Three terminals demonstrating transparency and the iconbox in the lower panel.

Consolidated panels. I moved the system monitors and system tray up. Although you can't see it in this shot, there's an auto-hiding panel in the lower-right that contains the icon box listing all windows from all desktops.

Setting up Gedit for Coding

2008-05-15T10:07:00.000-07:00

So I've become quite accustomed to using 'vim' for my coding. It's very flexible with its syntax highlighting features. So far I haven't found any means of autocompletion or code/class/symbol listing - these would be nice features, but I also don't have a real recommendation for how to implement it in 'vim'. [Update: apparently there's a feature, "omnifunc" which does this: http://amix.dk/blog/viewEntry/19021 - I was wrong.]

I've therefore started experimenting with Gedit again. I've used it a lot in the past, before I picked up 'vim'. It turns out there are a number of plugins and configuration settings that make it quite useful as a programming environment.

Basic configuration is not so complicated: enable most everything in the gedit preferences.

I sometimes print my code too, for debugging or auditing purposes, so here's my setup...

I also enable most of the default plugins (default in Ubuntu anyway):

Change Case
Document Statistics
File Browser Pane
Indent Lines
Insert Date/Time
Modelines
Spell Checker (though this isn't always useful in coding)
Tag List
Text Encryption

I then download and install the following 3rd party plugins

Note that these also have some package dependencies so install them:

sudo apt-get install tidy exuberant-ctags

More plugins can be found at the Gedit Plugin page.

Then just enable the Side Pane under the View menu (or press F9) to use the various navigation tools. I've found the File Browser particularly useful, as it gives me a friendly interface to manage the project hierarchy.

For fun, a screenshot of this in use:

OOP: Object Oriented Paradigm

2008-04-28T17:42:00.000-07:00

OK, so in my adventures of building content management systems, particularly in the templating aspects, I've realized a few things...

We (most humans) think in terms of objects (things) and behaviors, pages and reports; not in terms of XML, regular expressions, formatted strings, blocks, nodes, or queries.
When I'm going to use a system I build, I want it to allow quick creation and modification of a website. It needs to be intuitive and natural.

Right now I'm in the middle of trimming down the functions of my template handler (again). I'm using PHP's DOM support to handle the documents, which provides me XPath support. I've discovered I like XPath a lot. I've also built a CSS to XPath translator, though in my own work I'm staying with XPath - the translator might be handy for other things, so I'm keeping it. XPath is slick though.

I'd like to explore a different paradigm of all this now. ... Much of this is written as a narrative, so bear with me, as it's still a mere vision.

I'm going to experiment with a paradigm of object-orientation in my templating system. The following expresses (in english) how I'd like to be able to create a website, simply and sensibly...

This is my website. It's based in this folder (e.g. /var/www/mysite). It's base title is "Sam's Website".

Each page on my site has a:
Title
Tagline
Description (and other meta)
Heading
Optional Subheading
Main navigation
Main Content zone
Paragraphs
Links
Images
Stylesheet
My website can also have the following kinds of (abstract) things on it (referring to modules):
Tables (extended for usability)
Forms (extended for usability)
Data views (based on tables and database queries)
Contact information (based on abstract tables)
Contact forms (based on abstract forms)

I tell it what things (in human terms) I want on my website, and how to handle them, and how they interact. Each thing is handled by a module. This makes it easier and faster to create a site because it lets me express myself in my natural thought. Then I tell it what kind of navigation I want: tags or a traditional hierarchy.

After I specify what I want on my site, it builds a template for me. It presents me with an editable view of the template, where I can adjust its look and feel. I also have a panel where I can edit attached stylesheets, and link element classes (CSS) to behaviors (JavaScript). When I'm done, I save the template and it takes me back to an administrative panel.

On the administrative panel, I have a tabbed interface.

On the 'General' tab, it presents notable entries from the website's internal log, including any errors that might have occurred since I last checked in, as well as customizable summary information. This summary may tell me how many users are registered, or how frequently content is customized - I can change this summary to fit the nature of my site. This page also provides a 'Publish' link to users in the 'Publishers' group. (See "Publishing..." down below)
On the 'Navigation' tab, I am given an overview of my site's structure. Selecting a page provides a menu, including 'rename', 'edit' and 'delete' options.

If I use a traditional hierarchy, it shows me the hierarchy. Drag-and-drop moves pages and directories/sections around in the hierarchy.
If I use a tagged navigation system, it shows me all my tags, and lists all tagged pages under their respective (multiple) tags. Drag-and-drop assigns another tag to a page.

On the 'Page Creator' tab, I get a WYSIWYG editor with a blank page. I can choose to open an existing page, or I can create a new one from the blank. I select a template to use in the page. It loads a menu of objects that I can use in a panel on the right. I drag and drop objects as desired, and then customize them. Each object can be assigned a class, as well as additional behavior specifications. I can drag-and-drop objects already on my page into new locations on the page.
On the 'Security' tab, I get a list of groups, and users belonging to each group (as well as groups within groups). I can create new users, edit existing users, and arbitrarily restructure the user-security features of the site. I also get a view of my site with security information attached, so I can control who sees what, who can edit what, etc.
On the 'Objects' tab, I see a list of objects available to me, as well as some of their implementations (if they have any). It searches a folder for object files (class.*.php), so I can add new ones just by uploading them to my site.

Publishing...
This system also differentiates between published and unpublished content. Pages are initially created in the 'unpublished' zone. All revisions occur in the 'unpublished' zone. This zone can be navigated just like its 'published' counterpart, to ease revision and verification of content. Each page is also marked with a revision date, so pages are only updated if they are revised (or if their templates change). Unpublished drafts can be locked until they are published. Authorized users can choose to publish the current draft of the site, which "renders" the entirely dynamic 'unpublished' pages into static content (except where specifically dictated as dynamic, such as live database queries). The rendered content is placed in the 'published' zone.

Anonymous visitors to the site see only the 'published' content. Each page in the 'published' zone provides a log-in link. When a user logs in, if they are authorized to customize content they are redirected from the published page to its equivalent in the 'unpublished' zone. For each object they have permission to edit, they are given a link to edit it. Clicking on this page loads an inline editing tool. When they save it, the 'unpublished' page is automatically updated.

The 'published' zone is rendered as largely static content because it's faster on the server end. Lower overhead processing is key for high-volume sites, so I want to cater to that.

I want to create a really intuitive, natural way of creating and maintaining a website. Anyone who reads this and likes the idea, and wants to contribute, let me know -- would appreciate the help ;)

What Microsoft Tax?

2007-12-08T21:20:00.000-08:00

After some discussion of purchasing vs building computers, with friends and family, I decided to do some research on the costs of computers bundled with Microsoft software, vs those with Ubuntu Linux bundled, from two well-established vendors. This post details the research in something like proper form, though it's been so long since I've had to write in this manner that I may have neglected some element of this writing style.

Abstract

This research examines the contrast of pricing between two vendors' similar hardware configurations, and the effect of a customer's software preferences on that pricing.

Methods

The vendors compared in this case were System76 and Dell. The systems compared were System76's Gazelle series Laptop, and the Dell 1420 laptop, with it's Ubuntu variant, the 1420N.

This comparison examines the following configurations:

Dell Inspiron 1420N with Ubuntu (basic)
System76 Gazelle (basic)
Dell Inspiron 1420 with Windows Vista (upgraded)
System76 Gazelle (upgraded)

Configurations listed as "basic" are slightly upgraded from the least expensive offerings, in the interest of examining realistic, long-term computing solutions for real-world users. Configurations listed as "upgraded" reflect hardware capacity upgrades required to support Windows Vista; the System76 option is similarly upgraded to maintain the validity of this comparison.

Microsoft Windows operating system also requires additional software to make it usable to real-world customers, including anti-virus protection and an office suite. Ubuntu Linux comes with most of these requirements already addressed, either by their being practically irrelevant (as in the case of anti-virus software), or by bundling the software with the base installation (as in the case of an office suite). Remaining needs of each customer are not included in this study, though can be addressed at no monetary cost using Ubuntu's software installation mechanisms, or at notable monetary cost by purchasing more software for a Windows system.

Findings

[The detailed findings in tabular format have been removed from this post because Blogger didn't play nice with the table. I've republished this via Google Documents, and the original table can be reviewed here. Continuing with the original document...]

The following points of variation identify the contrast between the "basic" configurations.

Dell runs at 83.3% of the processing speed of System76
Dell runs at 112.5% of the display resolution of System76
Dell does not support DVD writing, while System76 does
Dell prices at 98.6% of System76's price

The following points of variation identify the contrast between the "upgraded" configurations.

Dell runs at 83.3% of the processing speed of System76
Dell provides 50% of the Video Memory of System76
Dell runs at 112.5% of the display resolution of System76
Dell (with Windows software) prices at 115.76% of System76's price

Conclusion

The cost of a comparable systems running the same software from these two vendors is similar, however System76 clearly leads the market on the quality/capacity of hardware delivered per customer dollar. Adding Microsoft software, as in the upgraded systems' case, seems to increase the price by roughly 16%. The utility of this 16% depends on the use case of the system, and must be contrasted with the ongoing maintenance and support costs of each alternative.

Of Templates And Parsing

2007-09-27T19:08:00.000-07:00

Long story short, I've rewritten my template parser again. That's the... 6th time? Something like that. The last 2 or 3 drafts have all used file-based data sources, rather than databases. At this point, that's a VERY good thing [in the context of this project]. Maintaining the DB wasn't so great a use of my time at the early stages.

My last draft rocked. It did everything I needed. It also was intended to support additional modules. That didn't work out so well. It also had a rather boorish built-in debugging mechanism. That also had some issues. It was bulky and slow, but still, it rocked.

I learned from that. Don't build a swiss army knife if you're neither swiss nor in the army.

So I rewrote it from scratch. As of now, that has proven to be an absolutely wonderful idea. The code is now 1/4 of the size of the original, and it's faster and a lot cleaner.

A couple months ago I also wrote an isolated debugger. That worked out pretty well, but it was having serious problems with my template handler. They didn't play nice. ... Fixing that is my next project.

Open Source Backup Script?

2007-09-27T13:16:00.000-07:00

Since this time last year, I've on-and-off been working on a script to a provide convenient, flexible, and robust snapshot backup mechanism for the servers I maintain.

Because I run Gentoo Linux on the servers, I initially modeled my script after existing "stage4" backup scripts. Because I maintain a rather diverse set of servers, I needed to be able to automate snapshot creation in a manner that was unique to each server, but administered quite consistently. I also needed to be able to store the resulting snapshots in an easily recoverable manner.

The resulting script is, in my opinion, very flexible by using saved configuration profiles, and very easy to use, with some degree of user-friendliness (e.g. colored/formatted text on the CLI, unless passed the "silent" argument).

The following syntax expresses its general functionality:

sudo stage4 archive systemprofile

Creates a 'squashfs' compressed archive file named according to the syntax specified in /etc/stage4/systemprofile/profile.conf (profile configuration)

sudo stage4 livecd systemprofile

Creates an 'iso9660' CD image file, based on the image specified in the profile configuration, which includes the "stage4 archive", and provides a (fairly) convenient way to restore/replicate a system from a backup snapshot

sudo stage4 mount /path/to/archive.squashfs /mnt/mountpoint

Mount the archive file to a specified mount point, after optional MD5 verification

sudo stage4 restore /mnt/mounted/archive /mnt/destination/filesystem

Copies the contents of the archive from its mountpoint to the target filesystem after verifying a few things...

sudo stage4 verify /path/to/archive.squashfs

Verify the "stage4 archive" using MD5 checksum comparison

So... I'm contemplating releasing it as a GPL-licensed script. It's written in BASH, so I'm tempted to think it's just silly to release it as such. I don't think it's that trivial though: the script is 25K in size, I find it very useful, and although there are lighter alternatives, it's proven to be very valuable functionality for my systems and organizations. I'd love to share it with everyone.

I just need a good way to do that. I can't seem to attach files to blog posts, and I don't have any consistent and readily available online file storage. I'd also like some advice from people who have prior experience with this sort of decision. ... Would love to hear from you if you read this.

Developing Comedy

2007-09-04T13:45:00.000-07:00

These are various quotes and comics I've found online, from `fortune`, or have heard in my (developing) career as a developer. (... department of redundancy department.) I'll continue updating it as I find more.

Some sources are cited, but not all. I'm not taking credit for them.

"Infinite Loops... yeah, those usually take somewhat longer than average."
"You generally don't want to return pointers to things that might not be there."

The following of XKCD

http://xkcd.com/138/
http://xkcd.com/227/
http://xkcd.com/303/
http://xkcd.com/273/
http://xkcd.com/149/
http://xkcd.com/287/
http://xkcd.com/278/
http://xkcd.com/256/

Application Development Observations

2007-01-18T17:54:00.000-08:00

I've been working on this template system as part of an on-going project at work, and invested some of my own time into it because I want to keep parts of it as my own IP later on. As one might note in previous posts (I don't know if I've published all of them), I've gone through several different paradigm shifts in how this application should be developed and what its underlying framework(s) should be.

This time, I had to employ a little patience, but I managed to step back and work on documentation from the get-go, describing ideas first, with a partial focus on user experience, rather than trying to move ahead on code immediately. Turns out that was a great idea...

First off, I needed to identify a few goals for the project:

Maintain maximal compliance with standards (i.e. XHTML, CSS)
Content should be simple to implement (convert from other systems etc)
Content should be manageable using 3rd party applications like Dreamweaver
System should be built to support up- and down-scaling of the application requirements

Facilitate core functionality with absolutely NO module requirements, NO proprietary software (other than template-handling kernel)

System should support development of proprietary modules

Module structure requirements should be minimal
Modules should be able to interact with the kernel and with each other

Kernel will be highly optimized, very efficient and fast

So yeah, I drafted a few examples of common scenarios where this kind of thing could be useful to a developer, and walked through how I figure the developer would prefer to deal with this kind of thing. (I suppose I can do this with relative ease, since I am a developer myself.)

The really great thing: planning has paid off. After a few hours of planning user experience, I started brain-storming on how to implement them most efficiently. Result: this is going to be one of the simplest meta-applications I've ever written, since its concepts are already laid out (textually, pseudocode) in great detail. The administration tools will simply act as a module to the template kernel, and link into its processing at a certain stage to implement the changes it needs. No need for doubling my efforts by creating an entirely separate interface.

I've experienced the same effect in regards to the module loading functionality -- planning and careful consideration has made this REALLY REALLY easy, as compared to previous attempts. ...

Rock on ;)

Open Source Halloween

2006-11-01T16:50:00.000-08:00

Right, so generally I don't take Halloween, Thanksgiving, and most other holidays seriously. Not even birthdays. Dunno why, they just never seemed all that useful to me. Go figure...

My wife is totally the opposite. She has lots of nostalgia for family gatherings and fun times around the afore mentioned days. In compliance with her desires, I (without too much resistance) give in to the whole ceremonial pumpkin carving business. Because of my disdain for the historic significance of Halloween, being a day of the dead and all, I figured I'd lighten things up a bit...

So I threw in a little of the Open Source spice to keep it interesting to those who, like me, take no interest in Halloween as a significant observance.

First off, there's Tux...

And the Ubuntu logo ;)

Next year I'll probably take on the Gentoo logo, but given its geometry, it becomes somewhat difficult to effectively carve into a pumpkin without supporting structure (kinda like Tux's head extension above).

Software Switch

2006-10-28T23:47:00.000-07:00

It's become apparent that Linux is truly evolving to accommodate the needs of regular users, as well as tech-heads.

A mildly tech-savvy friend of mine recently toasted his system using Windows to repartition a disk, and wanted help reinstalling. I mentioned a few alternatives for recovering it, and noted that Linux is useful in fixing Windows' various problems. He thought about that for a moment and asked me about trying Linux as an every-day OS, since he had never touched it before. I sent him a link to the Ubuntu site, recommended that he download the i386 version, and he took off with it, no trouble.

My mom is a quality control analyst for a food processing firm, and my father owns and operates a small business. I set up a workstation at their home using Ubuntu Linux, configured their email client, and left for a few weeks. They're totally satisfied, and generally more comfortable in Linux, actually, than they were with their Windows alternative.

My 8-year-old brother installed Ubuntu single-handedly, asking me only two questions: which timezone should he select, and which keyboard mapping.

The more I use it, personally, the less I want Windows. Linux gives me more control, is more reliable, and doesn't bog down the way Windows does over a 6-month period. It's simply a smarter system.

With the release of Ubuntu 6.10 last week, I'm really quite impressed at how convenient, polished, and generally friendly everything is. This is a serious desktop OS, no question about it. It even runs quite tolerably on this eMachine I got for free - 1.4ghz Celeron, 256mb of SDRAM, piece of junk. [I'm using it now to post this message.]

As for a server operating system, this issue has been settled for a long time. My Gentoo server emails me daily updates of all its activities and network traffic, security and system health diagnostics, virus scans, backup and synchronization reports, weekly software update reports, individual reports of each package as they're installed. That thing is solid as a rock and just works. It took some time to configure, mind you, but after a couple hours of editing text files, this thing beats the pants of any Microsoft server I've ever seen.

I'm convinced. At this juncture, Microsoft has a lot of work to do before I'll be spending more $$$ on their software.

CMS Development Continued

2006-08-01T23:01:00.000-07:00

Aye, the last couple months have been focused largely on the development of my Content Management System. It's become progressively clearer that the demands of regulating parties require us to use a more dynamic template-handling system than previously expected. This time it's going to revolve primarily around XML parsing, so that both the page and the template can be handled as separate XML documents, and then combined. Using this method allows the pages to be edited separately by numerous web development tools, most notably Dreamweaver and other WYSIWYG editors, without requiring any special or proprietary syntax -- it's all standards-compliant! (YAY!)

[UPDATE] This project has completed a functional prototype as of September 12th, 2006. Development will continue in pursuit of optimization of the code, modularity of its component functions, and planning for future implementation options.

Ubuntu

2006-06-13T23:26:00.000-07:00

Well, I finally got around to wiping Windows off my hard disk and put Ubuntu 6.06 Linux on it. It's wonderful! Now I can do all my web development work on my laptop without needing an internet connection at all times! And I still get the back-up functionality of a synchronization system similar to what WinSCP afforded me, but now it's via Unison on a 15-minute Cron job. Whew! I'm finally Microsoft-free, and I love it!

CMS Development

2006-06-13T20:59:00.000-07:00

Of late, I've been developing a simplified Content Management System. My first attempt at it stored all the content in a MySQL database. This has some advantages, but I found that I wanted more functionality than what my data relationships and structures allowed. I tried to restructure things a bit, and determined that I would have to clear the drawing boards completely in order to accomplish what I desired. As a result, I abandoned MySQL in favor of a proprietary means of storing and retrieving the necessary data relationships.

At this point, I'm rewriting the entire CMS engine to pull data from files, directly on the file system, without the need for any external database engine. When complex data structures are needed (such as those which I attempted to build using MySQL), they are handled using a fairly simple adaptation of the PHP serialization function. This adaptation provides a means of encrypting the data, and storing it securely (such that it cannot be arbitrarily downloaded by users). ~~As of June 10 2005, a demonstration can be seen here~~.

[UPDATE] Due to unforeseen complications in the initial draft of the CMS project, multiple alternative template engines have been developed. The current (and simplest, and thus fastest) version does not require database access, and uses filesystem storage to provide easy access to HTML content. Also, this method implements a parsing engine which does not use customized (read: proprietary) X/HTML tags to identify content and regions and such; instead, it uses a Regular Expression to identify standardized tags in a manner which allows WYSIWYG editors (such as FCKEditor) to handle the code more effectively.

In this project, I'm also attempting to create an adaptable development framework, combining a variety of useful works by other developers. These include a tool which assigns JavaScript behaviors to DOM objects, some Asynchronous Javascript and XML (AJAX) scripts which I've been adapting, ~~and a WYSIWYG editor similar to the well-reputed FCKEditor (careful spelling)~~. I'm currently looking into a browser-based syntax-highlighting code editor, similar to Helene or About:Edit. ~~I'm almost prepared to write my own, since that which already exists in the market has so many bugs, or isn't open-source.~~ I'd like to find one that was open-source and fairly robust, so I don't have to re-invent the wheel.