The Corrosion of Aaron Stone

In any project, feature-creep can be a problem. In an open source project, this can be particularly acute when there are few developers. Worse yet, someone becomes super-active just until their major new feature lands and then they disappear! And then there’s the problem of noisy people who won’t stop whining about getting some peculiar feature, or some major new functionality, either or both of which are just not interesting to anybody else.

An effective plugin strategy actually embraces these people and their pet features, providing a mechanism for decoupling code from the code project, yet allowing key features to be injected into the running system from a separate and contained plugins area. Note the words separate and contained. Both are very important.

The plugins area must be separate because you want to protect the clear messaging of your application’s features. You also want to keep plugins separate because there can be a tendency to make everything a plugin. Don’t. Just don’t. Really. Core features are core. They are not plugins. Nobody wants a framework that does nothing until you’ve loaded a jillion plugins into it in order to create a working application. So separate is a double-edged sword: keep the non-essential stuff away from the core applications, and keep all the essential stuff inside the core application. Sometimes it means swallowing a plugin into the app. If it makes sense, do it.

The plugins must be contained. And, ideally, also self-contained. That is, you have a directory like ‘/your/app/plugins/some_crappy_plugin’ for each plugin. The plugins are contained within ‘/your/app/plugins’ and each one is further self-contained another directory level below that. Then, provide an API that allows the plugins to act at a distance. Yes, act at a distance. Normally this is something that you don’t want because it is hard to figure out. But in the case of plugins, it is just right. The plugin does not need to patch into the main app code, but rather register itself with the main app and declare which of its functions should be called from which parts of the main app. It’s runtime integration, runtime configurability, and runtime enable/disable. Sure it can be slower. But it’s so much better than having people distributing patches that implement their functionality by hacking up your beautiful code.

So I’ve run into a problem with mod_perlite, and it’s that I cannot just override Perl’s exit function with a straight perl_destruct call, a longjmp back to the Apache handler, or anything else at all.

• Here’s how it’s handled in mod_perl.
• And then there’s this possible workaround using goto.
• Shockingly, this works, too.

Here’s my favorite…

bar.pl:

 exit;

Foo.pm:

package Foo;

sub new {
return bless { };
}

sub DESTROY {
print "Foo is destroyed\n";
}

1;

foo.pl:

use Foo;

my $foo = Foo->new;

print "I have a foo!\n";

BEGIN { *CORE::GLOBAL::exit = sub { goto EXIT; }; }

eval {
bar("bar.pl");
}; if ($@) {
print "$@ happens\n";
}

print "And I'm still here\n";

sub bar {
eval require $_[0];
EXIT:
print "Exiting!\n";
goto REALLY_EXIT;
}

REALLY_EXIT:

Yep, that actually works!

I have a foo!
Exiting!
Foo is destroyed.

Once upon a time, I worked* on an open source groupware application called TWIG, The Web Information Gateway. I started posting fixes, wrote some code to scratch my itches, and pretty soon I got CVS commit access. I wrote a new module for scheduling meetings, and the guys on the development team loved it, so I committed it, and life was good. Pretty soon, I was using this module to schedule meetings in rooms that got to be conflicting and overlapped. I also had heard that back at my high school, there was a particular teacher who thought she was the Goddess-of-all-Writing and therefore could simply walk up to the whiteboard in the main computer lab and erase other teachers from the schedule in order to accommodate her clearly superior, holy and blessed classes.

So I wrote a reservations module that integrated into this groupware platform, and it worked, and I got positive feedback from the folks back at my old high school, folks on the twig users mailing list, and from the development team, so I committed it.

The maintainer flipped out. “NO MORE MODULES!” he said, “REMOVE THIS OR I WILL REVOKE YOUR CVS ACCESS!”

Other developers and users came to my defense and said, “But this module is useful, it’s excellent, it complements the scheduling and meetings features. It’s killer!”

“DIE MODULES, DIE!” the maintainer said.

The project died.

Actually, I still use TWIG personally, and I still hack on it when I have time, and every now and then there are new users who post to the mailing list, and I generally respond to help them out within a few hours. But the project is effectively dead.

This month a number of Sieve extensions became published RFCs, along with an update to the Sieve base spec itself:

• [RFC 5228] Sieve: An Email Filtering Language.
• [RFC 5229] Sieve Email Filtering: Variables Extension.
• [RFC 5230] Sieve Email Filtering: Vacation Extension.
• [RFC 5231] Sieve Email Filtering: Relational Extension.
• [RFC 5232] Sieve Email Filtering: Imap4flags Extension.
• [RFC 5233] Sieve Email Filtering: Subaddress Extension.
• [RFC 5235] Sieve Email Filtering: Spamtest and Virustest Extensions.

Time for teh libSieve hacken!

For the first time in a long time, I spent a weekend hacking on TWIG! Tons of things work now in TWIG 4, lots of code flow improvements, CSS and Javascript improvements for Web 2.0 (more on that in a minute), database cleanliness and efficiency improvements, Reservations feature checked back in for the first time in a very long time, HelpDesk works, and the guts of a totally new Mail backend based on the IlohaMail IMAP library.

Back when the world was about separation content from presentation, TWIG 2 was all about mixing them together in the early tradition weekend-hack PHP apps, despite TWIG being a very large and otherwise well organized framework. Now that we’re in Web 2.0 days, the point at which content mixes with presentation to form an output web page has moved from being entirely on the server to being partially on the server and partially on the client. This is the Ajax way.

When I first started working on TWIG 4, I thought I would need a structural mechanism to allow data to be mixed into the source HTML or to be retrieved via Ajax. But it’s been on the table for so long now that Web 2.0 caught up with us and every reason for not relying on Javascript — old browsers, mobile devices, screen readers — now support at least basic Ajax functionality. With this now nearly universal support for running Javascript and requesting data via Ajax, I’ve now worked it into TWIG 4 as a requirement and a basic part of the architecture.

Next time, writing your own mashups using the TWIG query API and native Wiki feature!

In http://www.dbmail.org/mantis/view.php?id=655, Mr. Maenaka writes:

MySQL’s automatic reconnection is unsafe because of the following reason.First, SET NAMES query is a required arbitration between server and client if both’s character encoding is different. This should be done at the time of connection open. (Of course you can issue SET NAMES between every query with the huge overhead though.) By the way, MySQL’s automatic reconnection is made transparent to the client. So the client never know that SET NAMES should be issued again. Therefore, if the connection is restored this way, the character encoding mismatch may occur (and some or all data is garbled or lost). 

No, really, seriously, you auto-reconnect without setting my connection-specific settings in the new connection? What kind of crack are you smoking at MySQL AB!? 

A problem I’ve been thinking about for a long time was how to build a backend service for a stateless web frontend. All of the solutions I have seen involve building a daemon that lives on the server and holds the persistent data. Or, in the case of PHP, weird ass service side session cookie things built right into the language.

The solution I want is a hack that is as full featured as building a daemon and does not require modifying the operating environment. And today it hit me: the two request monty.

Here’s the setup: you have a script that’s running in the web server, it has an execution time limit, you’re allowed to create and listen on sockets. Yes, I’m thinking of PHP safe mode here. To create the service, your web app makes a request back to the host web server, spinning up the service script. That script then creates a socket listening to some local port, tries to increase its max execution time, sets a timer for just under whatever that time limit is, writes this information to some common location (typically the database behind your web app) then begins listening on the socket for requests for whatever kind of persistent data you need. When the timer fires, the script makes a request back to the host web server, spinning up a new service script. The new service spins up on some other port, connects to the first service and grabs all of its data, then writes its information to the database and begins serving persistent data to your application.

Total hack beyond hack, but satisfies the requirements: does not require a new daemon, does not require new modules on the server, runs in safe mode, runs with time limits. 

Lately at work a few folks have been batting around possible solutions to the perceived problem of how hard it is to run Perl code from Apache. Of course there are very good solutions, mod_cgi, mod_fastcgi, and best of all mod_perl. But there are very good reasons why web hosts shy away from CGI, and why they’re deathly afraid of FastCGI and mod_perl. In a nutshell, it all comes down to persistence. PHP is incredibly popular because there is no persistence!

 Byrne Reese had the idea of taking mod_php and ripping out the calls to the PHP interpreter, replacing them with a Perl interpreter. “I don’t think it’s quite that simple…” I said. But Byrne was 80% correct. The other 20% is grabbing the Apache - PerlIO layer from mod_perl and using that to shovel data from STDIN and STDOUT in and out of the Perl script. The resulting code is pretty simple!

 mod_perlite is over at the Six Apart open source code repository, and I’ve been working on it off and on for just a few days. It builds, loads into Apache extremely simply, and returns “Just Another Perl Hacker” anytime you try to get a page that ends in “.pl”

 Fundamentally, mod_perlite tries not to solve all your problems. It is specifically targeted at being 80% good at the 80% problem. With luck, we’ll be able to get it onto 64% of web servers like PHP and pals.

 To Do:

• Thrash at the Apache - PerlIO interface some more.
• Develop a script caching model (ala Zend Accelerator or APC).
• Add a script run-timer to kill long-running scripts (ala PHP’s max_execution_time limit).

The crux of my SSL management headache is that I have a lot of domains, but only four public IP addresses. I bet that most home-operated sites are like that. I also have a number of different applications running — HTTPS, SMTPS, and IMAPS in particular. For each HTTPS domain, I need to have an SSL key that is bound to that domain. But since most of the domains are virtual hosts on a single IP, I don’t know the domain until after the SSL negotiation. TLS is supposed to solve some of this, I think, but there’s enough SSL out there that I need to deal with it. A tool that can tell me that I have { X } number of HTTPS domains but only { X, n > 0 : X-n } IPs, and allow me to pick which ones get keys, then generate the keys for me, would be grand! Furthermore, I often create convenience sub-domains for particular applications to facilitate portable DNS inside and outside of my home firewall. smtp.serendipity.cx, for example, resolves differently inside and outside my firewall. Outside the firewall, it resolves the same as serendipity.cx. Inside the firewall, it does not, since I don’t NAT my servers from the inside. So I need separate SSL keys for these two domains. But they’re also different apps, so there’s no IP conflict. Am I making sense? Yeah, so I need an open source SSL key management app. Or I need to write one

Well, it finally happened. I caved. I bought Internets everywhere with a Sprint plan and the Novatel USB727 doohickey. Bonus points to Novatel for packing a GPS and a EVDO rev A modem into such a small package and providing working Linux instructions involving only standard kernel serial drivers!

My critique of the instructions is also a strength: they’re looong, repeated several times for various distributions, but also totally idiot-proofed. At the very end, the steps to quickly get the modem dialing from the command line are presented.

Instruction for using the Sprint U727 in Linux, (and very likely the Verizon USB727, too). Here’s what works for me:

• First, add this line to /etc/modprobe.conf:options usbserial vendor=0×1410 product=0×4100
• Create /etc/wvdial.conf with these contents:
  
[Dialer Defaults]
Modem = /dev/ttyUSB0
Baud = 460800
Init1 = ATZ
Init2 = ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0
ISDN = 0
Modem Type = USB Modem
Phone = #777
Username = ''
Password = ''
Carrier Check = no
Stupid Mode = yes
• Then, each time you want to use the modem:

1.  Insert the USB modem.
2. Wait for the device to be recognized as a cdrom.
3. eject /dev/cdrom(since I don’t have any other cdrom - you might need to eject /dev/cdrom1 or something)
4. Wait a beat.
5. wvdial
6. route add default ppp0