Why Sponsor Oils? | blog | oilshell.org

Comments on Scripting, CGI, and FastCGI

2024-06-17 (Last updated 2024-06-26)

Last week, DreamHost upgraded the shared server that this site runs on. The new configuration can no longer run FastCGI reliably, which means that our tiny but useful wwz app went down, and other apps are "at risk".

wwz is now back up as a regular CGI app, so users shouldn't notice much of a change.

Nonetheless, I started a long search for a new shared host, with shell access.

What's shared hosting? It's when you rent part of a big beefy computer, with a Unix kernel and a web server, that somebody else maintains. No virtualization, and no cloud.

To give you an idea, the new DreamHost box has 128 cores/threads and 512 GB of RAM. I have no idea how many accounts share it, but it's still fast. So static content will remain on DreamHost, but dynamic pages will live elsewhere.

I kept track of my search with many messages on our #server-side Zulip stream. This led to the question:

The answer is tied pretty closely to purpose of the Oils project. So I collected comments I've written over several years, with the intention of writing:

That post grew too large, so this one contains just the comments about CGI and FastCGI, with some definitions and motivation.

Table of Contents
History / Motivation
The Demise of the Mildly Dynamic Website
The Long Death of CGI.pm
CGI - Is it relevant? Is it slow?
FastCGI - I think it's dead, as of 2024
Serverless is like FastCGI
Contrast: Gateway Interfaces That Aren't Unix-y
Related: FastCGI and Rust / What is Self Hosted?
Collections of Processes
kcgi source


Readers who "grew up" with the cloud might not know what these protocols are.


CGI is a Unix-y way for a program to generate a web page on the fly. It was the first way of creating dynamic web pages.

How does it work?

  1. A web server like Apache receives an HTTP request over the network.
  2. It encodes the request into the environment and stdin of a new process, which may be written in any language.
  3. The process writes the HTTP response to stdout.
  4. The web server sends that response back over the network.


FastCGI is a less common protocol used for the same purpose:

So FastCGI is designed to address the deficiencies of CGI, though it's arguable how good a job it does.

History / Motivation

The Demise of the Mildly Dynamic Website

This post resonated with me — I want lightweight scripting for the web. I want to write 10 lines of useful Python or shell, and deploy it instantly.

I also want to choose among many hosting providers that work the same way, e.g. using rsync or git to deploy.

PHP enabled dynamic web applications for the masses.

But I don't want to be limited to PHP. And I'm similarly unhappy with cloud solutions:

Recently we've seen the rise in popularity of AWS Lambda, a “functions as a service” provider. From my perspective this is literally a reinvention of CGI, except a) much more complicated for essentially the same functionality, b) with vendor lock-in, c) with a much more complex and bespoke deployment process which requires the use of special tools.

2024-06-26 update - This article was discussed on Hacker News a few days ago. I emphasized that CGI and FastCGI can scale exactly like AWS Lambda. PHP's stateless model scale to all Facebook servers worldwide, and CGI and FastCGI can do the same.

The Long Death of CGI.pm

This post is from 2015, and observes that many users will be discouraged from writing CGI scripts in Perl.

CGI.pm has been removed from the core Perl distribution. From 5.22, it is no longer included in a standard Perl installation.

I just spent a week researching shared hosting, and none of them advertised Perl support. They occasionally mention CGI.

The same thing happened in the Python world: cgi.py was deprecated with Python 3.11, and will be removed in Python 3.13. (PEP 594, 2019)

(Matěj Cepl points out this fork: https://github.com/jackrosenthal/python-cgi)

CGI - Is it relevant? Is it slow?

OK, we've established that that there are diminishing options for web scripting, which disappoints me.

Now let's talk about how CGI and FastCGI work.

My comment notes that starting a new Unix process per request, as CGI does, is not necessarily slow. When you write a CGI program in C, you're still starting a process, but it can complete in 1 to 5 milliseconds, not 100 to 500 milliseconds.

A program that proves the point is cgit, which is CGIs written in C, and you can see it’s plenty fast.


Instead, the main performance issues are:

  1. Interpreter startup time
  2. Lack of persistent state

FastCGI - I think it's dead, as of 2024

Now let's talk about FastCGI. My recent experiences lead me to believe it's dead:

  1. Due to the server upgrade mentioned above, it no longer works reliably on DreamHost.
  2. I searched for a new hosting company for days, and it was hard to find anybody advertising FastCGI. It was even rarer to find good documentation about it.
  3. I found a high quality host, Mythic Beasts, but they discourage FastCGI in their docs.

That said, let's review some past threads about FastCGI.

FastCGI has some bad parts, but the good part is process management. I just drop a file onto the server, and it "lazily" starts concurrent worker processes.

In other words, it gives me PHP-like deployment, but for Python.

Serverless is like FastCGI

Same story on lobste.rs:

My comment noted that FastCGI is like "functions as a service", or "serverless".

This is a great blog post which I've quoted before, though it's now gone (404). The hundreds of comments have more discussion of "serverless" cloud products, and the correspondence with FastCGI and CGI.

Contrast: Gateway Interfaces That Aren't Unix-y

These two web servers seem to use interior shared library plugins, more so than exterior protocols:

  1. uwsgi
  2. Phusion Passenger

The alternative is for a Python WSGI app to be wrapped in a normal HTTP server like gunicorn. A proxy server like nginx can then handle load balancing and SSL.

It's two processes either way — it's just a matter of where you draw the boundary, and what protocol is at the boundary.

I think the author of "FastCGI is useless" above would also agree that the binary uwsgi protocol is also useless.

I'd like to use gunicorn and HTTP, but oddly OpalStack has native support for uwsgi, and no support for supervising long-running HTTP processes. A downside of shared hosting is that you're constrained by lack of root access.


That's all for now. Let me know in the comments if you've run FastCGI on a shared host recently!


Related: FastCGI and Rust / What is Self Hosted?

(2024-06-26 update)

Nick Fagerlund is a fellow FastCGI enthusiast (and DreamHost customer):

Is there any way that I can write little apps using modern, nice-to-develop-in web frameworks (all of which generally expect to run their own server process), but host them on an affordable shared hosting account that won't allow me to run my own separate web server?

Can't I just live the PHP lifestyle, without having to write PHP?

I'm starting to think that the web needs new primitives. We now run some things commonly but the abstractions over them are not ideal. So people target (proprietary) systems directly. The modern web needs the CGI type protocols for queues, for authentication, for columns stores, for caches etc.

Why does it need that? I think it needs it to lower the cost of building small scale open source software.

Collections of Processes

Comments About Build Systems and CI Services (2021)

I'm still thinking about these ideas:

I want to unify the "build time" ideas with the "runtime" ideas:

kcgi source

I ran across this library:

I wondering if I could build it into Oils, so we could have lightweight FastCGI scripts in OSH and YSH.

But I'm not sure it belongs. Our philosophy is to reduce the amount of hand-written C and C++ code in the Oils binary. We want a stable core with few dependencies.

~/src/kcgi-0.9.4$ wc -l *.[ch] | sort -n

   495 output.c
   594 wrappers.c
   599 kcgi.h
   762 kcgihtml.c
   770 fcgi.c
  1121 kcgiregress.c
  1135 kfcgi.c
  1268 kcgi.c
  1903 child.c
 13743 total

Remember, Oils is pure C++ now — no more Python — which opens up many possibilities!