Luke Plant: Everything is an X

“Everything is an X” is a very high level pattern that you see applied in the design of lots of systems, including programming languages and user interfaces. It has a lot of advantages, and some disadvantages. I’ll discuss some of these, then look at some examples, which will be necessary to understand what I’m really talking about.

When we say “everything” here, we’re talking very loosely — many things in the system are not, in fact, instances of “X”, but are something lower level, or just completely different. “Everything is an X” really means “a surprising number of things in this system are an X”.

Advantages

• Simplicity of implementation for the implementer. They don’t have to design and implement a new interface for every new thing in the system. Instead they implement a single interface that they can re-use everywhere and refine as necessary.

• Simplicity for the user. They only have to learn one thing, and knowledge gained in one area is immediately applicable to many others.

• Meta-powers. Not only can the user apply knowledge or skills from one instance in another area, very often the “X” interface can be applied at a higher level, and this gives the user super-powers — they can go from manipulating X to manipulating meta-X with very little extra knowledge needed.

  When there are multiple meta-levels that don’t seem to stop, another name for this pattern is “X all the way down“.

• Elegance and beauty. Beyond utility, systems designed like this have an aesthetic value that I think is motivating for both the implementer and the user.

Disadvantages

• For some of the things in the system, making them behave as “X” can be a very bad and awkward fit. This could have the effect of:
  • producing an under-powered interface for some things, which has to be patched up in some way.
  • giving an over-powered, bloated interface for other things.
  • making “X” too broad to be useful.
• Giving the user a higher-level, meta interface could bring challenges, and limit how you restructure or optimise internals because you’ve given the user too much.
• There are also issues when you have to break the “everything is an X” pattern for some pragmatic reason — you end up with an ugly corner of the design.

Examples

Programming

• Java: everything is a class (sort of).

  This gives a certain uniformity in how APIs work that tends to reduce the amount of work you have to do when learning a new library.

  In terms of meta-levels, Java doesn’t have so much. It has some reflection abilities that allow classes to be represented as runtime objects, although it is not very deep in terms of what it offers (contrast Python below).

  The limited power for users is undoubtedly useful for compiler writers, however — you can get excellent performance with Java due to the more limited nature of the language (again, contrast Python below).

  Java breaks its own pattern of basing everything on classes with “primitive types”, which means you have also have to deal with boxing, auto-boxing etc. Javascript has similar issues with Number.

• Python: everything is an object.

  Note that this is very different from Java’s version of “everything is a class/object” — far more things are actually runtime objects in Python, including functions, methods, types/classes and modules, as well as instances of classes. This makes parameterisation an extremely powerful and general pattern in Python.

  You also get super powers that come from how meta-classes work:

  • instances are objects
  • classes (the things that produce instances) are objects
  • meta-classes (the things that produce classes) are objects

  So meta-class programming is just normal programming. It can be done at a REPL, or with any other technique that you have already learned. Using meta-classes of course requires learning new things, but things still work in very familiar ways. Python feels like it’s made out of Python all the way down.

  We also see the disadvantages. For example, Python integers are full Python objects, which is extremely wasteful of memory compared to compact representations (like C would use, or numpy uses for example). Plus, they are immutable (for good reasons), so doing simple addition causes memory allocations. This is pretty horrifying from an efficiency point of view!

• ML family languages, like Haskell/OCaml and relatives: everything is an expression.

  So, for example, there is no if statement, there is instead an if expression. Assignments, too, are actually expressions — for example a let statement is instead an expression that defines some local values and returns the value of the ‘body’.

  Exceptions: Top level assignments, and type signatures. At least in Haskell, the type system feels quite separate. Due to this, AFAICS the meta level is kind of missing in Haskell. Haskell “super-powers” come from a bunch of different features, such as deriving and TemplateHaskell, and programming at type level, especially with various language extensions. These are all cool features, but they have to be learned separately, and I think this is what gives Haskell the feeling of being a big and intimidating language.

• Lisp: everything is an s-expression.

  In contrast to Haskell, this goes much further and includes top level statements. This uniformity of syntax is hugely helpful in writing macros, probably the biggest meta-power that Lisp boasts.

User interfaces

• Emacs: everything is a buffer.

  This is one of Emacs’ big ideas — it has very few different UI elements, because almost everything appears in a buffer. As well as the text of a file to edit, you also have:

  • help manuals
  • auto-generated help for the buffer you are using (such as listing all keyboard shortcuts).
  • search results (this is an essential part of my workflow for recursive searching that remembers context)
  • many, many more things.

  Once you have learned how to navigate and use buffers, you know how to navigate these things.

  Of course, there are other UI elements, such as the toolbar and menubar, to make Emacs slightly less terrifying to new users. But these tend to be turned off by experienced users — they’re not very useful because…they’re not buffers!

  You also gain meta-powers: the list of all buffers is of course a buffer, and you can manipulate the buffer-buffer using normal mechanisms (e.g. delete line to kill a buffer).

• RDMSs: everything is a table/relation, and can be queried as such.

  With some implementations such as PostgreSQL, as well as the normal relations you have created, internal structures are also presented as relations, including the lists of tables/relations/indexes, and runtime information.

  So you can get a table of all tables:

  SELECT table_name
  FROM information_schema.tables
  WHERE table_schema = 'public'
  ORDER BY table_name;
  

  And query the currently running queries:

  SELECT * FROM pg_stat_activity;
  

  If you extended the meta-powers, you could create/delete tables by inserting into/deleting from the “tables table”, and similarly add/remove/alter columns by manipulating the “columns table”. Your schema then becomes data, and your power has become a super-power. I don’t know if any DB has implemented this, or if the advantages would be really compelling over normal DDL.

• Visidata: everything is a sheet.

  This seems to go further than PostgreSQL in terms of giving you meta-powers and sticking to the pattern. In addition to the normal data sheets:

  • rows are sheets (or can be viewed as such)
  • column meta-data is a sheet (and not just read-only - you can manipulate it)
  • the list of sheets is a sheet
  • configuration is a sheet

  …and more.

  I’m loving the power of Visidata, which is basically “Excel as a text user interface”, and I’ve realised that a lot of its power comes from its lack of power.

  In Excel, you can have multiple tables in a sheet, and put them wherever you like. You can have headers, or not, you can have different format for every cell etc. In Visidata, you have exactly one table per sheet, you must have headers, each column has exactly one format etc. The enforced uniformity of “Everything is a sheet (and a sheet has a tightly defined structure)” is actually crucial to making Visidata so much nicer than Excel for many tasks — without it, most of the interface shortcuts and all of the meta-powers would fail.

• REST — everything is a resource.

  The idea of everything in your system being a “resource” which you can manipulate with a few well-understood verbs (GET, PUT, DELETE etc.) is a really attractive part of REST.

  It comes with some meta-powers - with things like OpenAPI, the list of resources is typically also served as a resource.

  Relatively frequently with REST I feel I hit the “awkward fit” disadvantage. Modelling some actions as “manipulating a resource” is very bizarre, and what you really want is just a RPC (for example, with “clone/copy/move” actions where there are hidden properties that you also want to be handled).

• Unix - everything is a file.

  This is a pretty successful abstraction, but it does leak quite a bit. Files turn out to be much harder than you think, even before you start treating things that aren’t files as files. And in some cases you might want to deal with files as if they are not files but memory anyway (mmap).

Questions

I’m sure there are many, many more examples of this pattern, and I’m only scratching the surface. EverythingIsa on c2.com has a longer list, for example, but without much analysis. What are your favourite examples? Are there examples where it works very well — or very badly? Are there other advantages/disadvantages that I’ve missed?

from Planet Python
via read more