What is TLXOS? Is it Linux? What parts of it are proprietary?

Yes, TLXOS is Linux.  It's is a slightly modified version of Debian stable (currently 10 (Buster) for 4.9.x or 8 (Jessie) for 4.8.x).  TLXOS RPi IoT is based on Raspbian instead of Debian because Debian doesn't support ARMHF on ARM < v7.  TLXOS RPi is based on Debian, not Raspbian.

The only significant differences between Debian(/Raspbian) and TLXOS are that (a) we are using our own kernel build with AUFS support, (b) we use a customized initramfs that mounts the root filesystem as a three-layer union, and (c) we have some closed-source software and shell scripts that act as an agent for our management software, and configures the O/S in accordance with TMS/Tlxconfig settings.  We call this TLXOS rather than Debian mostly so that you don't go pestering Debian people about our customizations; it's still Debian in most respects, and you are free to "apt-get install" whatever you want.

TLXOS' root filesystem is an AUFS (Another Union File System) amalgam of three diffferent layers:

1. /actualroot, the read-only base layer.  We refer to this, along with /tfm and /boot filesystems, as "firmware" (it is really only software, but it's standard practice with embedded Linux to refer to this as "firmware").  When you reset to "factory" defaults (another silly embedded Linux naming convention!) your O/S will be reset to the state described by this filesystem, plus essential extras such as your license (i.e. /config will be (mostly) wiped).
2. /config, the persistent middle layer.  This is where runtime changes to your base firmware are "permanently" stored.  This filesystem is only permanent in the sense that its contents remain intact over power cycles; if you reset to factory defaults, the contents of /config will be destroyed (with the exception of your license file and some absolutely essential settings), and if you upgrade TLXOS, most of it will be wiped (/home/tlx and parts of the root filesystem related to installed files are the only things that are preserved).
3. /run/overlay, the volatile top layer.  This is a ramdisk that accumulates changes made during the current session.  When the system is gracefully shut down or rebooted, these changes (with the exception of some changes that are regarded as expendable and needlessly write-intensive, such as log files (see /etc/persist/exclusions) will be merged into the persistent middle layer (/config).  If power is unexpectedly removed, everything in /run/overlay will be lost; this ensures that your system will always return to a known-good state.

Details of files that do not belong to Debian (or more accurately, files outside of dpkg package management, which include both closed- and open-source files) can be found in /usr/share/doc/UNPACKAGED/README.  Basically, tms_client (the TMS agent) and Tlxconfig (the local configuration tool) are the only parts of TLXOS that are closed source.