OpenVMS

"Albert the Cheshire Cat" mascot for VAX/VMS, used by the DECUS VAX SIG[26][27]

In April 1975, Digital Equipment Corporation embarked on a hardware project, code named Star, to design a 32-bit virtual address extension to its PDP-11 computer line. A companion software project, code named Starlet, was started in June 1975 to develop a totally new operating system, based on RSX-11M, for the Star family of processors.[9] These two projects were tightly integrated from the beginning. Gordon Bell[28] was the VP lead on the VAX hardware and its architecture. Roger Gourd was the project lead for the Starlet program, with software engineers Dave Cutler (who would later lead development of Microsoft's Windows NT), Dick Hustvedt, and Peter Lipman acting as the technical project leaders, each having responsibility for a different area of the operating system. The Star and Starlet projects culminated in the VAX-11/780 computer and the VAX/VMS operating system. The Starlet name survived in VMS as a name of several of the main system libraries, including STARLET.OLB and STARLET.MLB.[29]

With the introduction of the MicroVAX range such as the MicroVAX I, MicroVAX II and MicroVAX 2000 in the mid-to-late 1980s, DIGITAL released MicroVMS versions specifically targeted for these platforms which had much more limited memory and disk capacity; e.g. the smallest MicroVAX 2000 had a 40MB RD32 hard disk and 2MB of RAM, and its CPU had to emulate some of the VAX floating point instructions in software. MicroVMS kits were released for VAX/VMS 4.4 to 4.7 on TK50 tapes and RX50 floppy disks, but discontinued with VAX/VMS V5.0.

Beginning in 1989, a short lived distribution of VMS named Desktop-VMS was sold with VAXstation systems. It consisted of a bundle of VMS, DECwindows, DECnet, VAXcluster support, and software that was designed to help non-technical users set up their workstation without assistance.[30][31] Desktop-VMS had its own versioning scheme beginning with V1.0, which corresponded to the V5.x releases of VAX/VMS.[32]

In July 1992,[33] Digital renamed VAX/VMS to OpenVMS as an indication for its support of "open systems" industry standards such as POSIX and Unix compatibility,[34] and to drop the VAX connection as the port to DIGITAL's 64-bit Alpha RISC processor was in process. The OpenVMS name was first used with the OpenVMS AXP V1.0 release in November 1992. Digital began using OpenVMS VAX instead of VAX/VMS with the V6.0 release in June 1993.[35]

"Vernon the Shark" logo for OpenVMS[36]

In 1988, after the cancellation of the PRISM project, Ken Olsen asked Bob Supnik to investigate ways that Digital could keep the performance of VAX/VMS systems competitive with RISC-based Unix systems.[37] A group of engineers known as the "Extended VAX" or "EVAX" task force was formed, who originally attempted to produce a RISC-like subset of the VAX architecture.[37][38] When this approach turned out to be a dead end, the group began investigating the feasibility of porting VMS and its applications to a clean-slate RISC architecture. The group decided to adopt the Prism architecture with modifications, which eventually became the Alpha.[39] The project to port VMS to the Alpha architecture began in 1989, and booted successfully on real hardware for the first time in 1991.[38]

The main challenge in porting VMS to a new architecture was that VMS and the VAX were designed together, meaning that VMS was dependent on certain details of the VAX architecture.[40] Furthermore, a significant amount of the VMS kernel, layered products, and customer-developed applications were implemented in VAX MACRO assembly code.[9] To port the VAX MACRO code, the MACRO-32 compiler was created, which translated VAX MACRO to Alpha object code.[41] Digital also created a VAX to Alpha translation facility, known as the VAX Environment Software Translator (VEST) to convert VAX executables where it was not possible to recompile the code for Alpha.[42] The VAX compilers for high level languages, many of which had their own bespoke code generators,[43] were converted to use a common compiler backend named GEM.[44] Certain low-level details of the VAX architecture, such as interrupt handling and atomic queue instructions, were emulated in PALcode - which further minimized the amount of changes required to port the VMS kernel to Alpha.

The VMS port to Alpha resulted in the creation of two separate source code libraries (based on a source code management tool known as the VMS Development Environment, or VDE)[4] for VAX, and for Alpha. The Alpha code library was based on a snapshot of the VAX/VMS code base circa V5.4-2.[45] 1992 saw the release of the first version of OpenVMS for Alpha AXP systems, designated OpenVMS AXP V1.0. In 1994, with the release of OpenVMS V6.1, feature (and version number) parity between the VAX and Alpha variants was achieved, this was the so-called Functional Equivalence release.[45] The decision to use the 1.x version numbering stream for the pre-production quality releases of OpenVMS AXP caused confusion for some customers, and was not repeated in the subsequent ports of OpenVMS to new platforms.[40]

When VMS was ported to Alpha, it was initially left as a 32-bit only operating system.[41] This was done to ensure backwards compatibility with software written for the 32-bit VAX. 64-bit addressing was first added for Alpha in the V7.0 release.[46] In order to allow 64-bit code to interoperate with older 32-bit code, OpenVMS does not create a distinction between 32-bit and 64-bit executables, but instead allows for both 32-bit and 64-bit pointers to be used within the same code.[47] This is known as mixed pointer support. The 64-bit OpenVMS Alpha releases support a maximum virtual address space size of 8TiB (a 43-bit address space), which is the maximum supported by the Alpha 21064 and Alpha 21164.[48]

One of the more noteworthy Alpha-only features of OpenVMS was OpenVMS Galaxy - which allowed the partitioning of a single SMP server to run multiple instances of OpenVMS. Galaxy supported dynamic resource allocation to running partitions, and the ability to share memory between partitions.[49][50]

"Swoosh" logo used by HP for OpenVMS

In 2001, just prior to its acquisition by Hewlett-Packard, Compaq announced the port of OpenVMS to the Intel Itanium architecture.[51] The Itanium port was the result of Compaq's decision to discontinue future development of the Alpha architecture in favour of adopting the then-new Itanium architecture.[52] The porting began in late 2001, and the first boot on took place on the 31st of January 2003.[53] The first boot consisted of booting a minimal system configuration on a HP i2000 workstation, logging in as the SYSTEM user, and running the DIRECTORY command.

The Itanium port was accomplished using source code maintained in common within the OpenVMS Alpha source code library, with the addition of conditional code and additional modules where changes specific to Itanium were required.[40] Whereas the VAX and Alpha architectures were specifically designed to support the low-level needs of OpenVMS, Itanium was not. This required certain architectural dependencies of OpenVMS to be replaced, or emulated in software. Some of the changes included:

• The Extensible Firmware Interface (EFI) is used to boot OpenVMS on the Integrity platform, taking over the role of the System Reference Manual (SRM) firmware on Alpha. Support for ACPI was also added to OpenVMS, since this is used to discover and manage hardware devices on the Integrity platform.[54]
• For Itanium, the functionality which was implemented using PALcode for Alpha was moved into a component of the OpenVMS kernel named the Software Interrupt Services (SWIS).[55]
• The Itanium port adopted a new calling standard based on Intel's Itanium calling convention, with extensions to support the OpenVMS Common Language Environment. Furthermore, it replaced the OpenVMS-specific executable formats used on the VAX and Alpha with the standard Executable and Linking Format (ELF) and DWARF formats.[56]
• IEEE 754 was adopted as the default floating point format, replacing the VAX floating point format that was the default on both the VAX and Alpha architectures. For backwards compatibility, it is possible to compile code on Itanium to use the VAX floating point format, but it relies on software emulation.[57]
• The operating system's internal structures were extended to support the 50-bit physical addressing available on Itanium, allowing 1PiB of memory to be addressed.[58] The Itanium port otherwise retained the mixed 32-bit/64-bit pointer architecture which was introduced in OpenVMS Alpha V7.0.

As with the VAX to Alpha port, a binary translator for Alpha to Itanium was made available, allowing user mode OpenVMS Alpha software to be ported to Itanium in situations where it was not possible to recompile the source code. This translator is known as the Alpha Environment Software Translator (AEST), and it also supported translating VAX executables which had already translated with VEST.[59]

Two pre-production releases, OpenVMS I64 V8.0 and V8.1, were available on June 30, 2003 and on December 18, 2003. These releases were intended for HP organizations and third-party vendors involved with porting software packages to OpenVMS I64. The first production release, V8.2, was released in February 2005. V8.2 was also released for Alpha, subsequent V8.x releases of OpenVMS have maintained feature parity between the Alpha and Itanium architectures.[60]

When VMS Software Inc. (VSI) announced that they secured the rights to develop the OpenVMS operating system from HP, they also announced their intention to port OpenVMS to the x86-64 architecture.[61] The porting effort ran concurrently with the establishment of the company, as well as the development of VSI's own Itanium and Alpha releases of OpenVMS 8.x.

The x86-64 port is targeted for specific servers from HPE and Dell, as well as certain virtual machine hypervisors.[62] Initial support was targeted for KVM and VirtualBox. Support for VMware was announced in 2020, and Hyper-V has been described as a future target.[63]

The x86-64 port is built from the same source code library as the Alpha and Itanium architectures, using conditional compilation to manage the architecture-specific code needed to support the x86-64 platform.[64] As with the Alpha and Itanium ports, the x86-64 port made some changes to simplify porting and supporting OpenVMS on the new platform:

• VSI adopted the open source LLVM compiler backend, replacing the proprietary GEM backend used in the Alpha and Itanium ports. A translator was developed to map the GEM IR to LLVM IR, allowing the existing compiler frontends to be reused. In addition, the open source Clang compiler was adopted as the officially supported C++ compiler for OpenVMS under x86-64.[65]
• On x86-64, OpenVMS makes more extensive use of UEFI and ACPI to detect and initialize hardware on boot. As part of this, VMS is now booted from a memory disk, instead of the traditional VMS boot mechanism – which relied on "boot drivers" containing a basic implementation of the filesystem, and which was tied to specific hardware devices. The changes to the boot process necessitated the creation of a "Dump Kernel" – this is a secondary kernel which is loaded in the background at boot time, and is invoked in case OpenVMS needs to write a crash dump to disk.[66]
• OpenVMS assumes the presence of four hardware-provided privilege levels to provide isolation between user applications, and various parts of the operating system. While x86-64 nominally provides four privilege levels, they are only equivalent to two of the privilege levels on the VAX, Alpha and Itanium. In the x86-64 port, the Software Interrupt Services (SWIS) module of the kernel is extended to emulate the missing privilege levels.[55]
• As with the Itanium port, the calling standard for x86-64 is an extension of the platform's standard calling convention, specifically the System V AMD64 ABI. Certain characteristics of the x86-64 architecture created challenges for defining a suitable calling standard. For example, due to the small number of general purpose registers for x86-64, the MACRO-32 compiler has to store the contents of the emulated VAX registers in an in-memory "pseudo registers" structure instead of using the processor's hardware registers as is done on Alpha and Itanium.[67]

The first boot was announced on the 14th May 2019. This involved booting OpenVMS on VirtualBox, and successfully running the DIRECTORY command.[68] Later in 2019, the first "real boot" was announced - this consisted of the operating system booting in a completely standard manner, a user logging into the system, and running the DIRECTORY command.[69] In May 2020, the V9.0 Early Adopter's Kit release was made available to certain customers. This contains the full OpenVMS operating system running in a VirtualBox VM with certain limitations - most significantly, little to no layered products are available, and code can only be compiled for x86-64 using cross compilers which run on Itanium-based OpenVMS systems.[16] Following the V9.0 release, VSI has released a series of monthly updates which add additional functionality, these are designated V9.0-A, V9.0-B, etc.[6]

The OpenVMS Executive comprises the privileged code and data structures which reside in the system space. The Executive is further subdivided between the Kernel, which consists of the code which runs at the kernel access mode, and the less-privileged code which runs at the executive access mode.[79]

The components of the Executive which run outside the Kernel include the Record Management Services, and certain system services such as image activation. The main distinction between the kernel and executive modes is that most of the operating system's core data structures can be read from executive mode, but require kernel mode to be written to.[80] Code running at executive mode can switch to kernel mode at will, meaning that the barrier between the kernel and executive modes is intended as a safeguard against accidental corruption as opposed to a security mechanism.[81]

The Kernel compromises the operating system's core data structures (e.g. page tables, the I/O database and scheduling data), and the routines which operate on these structures. The Kernel is typically described as having three major subsystems: I/O, Process and Time Management, Memory Management.[79][80] In addition, other functionality such as logical name management, synchronization and system service dispatch are implemented inside the kernel.

An OpenVMS Command Language Interpreter (CLI) implements a command line interface for OpenVMS; responsible for executing individual commands, as well as command procedures (equivalent to shell scripts or batch files).[82] The standard CLI for OpenVMS is the DIGITAL Command Language, although other options are available as well.

Unlike Unix shells, which typically run in their own isolated process and behave like any other user mode program, OpenVMS CLIs are an optional component of a process, which exist alongside any executable image which that process may run.[83] Whereas a Unix shell will typically run executables by creating a separate process using fork-exec, an OpenVMS CLI will typically load the executable image into the same process, transfer control to the image, and ensure that control is transferred back to CLI once the image has exited and that the process is returned to its original state.[79] A CLI gets mapped into a process' private address space through execution of the LOGINOUT image, which can either be executed manually, or automatically by certain process creation system services.[48]

Due to fact that the CLI is loaded into the same address space as user code, and that the CLI is responsible for invoking image activation and image rundown, the CLI is mapped into memory at supervisor access mode. This is in order to prevent accidental or malicious manipulation of the CLI's code and data structures by user mode code.[79][83]

VMS was originally designed to be used and managed interactively using Digital's text-based video terminals such as the VT100, or hardcopy terminals such as the DECwriter series. Since the introduction of the VAXstation line in 1984, VMS has optionally supported graphical user interfaces for use with workstations, or graphical terminals connected to a server.

OpenVMS Alpha V8.4-2L1, showing the DCL CLI in a terminal session

The DIGITAL Command Language has served as the primary Command Language Interpreter (CLI) of OpenVMS since the first release.[84][85][10] Other official CLIs available for VMS include the RSX-11 MCR (VAX only), and various Unix shells.[86] Digital provided tools for creating text-based user interface applications - the Form Management System (FMS) and Terminal Data Management System (TDMS), later succeeded by DECforms.[87][88][89] A lower level library, comparable to Unix curses called the Screen Management Services (SMG$) also exists.[90]

VWS 4.5 running on top of VAX/VMS V5.5-2

DECwindows XUI window manager running on top of VAX/VMS V5.5-2

Over the years, VMS has gone through a number of different GUI toolkits and interfaces:

• The original graphical user interface for VMS was a proprietary windowing system known as the VMS Workstation Software (VWS), which was first released for the VAXstation I in 1984.[91] It exposed an API called the User Interface Services (UIS).[92] It ran on a limited selection of VAX hardware.[93]
• In 1989, DEC replaced VWS with a new X11-based windowing system named DECwindows.[94] It was first included in VAX/VMS V5.1.[95] Early versions of DECwindows featured an interface built on top of a proprietary toolkit named the X User Inteface (XUI). A layered product named UISX was provided to allow VWS/UIS applications to run on top of DECwindows.[96]
• In 1991, DEC replaced XUI with the Motif toolkit, creating DECwindows Motif.[97][98] As a result, the Motif Window Manager became the default DECwindows interface in OpenVMS V6.0,[95] although the XUI window manager remained as an option.
• In 1996, as part of OpenVMS V7.1,[95] DEC released the "New Desktop" interface for DECwindows Motif.[99] The New Desktop consisted of a significant subset of the Common Desktop Environment. On Alpha and Itanium systems, it is still possible to select the older MWM-based UI (referred to as the "DECwindows Desktop") at login time. The New Desktop was never ported to the VAX releases of OpenVMS.

Versions of VMS running on DEC Alpha workstations in the 1990s supported OpenGL[100] and Accelerated Graphics Port (AGP) graphics adapters. VMS also provides support for older graphics standards such as GKS and PHIGS.[101][102] Modern versions of DECwindows are based on X.org Server.[10]

OpenVMS supports clustering (first called VAXcluster and later VMScluster), where multiple systems run their own instance of the operating system, but share disk storage, processing, a distributed lock manager, a common management and security domain, job queues and print queues, providing a single system image abstraction. The systems are connected either by proprietary specialized hardware (Cluster Interconnect) or an industry-standard Ethernet LAN. OpenVMS supports up to 96 nodes in a single cluster, and allows mixed-architecture clusters, where VAX and Alpha systems, or Alpha and Itanium systems can co-exist in a single cluster. VMS clusters allow the creation of applications which can withstand planned or unplanned outages of part of the cluster.[103][19]

OpenVMS provides feature-rich facilities for file management. The typical user and application interface into the file system is via the Record Management Services (RMS), although applications can interface directly with the underlying file system through the QIO system services.[104] RMS supports multiple record-oriented file access methods and record formats (including a stream format where the file is treated as a stream of bytes, similar to Unix). RMS also supports remote file access via DECnet,[105] and optional support for journaling.[106]

The file systems supported by VMS are referred to as the Files-11 On-Disk Structures (ODS), which provide disk quotas, access control lists and file versioning.[107] The most significant structure levels are ODS-2, which is the original VMS file system, and ODS-5, which extended ODS-2 with support for Unicode file names, case sensitivity, hard links and symbolic links.[108] VMS is also capable of accessing files on ISO 9660 CD-ROMs and magnetic tape with ANSI tape labels.[109]

Alongside the OpenVMS Alpha V7.0 release in 1995, Digital released a log-structured file system named Spiralog which was intended as a potential successor to Files-11.[110] Spiralog shipped as an optional product, and was discontinued at the release of OpenVMS Alpha 7.2.[111] Spiralog's discontinuation was due to a variety of problems, including issues with handling full volumes.[112] The developers of Spiralog began work on a new file system in 1996, which was put on hold and later resumed by VSI in 2016 as the VMS Advanced File System (VAFS, not to be confused with Digital's AdvFS for Tru64).[113][114] VAFS no longer appears on recent roadmaps, and instead VSI have discussed porting the open source GFS2 file system to OpenVMS.[78][115] One of the major motivations for replacing the Files-11 structures is that they are limited to 2TiB volumes.[108]

Digital's DECnet protocol suite is tightly integrated into VMS, allowing remote logins, as well as transparent access to files, printers and other resources on VMS systems over a network.[116] Modern versions of VMS support both the traditional Phase IV DECnet protocol, as well the OSI-compatible Phase V (also known as DECnet-Plus).[117] Support for TCP/IP is provided by the optional TCP/IP Services for OpenVMS layered product (originally known as the VMS/ULTRIX Connection, then as the ULTRIX Communications Extensions or UCX).[118][119] TCP/IP Services is based on a port of the BSD network stack to OpenVMS,[120] along with support for common protocols such as SSH, DHCP, FTP and SMTP. Due to the fact that the official TCP/IP stack was introduced relatively late, multiple third party TCP/IP stacks have been created for VMS. Some of these third party TCP/IP stacks remain under active development, such as TCPware and MultiNet.[121]

Digital sold a software package named PATHWORKS (originally known as the Personal Computer Systems Architecture or PCSA) which allowed personal computers running MS-DOS, Microsoft Windows or OS/2, or the Apple Macintosh to serve as a terminal for VMS systems, or to use VMS systems as a file or print server.[122] PATHWORKS was based on LAN Manager and supported either DECnet or TCP/IP as a transport protocol. PATHWORKS was later renamed to Advanced Server for OpenVMS, and was eventually replaced with a VMS port of Samba at the time of the Itanium port.[123]

Digital provided the Local Area Transport (LAT) protocol which allowed remote terminals and printers to be attached to a VMS system through a terminal server.[124]

OpenVMS represents system time as the signed 64-bit number of 100 nanosecond intervals (that is, ten million units per second; also known as a 'clunk'[125][126]) since the epoch.[79] The epoch of OpenVMS is midnight preceding November 17, 1858, which is the start of Modified Julian Day numbering.[127] This will allow time computations up to 31-JUL-31086 02:48:05.47 before overflow occurs.[127]

While the native OpenVMS time format can range far into the future, applications based on the C runtime library will likely encounter timekeeping problems beyond January 19, 2038 due to the Year 2038 problem. Applications may also encounter the Year 10,000 problem due to fixed-length date fields.[128]

Digital (and its successor companies) provided a wide variety of programming languages for VMS. Officially supported languages on VMS, either current or historical, include:[86][129]

Among OpenVMS's notable features is the Common Language Environment, a strictly defined standard that specifies calling conventions for functions and routines, including use of stacks, registers, etc., independent of programming language.[67] Because of this, it is possible and straightforward to call a routine written in one language (for example, Fortran) from another (for example, COBOL), without needing to know the implementation details of the target language. OpenVMS itself is implemented in a variety of different languages and the common language environment and calling standard supports freely mixing these languages.[130][131] Digital created a tool named the Structure Definition Language (SDL), which allowed data type definitions to be generated for different languages from a common definition.[132]

Digital provided a collection of software development tools as a layered product named DECset (originally named VAXset).[86] This consisted of the Language-Sensitive Editor (LSE), a version control system (the Code Management System or CMS), a build tool (the Module Management System or MMS), a static analyzer (the Source Code Analyzer or SCA), as well as testing tools (the Digital Test Manager and the Performance and Coverage Analyzer or DTM and PCA respectively).[133] In addition, a number of text editors are included in the operating system, including EDT, EVE and TECO.[134]

The OpenVMS Debugger supports all DEC compilers and many third party languages. It allows breakpoints, watchpoints and interactive runtime program debugging either using a command line or graphical user interface.[135] A pair of lower-level debuggers, called DELTA and XDELTA, can be used to debug privileged code in additional to normal application code.[136]

In 2019, VSI released an officially-supported Integrated Development Environment for VMS based on Visual Studio Code.[62] This allows VMS applications to be developed and debugged remotely from a Microsoft Windows, macOS or Linux workstation.[137]

The "Grey Wall" of VAX/VMS documentation, at Living Computers: Museum + Labs

Digital created a number of optional database products for VMS, some of which were marketed as the VAX Information Architecture family.[138] These products included:

• Rdb - A relational database system which originally used the proprietary Relational Data Operator (RDO) query interface, but later gained SQL support.[139]
• DBMS - A database management system which uses the CODASYL network model and Data Manipulation Language (DML).
• Digital Standard MUMPS (DSM) - an integrated programming language and key-value database.[86]
• Common Data Dictionary (CDD) - a central database schema repository, which allowed schemas to be shared between different applications, and data definitions to be generated for different programming languages.
• DATATRIEVE - a query and reporting tool which could access data from RMS files as well as Rdb and DBMS databases.
• Application Control Management System (ACMS) - A transaction processing monitor, which allows applications to be created using a high-level Task Description Language (TDL). Individual steps of a transaction can be implemented using DCL commands, or Common Language Environment procedures. Interfaces can be implemented using TDMS, DECforms or Digital's ALL-IN-1 office automation product.[140]
• RALLY, DECadmire - Fourth-generation programming languages (4GLs) for generating database-backed applications.[141] DECadmire featured integration with ACMS, and later provided support for generating Visual Basic client-server applications for Windows PCs.[142]

In 1994, Digital sold Rdb, DBMS and CDD to Oracle, where they remain under active development.[143] In 1995, Digital sold DSM to InterSystems, who renamed it Open M, and eventually replaced it with their Caché product.[144]

Examples of third-party database management systems for OpenVMS include MariaDB,[145] Mimer SQL[146] and System 1032.[147]

In a manner similar to Unix, VMS defines several standard input and output channels[82] with these logical names:

SYS$INPUT – Standard input. Used interactively, this represents the terminal keyboard. Used in a batch file, it is batch file lines not preceded with a $ symbol, or specified as an input deck using the DECK command.

SYS$OUTPUT – Standard output. Used interactively, this is the terminal display. Used in a batch file, it outputs to the screen if the file is run interactively or to the log file when the file is run noninteractively.

SYS$ERROR – Standard error. Used interactively, this is the terminal display. In a batch file, it is the terminal display when the file is run interactively, or to the log file when the file is run noninteractively, or in the special case of RUN /DETACH, to the output file or device specified with the /ERROR= parameter.

SYS$COMMAND – Does not have a direct analogue in the Unix model. Used interactively, it will read from the terminal. Used in a batch file when run interactively, it will read from the terminal. Used in a batch file run noninteractively, it will read from the SYS$INPUT stream (if one is defined), otherwise it will read nothing and return end of file. /dev/tty on Unix is similar to SYS$COMMAND in interactive sessions, but is not available in non-interactive sessions.

OpenVMS provides various security features and mechanisms, including security identifiers, resource identifiers, subsystem identifiers, ACLs, and detailed security auditing and alarms. Specific versions evaluated at DoD NCSC Class C2 and, with the SEVMS security enhanced services support, at NCSC Class B1, per the NCSC Rainbow Series.[148] OpenVMS also holds an ITSEC E3 rating (see NCSC and Common Criteria).[149][150] Passwords are hashed using the Purdy Polynomial.

A 33-year-old vulnerability in VAX/VMS and OpenVMS Alpha, was discovered in 2017 and assigned the CVE ID CVE-2017-17482. On the affected platforms, this vulnerability allowed an attacker with access to the DCL command line to bypass system security and take full control of the system. The vulnerability relies on exploiting a buffer overflow bug in the DCL command processing code, the ability for a user to interrupt a running image (program executable) with CTRL/Y and return to the DCL prompt, and the fact that DCL retains access to the privileges of the images that it requests to be loaded into the DCL process when the image is interrupted.[151] The buffer overflow bug allowed shellcode to be executed with the privileges of an interrupted image, and thus allowing an attacker to carry out a privilege escalation attack.[152]

VAX/VMS originally included an RSX-11M compatibility layer named the RSX Application Migration Executive (AME) which allowed user mode RSX-11M software to be run unmodified on top of VMS.[85] This relied on the PDP-11 compatibility mode implemented in the VAX-11 processors.[153] The RSX AME played an important role on early versions of VAX/VMS, which used re-used certain RSX-11M user space utilities before native VAX versions had been developed.[9] This was discontinued in VAX/VMS V3.0 when all compatibility mode utilities were replaced with native implementations, and RSX AME was removed from the base system.[154] At this point, RSX AME was replaced with an optional layered product on VAX named VAX-11 RSX, which relied on software emulation to run PDP-11 code on newer VAX processors.[153] A VAX port of the RTEM compatibility layer for RT-11 applications was also available from Digital.[155]

Various official Unix compatibility layers were created for VMS. The first of which was DEC/Shell - which was a layered product consisting of port of the Version 7 Unix Bourne Shell and several other Unix utilities to VAX/VMS.[86] In 1992, Digital released the POSIX for OpenVMS layered product, which included a shell based on the Korn Shell.[156] POSIX for OpenVMS was later replaced by the open source GNV (GNU's not VMS) project, which was first included in OpenVMS media in 2002.[157] Amongst other GNU tools, GNV includes a port of the Bash shell to VMS.[158] Examples of third party Unix compatibility layers for VMS include Eunice.[159]

Digital licensed SoftPC (and later SoftWindows), and sold it as a layered product for both the VAX and Alpha architectures, allowing Windows and DOS applications to run on top of VMS.[160][161]

Some of the open source applications which have been ported to OpenVMS include:[62][129]

There are a number of community projects to port open source software to VMS, include VMS-Ports[162] and GNV (GNU's Not VMS).[163]