Interfacing TeXmacs with other programs

1.Interfacing TeXmacs with other programs

1.Introduction

In this chapter we describe how to interface TeXmacs with an extern application. Such interfaces should be distributed in the form of plugins. The plug-in may either contain the extern application, or provide the “glue” between TeXmacs and the application. Usually, interfaces are used interactively in shell sessions (see Insert→Session). But they may also be designed for background tasks, such as spell checking or typesetting.

The communication between TeXmacs and the application takes place using a customizable input format and the special TeXmacs meta-format for output from the plug-in. The meta-format enables you to send structured output to TeXmacs, using any common format like verbatim, LaTeX, Postscript, HTML, or TeXmacs itself. This is useful when adding a TeXmacs interface to an existing system, since LaTeX or Postscript output routines are often already implemented. It will then suffice to put the appropriate markers in order to make a first interface with TeXmacs.

As soon as basic communication between your application and TeXmacs is working, you may improve the interface in many ways. Inside shell sessions, there is support for prompts, default inputs, tab-completion, mathematical and multi-line input, etc. In general, your application may take control of TeXmacs and modify the user interface (menus, keyboard, etc.) or add new Scheme routines to TeXmacs. Your application may even extend the typesetter.

In the directory $TEXMACS_PATH/examples/plugins, you can find many examples of simple plug-ins. In the next sections, we will give a more detailed explanation of the interfacing features of TeXmacs on the hand of these examples. In order to try one of these examples, we recall that you just have to copy it to either one of the directories

    $TEXMACS_PATH/plugins
    $TEXMACS_HOME_PATH/plugins

and run the Makefile (if there is one).

2.Basic input/output using pipes

The configuration and the compilation of the minimal plug-in is described in the chapter about plug-ins. We will now study the source file minimal/src/minimal.cpp. Essentially, the main routine is given by

int
main () {
  display-startup-banner
  while (true) {
    read-input
    display-output
  }
  return 0;
}

By default, TeXmacs just send a '\n'-terminated string to the application as the input. Consequently, the code for read-input is given by

char buffer[100];
cin.getline (buffer, 100, '\n');

The output part is more complicated, since TeXmacs needs to have a secure way for knowing whether the output has finished. This is accomplished by encapsulating each piece of output (in our case both the display banner and the interactive output) inside a block of the form

Here DATA_BEGIN and DATA_END stand for special control characters:

#define DATA_BEGIN   ((char) 2)
#define DATA_END     ((char) 5)
#define DATA_ESCAPE  ((char) 27)

The DATA_ESCAPE is used for producing the DATA_BEGIN and DATA_END characters in the message using the rewriting rules

DATA_ESCAPEDATA_BEGIN		DATA_BEGIN
DATA_ESCAPEDATA_END		DATA_END
DATA_ESCAPEDATA_ESCAPE		DATA_ESCAPE

The format specifies the format of the message. For instance, in our example, the code of display-startup-banner is given by

cout << DATA_BEGIN << "verbatim:";
cout << "Hi there!";
cout << DATA_END;
cout.flush ();

Similarly, the code of display-output is given by

cout << DATA_BEGIN << "verbatim:";
cout << "You typed " << buffer;
cout << DATA_END;
cout.flush ();

Remark 1. For synchronization purposes, TeXmacs will assume that the output is finished as soon as it encounters the DATA_END which closes the initial DATA_BEGIN. So all output has to be inside one single outer DATA_BEGIN-DATA_END block: if you send more blocks, then TeXmacs will retake control before reading all your output. For certain formats (such as verbatim), it is possible to nest DATA_BEGIN-DATA_END blocks though, as we will see below.

Remark 2. In our example, the C++ code for the application is included in the plug-in. In the case when you are writing a TeXmacs interface for an existing application myapp, the convention is to create a –texmacs option for this program. Then it is no longer necessary to have myapp/src and myapp/bin directories for your plug-in and it suffices to configure the plug-in by putting something like the following in myapp/progs/init-myapp.scm:

(plugin-configure myapp
  (:require (url-exists-in-path? "myapp"))
  (:launch "myapp --texmacs")
  (:session "Myapp"))

In the case when you do not have the possibility to modify the source code of myapp, you typically have to write an input/output filter tm_myapp for performing the appropriate rewritings. By looking at the standard plug-ins distributed with TeXmacs in

    $TEXMACS_PATH/plugins

you can find several examples of how this can be done.

3.Formatted and structured output

In the previous section, we have seen that output from applications is encapsulated in blocks of the form

Currently implemented formats include verbatim, latex, html, ps, and scheme. Certain formats, such as verbatim, allow the message to recursively contain blocks of the same form. The scheme format is used for sending TeXmacs trees in the form of Scheme expressions.

The formula plug-in

The formula plug-in demonstrates the use of LaTeX as the output format. It consists of the files

    formula/Makefile
    formula/progs/init-formula.scm
    formula/src/formula.cpp

The body of the main loop of formula.cpp is given by

int i, nr;
cin >> nr;
cout << DATA_BEGIN << "latex:";
cout << "$";
for (i=1; i<nr; i++)
  cout << "x_{" << i << "}+";
cout << "x_{" << i << "}$";
cout << DATA_END;
cout.flush ();

Similarly, the use of nested output blocks is demonstrated by the nested plug-in; see in particular the source file nested/src/nested.cpp.

Remark 3. At the moment, we only implemented LaTeX as a standard transmission format for mathematical formulas, because this is the format which is most widely used. In the future, we intend to implement more semantically secure formats, and we recommend you to keep in mind the possibility of sending your output in tree format.

Nevertheless, we enriched standard LaTeX with the \* and \bignone commands for multiplication and closing big operators. This allows us to distinguish between

    a \* (b + c)

(i.e. multiplied by ) and

    f(x + y)

(i.e. applied to ). Similarly, in

    \sum_{i=1}^m a_i \bignone + \sum_{j=1}^n b_j \bignone

the \bignone command is used in order to specify the scopes of the \sum operators.

It turns out that the systematic use of the \* and \bignone commands, in combination with clean LaTeX output for the remaining constructs, makes it a priori possible to associate an appropriate meaning to your output. In particular, this usually makes it possible to write additional routines for copying and pasting formulae between different systems.

The markup plug-in

It is important to remind that structured output can be combined with the power of TeXmacs as a structured editor. For instance, the markup plug-in demonstrates the definition of an additional tag foo, which is used as an additional primitive in the output of the application. More precisely, the markup plug-in consists of the following files:

    markup/Makefile
    markup/packages/session/markup.ts
    markup/progs/init-markup.scm
    markup/src/markup.cpp

The style package markup.ts contains the following definition for foo:

The foo tag is used in the following way in the body of the main loop of markup.cpp:

char buffer[100];
cin.getline (buffer, 100, '\n');
cout << DATA_BEGIN << "latex:";
cout << "$\\foo{" << buffer << "}$";
cout << DATA_END;
cout.flush ();

Notice that the style package markup.ts also defines the markup-output environment:

This has the effect of centering the output in sessions started using Insert→Session→Markup.

4.Output channels, prompts and default input

Besides blocks of the form

the TeXmacs meta-format also allows you to use blocks of the form

Here channel specifies an “output channel” to which the body message has to be sent. The default output channel is output, but we also provide channels prompt and input for specifying the prompt and a default input for the next input in a session. Default inputs may be useful for instance be useful for demo modes of computer algebra systems. In the future, we also plan to support error and status channels.

The prompt plug-in

The prompt plug-in shows how to use prompts. It consists of the files

    prompt/Makefile
    prompt/progs/init-prompt.scm
    prompt/src/prompt.cpp

The routine for displaying the next prompt is given by

void
next_input () {
  counter++;
  cout << DATA_BEGIN << "prompt#";
  cout << "Input " << counter << "] ";
  cout << DATA_END;
}

This routine is both used for displaying the startup banner

cout << DATA_BEGIN << "verbatim:";
cout << "A LaTeX -> TeXmacs converter";
next_input ();
cout << DATA_END;
cout.flush ();

and in the body of the main loop

char buffer[100];
cin.getline (buffer, 100, '\n');
cout << DATA_BEGIN << "verbatim:";
cout << DATA_BEGIN;
cout << "latex:$" << buffer << "$";
cout << DATA_END;
next_input ();
cout << DATA_END;
cout.flush ();

5.Sending commands to TeXmacs

The application may use command as a very particular output format in order to send Scheme commands to TeXmacs. In other words, the block

will send the command cmd to TeXmacs. Such commands are executed immediately after reception of DATA_END. We also recall that such command blocks may be incorporated recursively in larger DATA_BEGIN-DATA_END blocks.

The menus plug-in

The nested plug-in shows how an application can modify the TeXmacs menus in an interactive way. The plug-in consists of the files

    menus/Makefile
    menus/progs/init-menus.scm
    menus/src/menus.cpp

The body of the main loop of menus.cpp simply contains

char buffer[100];
cin.getline (buffer, 100, '\n');
cout << DATA_BEGIN << "verbatim:";
cout << DATA_BEGIN << "command:(menus-add \""
     << buffer << "\")" << DATA_END;
cout << "Added " << buffer << " to menu";
cout << DATA_END;
cout.flush ();

The Scheme macro menus-add is defined in init-menus.scm:

(define menu-items '("Hi"))

(tm-menu (menus-menu)
  (for (entry menu-items)
    ((eval entry) (insert entry))))

(tm-define (menus-add entry)
  (set! menu-items (cons entry menu-items)))

(plugin-configure menus
  (:require (url-exists-in-path? "menus.bin"))
  (:launch "menus.bin")
  (:session "Menus"))

(menu-bind plugin-menu
  (:require (in-menus?))
  (=> "Menus" (link menus-menu)))

The configuration of menus proceeds as usual:

(plugin-configure menus
  (:require (url-exists-in-path? "menus.bin"))
  (:launch "menus.bin")
  (:session "Menus"))

6.Background evaluations

Until now, we have always considered interfaces between TeXmacs and applications which are intended to be used interactively in shell sessions. But there also exists a Scheme command

(plugin-eval plugin session expression)

for evaluating an expression using the application. Here plugin is the name of the plug-in, session the name of the session and expression a Scheme expression which represents a TeXmacs tree.

The substitute plug-in

Background evaluations may for instance be used in order to provide a feature which allows the user to select an expression and replace it by its evaluation. For instance, the substitute plug-in converts mathematical LaTeX expressions into TeXmacs, and it provides the ? keyboard shortcut for replacing a selected text by its conversion. The plug-in consists of the following files

    substitute/Makefile
    substitute/progs/init-substitute.scm
    substitute/src/substitute.cpp

The main evaluation loop of substitute.cpp simply consists of

char buffer[100];
cin.getline (buffer, 100, '\n');
cout << DATA_BEGIN;
cout << "latex:$" << buffer << "$";
cout << DATA_END;
cout.flush ();

Moreover, the configuration file init-substitute.scm contains the following code for replacing a selected region by its evaluation

(define (substitute-substitute)
  (import-from (texmacs plugin plugin-cmd))
  (if (selection-active-any?)
      (let* ((t (tree->stree (the-selection)))
             (u (plugin-eval "substitute" "default" t)))
        (clipboard-cut "primary")
        (insert (stree->tree u)))))

as well as the keyboard shortcut for ?:

(kbd-map
  ("C-F12" (substitute-substitute)))

Notice that these routines should really be defined in a separate module for larger plug-ins.

The secure plug-in

Another example of using an interface in the background is the secure plug-in which consists of the files

    secure/Makefile
    secure/packages/secure.ts
    secure/progs/init-secure.scm
    secure/progs/secure-secure.scm
    secure/src/secure.cpp

Just as substitute.cpp above, the main program secure.cpp just converts mathematical LaTeX expressions to TeXmacs. The secure-secure.scm module contains the secure Scheme routine latexer:

(tm-define (latexer s)
  (:type (tree -> object))
  (:synopsis "convert LaTeX string to TeXmacs tree using plugin")
  (:secure #t)
  (plugin-eval "secure" "default" (tree->string s)))

It is important to define latexer as being secure, so that it can be used in order to define additional markup using the extern primitive. This is done in the style file secure.ts:

After compilation, installation, relaunching TeXmacs and selecting Document→Use package→secure, you will now be able to use latexer as a new primitive. The primitive takes a mathematical LaTeX expression as its argument and displays its TeXmacs conversion.

7.Mathematical and customized input

The TeXmacs meta-format allows application output to contain structured text like mathematical formulas. In a similar way, you may use general TeXmacs content as the input for your application. By default, only the text part of such content is kept and sent to the application as a string. Moreover, all characters in the range 0–31 are ignored, except for '\t' and '\n' which are transformed into spaces. There are two methods to customize the way input is sent to your application. First of all, the configuration option

(:serializer ,routine)

specifies a scheme function for converting TeXmacs trees to string input for your application, thereby overriding the default method. This method allows you for instance to treat multi-line input in a particular way or the perform transformations on the TeXmacs tree.

The :serialize option is a very powerful, but also a very abstract way to customize input: it forces you to write a complete input transformation function. In many circumstances, the user really wants to rewrite two dimensional mathematical input to a more standard form, like rewriting to ((a)/(b)). Therefore, a second way for customizing the input is to use the command

 (plugin-input-converters myplugin
   rules)

This command specifies input conversion rules for myplugin for “mathematical input” and reasonable defaults are provided by TeXmacs. Each rule is of one of the following two forms:

Leaf transformation rules

Given two strings symbol and conversion, the rule

(symbol conversion)

specifies that the TeXmacs symbol symbol should be converted to conversion.

Tag transformation rules

Given a symbol tag and a Scheme function routine, the rule

(tag routine)

specifies that routine will be used as the conversion routine for tag. This routine should just write a string to the standard output. The Scheme function plugin-input may be used for the recursive transformation of the arguments of the tag.

The input plug-in

The input plug-in demonstrates the use of customized mathematical input. It consists of the files

    input/Makefile
    input/packages/session/input.ts
    input/progs/init-input.scm
    input/progs/input-input.scm
    input/src/input.cpp

The Scheme configuration code in init-input.scm is given by

(plugin-configure input
  (:require (url-exists-in-path? "input.bin"))
  (:launch "input.bin")
  (:session "Input"))

(when (supports-initialize?)
  (lazy-input-converter (input-input) input))

The predicate supports-initialize? tests whether the plug-in is indeed operational (that is, whether input.bin exists in the path). The conversion rules in the module (input input) are added in a lazy manner. In other words, the file input-input.scm will only be loaded when we explicitly request to make a conversion. The conversion rules in input-input.scm are given by

(plugin-input-converters input
  (frac input-input-frac)
  (special input-input-special)
  ("<vee>" "||")
  ("<wedge>" "&&"))

This will cause and to be rewritten as || and && respectively. Fractions are rewritten as ((a):(b)) using the routine

(define (input-input-frac t)
  (display "((")
  (plugin-input (car t))
  (display "):(")
  (plugin-input (cadr t))
  (display "))"))

In the additional style file input.ts we also defined some additional markup special:

This tag is rewritten using the special conversion rule

(define (input-input-special t)
  (display "[[[SPECIAL:")
  (plugin-input (car t))
  (display "]]]"))

As to the C++ code in input.cpp, the startup banner automatically puts the shell session in mathematical input mode:

cout << DATA_BEGIN << "verbatim:";
cout << DATA_BEGIN << "command:(session-use-math-input #t)"
     << DATA_END;
cout << "Convert mathematical input into plain text";
cout << DATA_END;
cout.flush ();

In the main loop, we content ourselves the reproduce the input as output:

char buffer[100];
cin.getline (buffer, 100, '\n');
cout << DATA_BEGIN << "verbatim:";
cout << buffer;
cout << DATA_END;
cout.flush ();

8.Tab-completion

By default, TeXmacs looks into your document for possible tab-completions. Inside sessions for your application, you might wish to customize this behaviour, so as to complete built-in commands. In order to do this, you have to specify the configuration option

(:tab-completion #t)

in your init-myplugin.scm file, so that TeXmacs will send special tab-completion requests to your application whenever you press ⇥ inside a session. These commands are of the form

DATA_COMMAND(complete input-string cursor-position)↩

Here DATA_COMMAND stands for the special character '\20' (ASCII 16). The input-string is the complete string in which the ⇥ occurred and the cursor-position is an integer which specifies the position of the cursor when you pressed ⇥. TeXmacs expects your application to return a tuple with all possible tab-completions of the form

Here root corresponds to a substring before the cursor for which completions could be found. The strings completion-1 until completion-n are the list of completions as they might be inserted at the current cursor position. If no completions could be found, then you may also return the empty string.

Remark 4. In principle, the tab-completion mechanism should still work in mathematical input mode. In that case, the input-string will correspond to the serialization of the TeXmacs input.

Remark 5. The way TeXmacs sends commands to your application can be customized in a similar way as for the input: we provide a :commander configuration option for this, which works in a similar way as the :serializer option.

The complete plug-in

A very rudimentary example of how the tab-completion mechanism works is given by the complete plug-in, which consists of the following files:

    complete/Makefile
    complete/progs/init-complete.scm
    complete/src/complete.cpp

The startup banner in complete.cpp takes care of part of the configuration:

cout << DATA_BEGIN << "verbatim:";
format_plugin ();
cout << "We know how to complete 'h'";
cout << DATA_END;
fflush (stdout);

Here format_plugin is given by

void
format_plugin () {
  // The configuration of a plugin can be completed at startup time.
  // This may be interesting for adding tab-completion a posteriori.
  cout << DATA_BEGIN << "command:";
  cout << "(plugin-configure complete (:tab-completion #t))";
  cout << DATA_END;
}

In the main loop, we first deal with regular input:

char buffer[100];
cin.getline (buffer, 100, '\n');
if (buffer[0] != DATA_COMMAND) {
  cout << DATA_BEGIN << "verbatim:";
  cout << "You typed " << buffer;
  cout << DATA_END;
}

We next treat the case when a tab-completion command is sent to the application:

else {
  cout << DATA_BEGIN << "scheme:";
  cout << "(tuple \"h\" \"ello\" \"i there\" \"ola\" \"opsakee\")";
  cout << DATA_END;
}
cout.flush ();

As you notice, the actual command is ignored, so our example is really very rudimentary.

9.Dynamic libraries

Instead of connecting your system to TeXmacs using a pipe, it is also possible to connect it as a dynamically linked library. Although communication through pipes is usually easier to implement, more robust and compatible with gradual output, the second option is faster.

In order to dynamically link your application to TeXmacs, you should follow the TeXmacs communication protocol, which is specified in the following header file:

    $TEXMACS_PATH/include/TeXmacs.h

In this file it is specified that your application should export a data structure

typedef struct package_exports_1 {
  char* version_protocol; /* "TeXmacs communication protocol 1" */
  char* version_package;
  char* (*install) (TeXmacs_exports_1* TeXmacs,
                    char* options, char** errors);
  char* (*evaluate) (char* what, char* session, char** errors);
} package_exports_1;

which contains an installation routine for your application, as well as an evaluation routine for further input (for more information, see the header file). TeXmacs will on its turn export a structure

typedef struct TeXmacs_exports_1 {
  char* version_protocol; /* "TeXmacs communication protocol 1" */
  char* version_TeXmacs;
} TeXmacs_exports_1;

It is assumed that each application takes care of its own memory management. Hence, strings created by TeXmacs will be destroyed by TeXmacs and strings created by the application need to be destroyed by the application.

The string version_protocol should contain "TeXmacs communication protocol 1" and the string version_package the version of your package. The routine install will be called once by TeXmacs in order to initialize your system with options options. It communicates the routines exported by TeXmacs to your system in the form of a pointer to a structure of type TeXmacs_exports_1. The routine should return a status message like

    "yourcas-version successfully linked to TeXmacs"

If installation failed, then you should return NULL and *errors should contain an error message.

The routine evaluate is used to evaluate the expression what inside a TeXmacs-session with name session. It should return the evaluation of what or NULL if an error occurred. *errors either contains one or more warning messages or an error message, if the evaluation failed. The formats being used obey the same rules as in the case of communication by pipes.

Finally, the configuration file of your plug-in should contain something as follows:

(plugin-configure myplugin
  (:require (url-exists? (url "$LD_LIBRARY_PATH" "libmyplugin.so")))
  (:link "libmyplugin.so" "myplugin_exports" "")
  further-configuration)

Here myplugin_exports is a pointer to a structure of the type package_exports_1.

Remark 6. It is possible that the communication protocol changes in the future. In that case, the data structures TeXmacs_exports_1 and package_exports_1 will be replaced by data structures TeXmacs_exports_n and package_exports_n, where n is the version of the protocol. These structures will always have the abstract data structures TeXmacs_exports and package_exports in common, with information about the versions of the protocol, TeXmacs and your package.

The dynlink plug-in

The dynlink plug-in gives an example of how to write dynamically linked libraries. It consists of the following files:

    dynlink/Makefile
    dynlink/progs/init-dynlink.scm
    dynlink/src/dynlink.cpp

The Makefile contains

tmsrc = /home/vdhoeven/texmacs/src/TeXmacs
CXX = g++
LD  = g++

lib/libtmdynlink.so: src/dynlink.cpp
        $(CXX) -I$(tmsrc)/include -c src/dynlink.cpp -o src/dynlink.o
        $(LD) -shared -o lib/libtmdynlink.so src/dynlink.o

so that running it will create a dynamic library dynlink/lib/libdynlink.so from dynlink.cpp. The tmsrc variable should contain $TEXMACS_PATH, so as to find the include file TeXmacs.h. The configuration file init-dynlink.scm simply contains

(plugin-configure dynlink
  (:require (url-exists? (url "$LD_LIBRARY_PATH"
                              "libtmdynlink.so")))
  (:link "libtmdynlink.so" "dynlink_exports" "")
  (:session "Dynlink"))

As to the C++ file dynlink.cpp, it contains a string

static char* output= NULL;

with the last output, the initialization routine

char*
dynlink_install (TeXmacs_exports_1* TM, char* opts, char** errs) {
  output= (char*) malloc (50);
  strcpy (output, "\2verbatim:Started dynamic link\5");
  return output;
}

the evaluation routine

char*
dynlink_eval (char* what, char* session, char** errors) {
  free (output);
  output= (char*) malloc (50 + strlen (what));
  strcpy (output, "\2verbatim:You typed ");
  strcat (output, what);
  strcat (output, "\5");
  return output;
}

and the data structure with the public exports:

package_exports_1 dynlink_exports= {
  "TeXmacs communication protocol 1",
  "Dynlink 1",
  dynlink_install,
  dynlink_eval
};

Notice that the application takes care of the memory allocation and deallocation of output.

10.Miscellaneous features

Several other features are supported in order to write interfaces between TeXmacs and extern applications. Some of these are very hairy or quite specific. Let us briefly describe a few miscellaneous features:

Interrupts

The “stop” icon can be used in order to interrupt the evaluation of some input. When pressing this button, TeXmacs will just send a SIGINT signal to your application. It expects your application to finish the output as usual. In particular, you should close all open DATA_BEGIN-blocks.

Testing whether the input is complete

Some systems start a multiline input mode as soon as you start to define a function or when you enter an opening bracket without a matching closing bracket. TeXmacs allows your application to implement a special predicate for testing whether the input is complete. First of all, this requires you to specify the configuration option

(:test-input-done #t)

As soon as you will press ↩ in your input, TeXmacs will then send the command

DATA_COMMAND(input-done? input-string)↩

Your application should reply with a message of the form

DATA_BEGINscheme:doneDATA_END

where done is either #t or #f. The multiline plug-in provides an example of this mechanism (see in particular the file multiline/src/multiline.cpp).

11.Writing documentation

Documentation for your plug-in myplugin should be put in the doc subdirectory of the main directory myplugin. We recommend to write at least the following three documentation files:

myplugin.en.tm

This file should mainly contain a traverse tag with links to the other documentation files, as described in the section “traversing the TeXmacs documentation”.

myplugin-abstract.en.tm

This file should contain a short description of the purpose of the plugin-in. If appropriate, then you should also describe how to get the plug-in and how to install it. The contents of this file should also be suitable for publication on the web site of TeXmacs.

myplugin-demo.en.tm

This file should contain a short demonstration of your plug-in, such as an example session.

The first two files are mandatory, if you want your plug-in to show up in the Help→Plug-ins menu. Please refrain from putting too many images in the documentation files, so as to keep the size of the documentation reasonable when integrated into the main TeXmacs distribution.

12.Plans for the future

There are many improvements to be made in the TeXmacs interface to computer algebra systems. First of all, the computer algebra sessions have to be improved (better hyphenation, folding, more dynamic subexpressions, etc.).

As to interfaces with computer algebra systems, out main plans consist of providing tools for semantically safe communication between several system. This probably will be implemented in the form of a set of plug-ins which will provide conversion services.