Fredrik from Utrecht in the Netherlands wrote to me in 2010 about publishing workflows. His experiences are similar to mine, so I quote from his email and add a commentary. I had explained to Fredrik about my background in publishing and talked about the typical problems experienced with publishing workflows. Fredrik’s comments are shown indented below:
My background is also in publishing, so I indeed know the problems you are talking about. I have worked with a number of different workflows, automated formatting and manual formatting, xml as source and Word as source. With all the solutions that used Word I saw problems, and typing xml is too much to ask for. I have seen cases where the authors write their articles in Word and others are hired to manually convert it into xml. It may well be the best option currently, but it should be able to improve the situation using a better editor.
I can only agree regarding Word and xml — the shortcomings of both are discussed in other posts on this blog.
Fredrik mentions the option of assistants manually converting Word documents into xml. What these people are really being asked to do is to manually interpret the document’s structure based on various hints the authors have placed in the Word file (use of local format overrides, use of styles, maybe even some kind of text coding system, etc) and ignoring the formatting “junk” that is not structurally relevant. Then they manually code the document structure using xml.
This is the point in the workflow where a smarter manually-assisted tool is needed to quickly interpret structure and eliminate junk. That’s my first goal with the structured editor project.
I expressed my opinion to Fredrik that even though modern word processors support the mechanism of paragraph styles, a “wrong turn” was taken in the early days of programs such as MacWrite and Word by allowing arbitrary local formatting overrides anywhere in a document.
Paragraph styles are not enough to fix the “wrong turn” that was taken by earlier word processors. If only styles are used it comes close to a useful approach, but because of the local formatting that is still available, it’s still too easy to make a mess. And in Word most users cannot see the difference between something formatted using styles or local markup, and really, you can’t blame them.
Once again, the inability to distinguish between local formatting and styles is highlighted as a limitation with conventional word processors, with Microsoft Word being the most widely used.
I suggested to Fredrik that we need to accept the reality of Word as an authoring tool, but could create a different tool that made the “correct structuring” of a document easier.
I agree with you that it is unrealistic to come up with a total replacement of Word. Some people will still use Word, and even if you can get them to write directly into your application, there is a great amount of legacy content that will be pasted into your application. What is your take on this? Is the main goal for your application to convert Word content into something more useful, or is the primary use of the application to actually write in it?
“Legacy content” means anything you did not create or whose creation you did not control. As publishers we deal with that frequently of course. And it’s at this point of receiving and “digesting” content supplied by others that there seems to be the most pain. It’s the step for which I have sought a tool for nearly twenty years, and have so far not been able to find one which understands the nature of this problem. That’s why I started the Structured Editor project.
I told Fredrik that the initial goal is quick and accurate structuring of unstructured content — because I think that’s the weakest point in current workflows.
In my research I looked at how people work using Word, and using a so-called “user-friendly” xml editor called Xopus.
Fredrik then supplied some data from his research.
I recently finished my interviews with the Word authors, and some interesting issues came up so far:
- Real WYSIWYG is not required. As long as a header is somewhat bigger and bold, and other distinct elements are formatted differently, the authors are satisfied.
This is a valuable insight. It suggests that what authors are looking for is visual feedback about structure, not a typographically perfect rendering of their final document. My own experience confirms this.
- A majority of authors working for publishers used codes between square brackets to indicate certain elements. This didn’t surprise me, the interesting thing however is that authors did not receive instructions to use codes in this way. They believe this is a good approach and like working this way.
What Fredrik appears to be saying here is that some authors invent their own coding system (i.e. a markup language), when one is not provided, in order to bring clarity to the process of structuring a document. This also suggests that manually adding markup to a document is acceptable to authors provided it is human-friendly. On this last point it is worth noting that although xml is promoted as a way to mark up text for publishing, it is definitely not a human-friendly markup system (and the same applies for html).
- It should be kept in mind that most authors do not work linearly. When I look at the most xml editors available today, I get the feeling they are built with a user in mind that finishes a section, clicks a button to create a new section, and starts typing this section. This is of course not the actual practice, as authors often add and remove headings and change the structure many times. The editor should be better suited for these use-cases.
Fredrik raises one of the major problems with typical xml editors as authoring tools — they impose a structure straightjacket too early in the process, at a time when flexibility is essential.
Word is actually better in this regard because it is more free-form. Block level structure in a Word document is inferred from the use of heading styles. Because there is no rigid container-based structure typical of xml, it is very easy to move things around and promote and demote the “level” of headings — thus changing the inferred hierarchical structure.
- Finally, I have my doubts about the hierarchical document structure that is used by most, if not all, xml document schemas. From a document engineering perspective, it may seem logical to use all sorts of nested sections. But this complicates editing these documents, as authors do not regard documents as hierarchically structured, and changing the structure is more complicated compared with a flatter structure.
Fredrik further clarifies this practical deficiency of xml as an authoring system, even if it is theoretically elegant. As he says, schemas come from a “document engineering” perspective. Remember that xml, and its predecessor sgml, were invented so that computers could understand and manipulate the structure and content of documents — for example, to format it in a consistent way, or to translate it’s representation into another form. Schemas arise from an engineering mind-set which seeks to impose rigid, mechanical constraints on document structure.
Authors and editors on the other hand are more likely to be of an artistic temperament with a rather different mind-set to the engineer. What they want in a tool is maximum flexibility and minimum constraints. A Fredrik says “authors do not regard documents as hierarchically structured”.
For all their faults, word processors like Microsoft Word appeal because they are very flexible. They offer a way to structure documents through styles, lists, tables and so on, while preserving as much flexibility as possible. Perhaps with the exception of content inside tables, all these elements of a Word document are “first class citizens” and can be moved around freely without any constraining system of hierarchical containment.
For me it’s still a bit vague what you’re planning to do, although I do get the reasoning behind it. Not more complexity than needed for the type of document and equal importance to structure and content. Are you planning to visualize this structure in some way?
I replied to Fredrik that visualising document structure is crucial. But it has to be done in a way that is not complicated or visually over-busy. I’m fortunate in having a background in graphic design as well as programming, because the UI design is very important.
In my view very few editing tools get the UI right. They visually bamboozle people — just look at Word’s toolbars — most users wouldn’t have a clue what all those buttons do. And xml editors — like Oxygen — are just as bad, if not worse.
From my point of view, this conversation confirmed that I’m on the right track with the concept of a structured editor. It also confirmed my own experience that Word is not the ideal tool and neither are most xml editors.
In a following post, I’ll elaborate on why document designers need an editing tool which is different from what authors need.