Paul Howson’s Website tdgq.com.au

A Film Restoration Diary

All Posts for This Blog (newest first)

Scanning and Restoring The Witness — Part 3

“The Witness” is a short, black and white film drama made at GTV9 circa 1960 by Denzil Howson and Mike Browning. The third in a set of posts completes the scanning and restorating this film.

The Film: “Yoga And The Individual” from 1966

In 1966, Melbourne film maker John Murray undertook to make a documentary film about the subject of Yoga. The film is called “Yoga and the Individual” and it won a Silver Award at the 1966 Australian Film Institute awards. This post discusses some of the film’s history and the recent scanning and restoration of an original copy.

Scanning and Restoring The Witness — Part 2

“The Witness” is a short, black and white film drama made at GTV9 circa 1960 by Denzil Howson and Mike Browning. The second in a set of posts about scanning and restorating this film.

Scanning and Restoring The Witness — Part 1

“The Witness” is a short, black and white film drama made at GTV9 circa 1960 by Denzil Howson and Mike Browning. The first in a set of posts about scanning and restorating this film.

Approaching the Soundtrack

With the video file of Marianne stabilised, dust-busted and awaiting only level adjustments, it’s time to turn attention to the soundtrack and the improvements which can be made there.

Dust-Busting with Loki

The Digital Vision Loki software at CameraQuip in Melbourne proved to be an inexpensive way to reduce dust marks on the Marianne video.

Situation Review After Two Years of Silence

This blog has been silent for two years, but progress on this restoration project has been on-going, albeit with some significant pauses due to other demands on my time.

As of April 2017 we had reached a satisfying conclusion to the problem of stabilising the jitter in “Marianne”, the 1963 Tarax Show Christmas Pantomime which is the subject of my current focus.

For practical purposes the jitter was eliminated, but the problem of dust and scratches remained.

Home-Grown Jitter Stabilisation — Part 7 — The Stabilisation Works!

Having spent the previous six posts explaining the jitter stabilisation process for the 1963 kine of Marianne, let’s take a look at how well it works.

So that you can see the jitter more clearly, I’ve prepared a 30 second sample video which shows a side-by-side, before-and-after comparison.

As explained earlier, this kine was scanned at 2K resolution, which is 2048 x 1556 pixels. The jitter detection and stabilisation was also done at this full resolution for maximum accuracy. However an entire 2K frame is far too large to show in this web page, so the sample video shows only the four corners of each frame. A 108 pixel by 108 pixel sample was taken from each of the four corners of each scanned frame and these have been put together into a composite image.

Home-Grown Jitter Stabilisation — Part 6 — A Stabilisation Workflow

For those of you not completely bamboozled by this excursion into the very-technical, here is a summary of the whole stabilisation process developed for the Marianne kine scan.

  • Convert the 2K scanned video into a series of tif files, one file per frame. There are 79675 frames, the dimensions of each frame being 2018 x 1556 pixels.
  • Divide these frames into sequences roughly corresponding to scene changes in the production. Each sequence typically 5–10 minutes duration. Some very short sequences used for fixing “splice bumps”.
  • Use this author’s custom software to analyse the frames and find an optimum reference frame for each sequence. This is one where the displacement of the frame is closest to the median (i.e. most common) value — i.e. it’s a “middle of the road” frame with respect to jitter.
  • The software compares each frame in the sequence with the reference frame, uses image correlation to determine the corner dot movements and calculates the displacement, rotation and vertical scaling of each frame relative to the reference frame.
  • An image transform is then used to reverse these distortions, thus cancelling out the jitter (not perfectly but almost). At the same time each frame is reduced in size from 2K (i.e 2048 x 1556 pixels) to PAL SD+ (720 x 576 pixels plus overscan).
  • The resulting corrected and downsampled frame images are saved to disk and then re-combined into a “stabilised” video file at PAL SD+ resolution.

That’s it in a nutshell, although in practice there were many technical challenges to overcome in making this work and the whole exercise ended up taking many months of part-time work — as usual with these things more than I initially expected!

Home-Grown Jitter Stabilisation — Part 5 — Odds and Ends

In this post we look at a few odds and ends that arose during the stabilisation exercise: Dealing with frame defects and splice bumps, adding frame numbers, and re-combining frames into a video file.

Dealing with Frame Defects

An unexpected challenge which arose when stabilising the thousands of frames in each sequence, was that some frames had defects which obscured one or more white corner dots and made it impossible to measure the movement of these corner dots.

Home-Grown Jitter Stabilisation — Part 4 — A Bit of Geometry and Maths

The last post described using image correlation to measure the motion of the white dots in each of the four frame corners. These white dots move together with the video raster. Stabilising the white dots will stabilise the unwanted movement or jitter of the video raster.

The following graphic used in an earlier post demonstrates again the three kinds of motion and the red arrows show the corresponding offsets of the corner dots.

Home-Grown Jitter Stabilisation — Part 3 — Image Correlation to Detect Movements

In the last couple of posts dealing with “Home-Grown Jitter Stabilisation”, we established:

  • The jitter comprises displacement, rotation and vertical scaling.
  • There is a white dot (partially) visible in each of the four corners of each scanned film frame. These dots move together with the video raster.
  • If we can measure the frame-to-frame movement of these dots, we can calculate and then correct (i.e. remove) the jitter.

So now we come to the topic of how to measure the frame-to-frame movement of these white dots.

Home-Grown Jitter Stabilisation — Part 2 — Three Kinds of Motion

Let’s put some visuals to this idea of stabilising the jitter in the kine scans.

As I mentioned in the last post, the jitter actually comprises three kinds of motion:

  • Displacement
  • Rotation
  • Vertical stretch

The following three diagrams illustrate each kind of motion. To make them clearly visible, the movements in these illustrations are proportionally much larger than in the actual film frames.

For each kind of motion, the little red arrows show the resulting movements of each corner artefact — the white dots which are stable relative to the video raster.

Home-Grown Jitter Stabilisation — Part 1 — Could it be Done?

As mentioned in the previous post, Promoscape had been unable to deal with the jitter: “With regard to stabilisation, we can’t find a cost effective and consistent way to achieve this with the footage.”

They explained that the jitter was not just horizontal and vertical movement (displacement) but included rotation as well.

I’ve talked about possible sources of this jitter in the post Assessing the Scans Part 2 — Jitter. There are a number of possible contributors to the jitter seen in the scans. Here again is the sample of 50 frames from Marianne (1963):

More Digital Restoration Tests with Promoscape in Melbourne

Twice before we’ve looked at digital restoration options:

Each of these tests produced promising results, but the cost of a full-length restoration was prohibitive.

Were there any other options? A bit of digging turned up another software product — “Diamant” from HS-Art in Austria — more or less the European alternative to PFClean. That led me to Promoscape in Melbourne, who were providing digital film restoration services using Diamant.

Adjusting Framing and Levels in Final Cut Pro

In the previous post we established the downsampling ratio for the 2K scans of “Marianne” (1963) which would result in a PAL SD frame — after cropping off the overscan.

Downsampling the 2K Scan

Why downsample the video at this stage rather than doing positional adjustments, grading and repairs at 2K resolution? The fact is, there’s not really any benefit in retaining all that resolution when the original analog video raster recorded to film had a resolution of roughly 300 lines vertically by perhaps 500 lines horizontally (see earlier posts for a more detailed discussion of this). And working with 2K scans does slow things down, even with a very fast modern computer and an external Thunderbolt hard drive.

Finding the Smallest Video Raster Crop Size for Downsampling Calculations

In the earlier post[1] about downsampling the kine scans from 2K to “PAL SD+”, we established the need to know the minimum dimensions of the video raster, which varies slightly in width and height over the duration of a reel. The final cropping boundary for a PAL SD video needs to fall within the minimum video raster size so that we get nice clean edges on the video frames.

This required scrubbing through the 2K scan and measuring the video raster dimensions at the locations where the raster seems to have a noticeably smaller size. The following table contains some measurements for “Marianne” (1963):

Where Have We Been and What’s Next?

Wow, we’re up to the 23rd post in this blog series on restoring the Tarax Show Christmas Pantomime kines. Let’s take a breather and review where we’ve been so far and then map out the next steps.

Where We’ve Been So Far

Here is a chronological list of the posts so far, grouped under headings. Each title is linked to its post (in case you want to go back and read any you missed).

The Relevance of Pixel Aspect Ratio

Congratulations if you got through the previous rather technical post about Pixel Aspect Ratio.

The next question is: how does this Pixel Aspect Ratio business affect the process of preparing these kine scans for DVD release?

Pixels Are Inherently Shapeless

When stored in a digital file on disk or DVD, a video frame is just a series of numbers, where each number (or set of numbers) describes the brightness and/or colour values of a pixel — a video frame being a 2-dimensional array of pixels.

The Mysteries of Pixel Aspect Ratio

I mentioned in the previous post that we would talk about Pixel Aspect Ratio (PAR). In this post we’ll talk about the historical background and in the next post we’ll look at how it affects video downsampling for this project.

Wikipedia provides a good introduction to PAR:

Most digital imaging systems display an image as a grid of tiny, square pixels. However, some imaging systems, especially those that must be compatible with standard-definition television motion pictures, display an image as a grid of rectangular pixels, in which the pixel width and height are different. Pixel Aspect Ratio describes this difference.

The Mathematics of Downsampling the High Definition Scans for PAL SD

In the previous post we established the need to downsample the high resolution 2K scans to a lower resolution for PAL SD release. We need to adjust this downsampling so that:

  1. The overscan (i.e. the area of the film frame beyond the boundaries of the video raster) is preserved.
  2. The dimensions of the video raster (after downsampling) are slightly larger than the PAL SD frame size of 720 x 576 pixels. This will allow final cropping to remove a little “bleed” around the edges of the video raster to produce a PAL SD frame in which the video raster (i.e. the original television image) neatly fills the frame.

Preserving the overscan implies that the downsampled video will have a larger frame size than PAL SD — in fact a non standard frame size which we will refer to as PAL SD+.

Editing Tools, Grading and When to Downsample

As I write this blog post (November 2015), I’m still covering work done a year or more ago. At the time I didn’t think to write about it. Looking back though it seemed an interesting journey.

So let’s see, we’ve covered preparatory work before scanning, various attempts and experiences with scanning the kines, some more successful than others, an analysis of the scans and some experiments with digital restoration.

Choosing a Tool for Grading the Scans

For DVD or internet release the scans will need “grading” (adjustment of grey levels), adjustment of positional errors (explained below) and the odd bit of editing and frame repair. That means a digital editing tool, and the obvious choice was Final Cut Pro X (FCPX) from Apple.

Experiments in Digital Restoration with PFClean

Some years ago there was a television series called “The Second World War in Colour”. It featured seldom seen colour film footage taken during the war. In the introduction to the accompanying book (of colour photographs) the author remarks that there is a widespread and unconscious perception that “…the war was fought in black and white” — a result of the fact that most of the historical newsreel footage and photographs are in black and white.

Mapping out a Restoration Roadmap

Having assessed the four Tarax Show Christmas Pantomime kine scans, it is time to plan out a restoration roadmap.

Restoration Steps

The steps now required to restore these scans ready for release on DVD are (not necessarily in this order):

  • Stabilisation of jitter if possible and practical. Tests have shown jitter can be most accurately measured and corrected at high resolution.
  • Downsampling of the scanned frames to PAL SD+ resolution — this means PAL SD resolution (720 x 576 pixels) plus allowance for the overscan.
  • Grading of the video levels so that white peaks and shadow detail are not clipped — i.e not too much contrast. At the same time, we don’t want the result to be “flat”. We want want to utilise the full “broadcast safe” tonal range.
  • Reduction of film artefacts: dust and scratches if possible and practical.
  • Cropping to PAL SD resolution which is required for DVD release.
  • Repair and equalisation of the soundtrack.
  • Mastering for DVD or release on the web.

In the next post, we’ll try out some commercial film restoration software and see how good a job it does at stabilisation and dust and scratch removal.

Re-Scanning The Golden Princess (1962)

In the previous four posts, we assessed four aspects of the Tarax Show Christmas Pantomime kine scans:

  • Anatomy of a Kine Frame
  • Jitter
  • Video Levels
  • Soundtrack

The most significant problems were jitter and dust and scratches. We will look at ways to reduce the effect of these in later blog posts.

However, the scan of The Golden Princess done on the Spirit Datacine suffers rather badly from some of the defects to which “line scanners” like the Spirit are susceptible.

Assessing the Scans Part 4 — Soundtrack

In assessing these kine scans we come finally and briefly to the sound tracks. Each kine has an optical soundtrack, some are variable area, others variable density.

Optical soundtracks on 16mm film are not exactly “high fidelity” audio. The two most significant limitations on quality are:

  • The frequency range. The soundtrack is recorded linearly along the edge of the film. The limitation on frequency response is the shortest wavelength which can be recorded and reproduced. When playing back an optical soundtrack on a traditional projector, the soundtrack passes over a very fine slit with a light source behind it. A detector on the other side of the film picks up the fluctuations in light intensity. The width of this slit determines the shortest wavelength which can be reproduced. It is generally accepted that 16mm optical soundtracks can reproduce frequencies up to about 6kHz.
  • Film grain, dust, etc. With the audio signal effectively being recorded onto and reproduced from a very thin slice of the film, film grain becomes a source of noise. Dust and scratches cause audible clicks, pops and dropouts. Hence optical soundtracks have a poor signal to noise ratio.

      • Given these limitations, the soundtracks on these kines are not too bad. The scans made on the Spirit Datacine need some equalisation to boost the high frequencies in the 2kHz to 6kHz range.

Assessing the Scans Part 3 — Video Levels

This post discusses video levels in the four kine scans. For the purposes of this discussion, we will talk about luminance values only since the Tarax Show scans are black and white and the colour information has been removed from the scans.

First, let’s look at a screenshot of a “video scope” and the corresponding frame, as displayed in Final Cut Pro.

A frame from one of the kine scans together with a video scope display that plots the luminance values in each scan line.

This kind of display has been around since the very early days of television — engineers used them to adjust camera video levels to ensure they fell within the required broadcast-safe range. In those days it would have been a CRT screen displaying analog voltages representing video levels.

Assessing the Scans Part 2 — Jitter

What I’m calling “jitter” is unwanted rapid small movements of the video image vertically and horizontally caused by the mechanical processes of recording to and reproducing from film.

By scanning an area of the kine film frame beyond the edges of the video raster, one can get some insights into the sources of jitter.

Jitter on the Spirit Datacine

The Spirit Datacine seems to be particularly susceptible to jitter. It scans the film while it is moving using a “line scanner” and attempts to detect the location of each frame by sensing the position of the sprocket holes. This mechanical sprocket hole detection is not perfect and small positional errors can occur in the direction of film motion. This is one cause of vertical jitter.

Assessing the Scans Part 1 — Anatomy of a Kine Frame

With four scans under our belt, let’s take a look at them in more detail as an aid to planning how best to restore them for DVD release.

The first thing to note is that all four scans have “2K” resolution — 2048 pixels wide by 1556 pixels high. That’s a lot more resolution than is needed to reproduce the detail on the film.

As explained in an earlier post1:

Dealing with the Vinegar Syndrome Reel

As I mentioned in the previous post, one kine reel had a noticeable “vinegar syndrome” smell. The Australian National Film and Sound Archive describes the problem:

“Vinegar syndrome is a term used to describe the chemical reaction that goes on during the deterioration of cellulose triacetate film support. The presence of the odour does not mean the film has degraded, but rather that the reaction is taking place.”

Vinegar syndrome causes shrinkage of the film base — this appears to have happened to the 1959 kine of “Merry Make-Believe”. The shrinkage had also caused the film to twist. Shrinkage was the most likely reason it would not play on a 16mm projector without constantly losing registration with the pull-down claw.

Collecting and Inspecting the 2K Scans at Cutting Edge

It was with excited anticipation that I returned again to Cutting Edge to see first hand the high resolution scans of the Tarax Show kines. With high quality, high resolution scans from the film, the project to restore these productions for DVD release could now move forward.

Justin had scanned all the reels sent, except for “Merry Make Believe” (1959), which was warping and jumping off the Spirit Datacine motion-sensing sprocket and losing registration. This reel had noticeable “vinegar syndrome” — a form of chemical degradation that affects acetate film bases and causes the material to shrink and warp. I would need to find another scanning solution for that reel (more about that in a later post).

Preparing the Films for Scanning on the Spirit

Having found a Spirit Datacine in Brisbane and tested some of the Tarax Show kines, it was time to prepare the full length 16mm reels for scanning.

Of the five Tarax Show Christmas Pantomimes that I had fully on film, I chose four for scanning: 1959, 1961, 1962 and 1963.

In addition to the full reels, there were some cans marked “offcuts” which suggested that pieces had been excised for use in the “Magic Box” show about early television which Denzil Howson presented in his later years.

Cutting Edge Brisbane to the Rescue with a Spirit Datacine

The discovery of a Spirit Datacine at Cutting Edge in Brisbane, only two hours away, was a relief after the at-a-distance negative experience with the Sydney company. This time I could see the equipment for myself, and discuss in person what was needed.

So in March 2013 I arranged to visit Justin at Cutting Edge, to see the Spirit machine up close, and to discuss the process for having films scanned.

Seeking Telecine Services in a Post-Film Era

When the Sydney telecine company failed to deliver a useable scan, the search was on again for a quality film scanning service.

I called The Post Lounge again. They told me they were closing their Gold Coast facility and not running the Spirit Datacine anymore. They were planning to sell it. Apparently there was very little production happening at the Gold Coast and future productions would probably be all-digital, without using any film at all. The bottom line was: they were out of the film and telecine business.

Troubles in Telecine Land — A Lesson in Not Believing Advertising Hype

It was not until 2012 that I was able to find time to arrange telecine transfers of the full length reels of the Tarax Show pantomimes.

However, things had changed drastically in telecine land since the Spirit Datacine test done by Larry at The Post Lounge in 2008.

Firstly, Larry had departed some years earlier and had not been replaced by another full-time telecine operator. The Post Lounge were now operating their Spirit Datacine on an as-needed basis using staff from their Brisbane office.

Exploring the Possibilities of Digital Restoration

The black and white television equipment installed at GTV9 in the late 1950s delivered remarkably good picture quality. This was especially so within the studio complex itself where the engineers maintained the equipment to a high standard and sought the best possible picture quality.

Before the arrival of videotape in 1959, making “kines” to 16mm film was the only way of preserving a broadcast. The use of film as an archival and/or distribution medium continued well into the 1960s.

Telecine — Initial Experiments and the Spirit Datacine

Around the same time as Studio One at GTV9 was transformed into Toyland for the last Tarax Show Christmas Pantomime, the BBC in London was getting ready to launch a new childrens’ television series about a curious character who travels through time and space in a police telephone box. Dr Who was also (mainly) a studio-based production, recorded to videotape and subsequently copied to film for distribution to television stations around the world.

About the GTV9 Film Recording System Circa 1960

In this post we’re going to get a bit more technical and look into how these film copies (“kines”) were made from the original television productions. We’re going to delve into some technical history of film recordings at GTV9 circa 1960.

I would like to acknowledge the assistance of former GTV9 engineers Ian Douglas and Andrew McKean in providing technical details of the film recording system.

Technical Parameters of the Australian Television System Circa 1960

The Australian black and white television system circa 1960 delivered 25 television “frames” per second. Each television frame had 625 scan lines. These 625 scan lines were split between two “fields”, with two successive fields making up each frame (hence 50 fields per second). Consequently each field contained only half of the 625 scan lines for each frame — i.e. around 300 scan lines per field. The relevance of this will become clear in a moment.

Compiling a List of Films and Their Condition

In the previous post I gave a brief history of the Happy/Tarax Show annual Christmas Pantomimes. The surviving film copies of these are the subject of this current restoration project.

Each one of the seven Christmas Pantomimes produced annually between 1957 and 1963 was recorded onto 16mm film. The resulting film copy is known as a “kinescope” or “kine”.

For the 1957 and 1958 productions, recording to film was done at the time of live broadcast. With the coming of videotape circa 1959, it became possible to pre-record programs for subsequent broadcast, and the later film recordings of the Pantomimes were made from videotape.

Preserving and Restoring the Tarax Show Christmas Pantomimes

In 1956 the Olympic Games ushered Melbourne into the television era. My father (Denzil Howson) had joined GTV9 as Assistant Programme Manager in September that year and we were one of the rare households in those early days with a TV set. It was an American “Admiral” set and it cost a lot of money. The Happy Show premiered on 19th January 1957 — just two days after GTV9 began regular transmissions.

About this Film Restoration Diary

Having grown up in a family where filmmaking and film technology were a regular presence, I absorbed an interest in filmmaking and the technology of film at an early age.

After working for many years in the field of graphic design and later web design, I inherited a collection of films made by my late father, some of which are of historical significance, in particular those dating from his work in the early days of Australian television circa 1956 to 1963.