I picked up Ken Williams’s book Not All Fairy Tales Have Happy Endings back in October and read it over my Thanksgiving break. I recommend it, especially if you’re interested in the history of video games or software. Even at 530 pages, it’s a very quick read, and I was able to get from start to finish in just a day. It’s a good complement to the Sierra chapters in Steven Levy’s book Hackers, which I also recommend for tech history buffs.
For those unfamiliar, Ken Williams founded Sierra On-Line in 1979 with his wife Roberta. After Roberta conceived of the game Mystery House for the Apple ][ (marking the invention of the graphical adventure genre), Ken programmed the game’s “engine” while Roberta fleshed out the game’s world, doing both the writing and illustrations. The pair marketed the game by demonstrating it at local computer stores, where store owners would thereafter offer the game for sale. Sierra grew from a husband-and-wife team into a multimillion-dollar powerhouse by the mid-to-late 1990s.
I’ve never met either of the Williamses. But in 2017, I exchanged a few e-mails with Al Lowe, one of Sierra’s most prolific game designers.
This story is mentioned briefly in Ken’s book (on page 205), but the longer version goes like this: Leading up to its launch in January 1983, Apple was hip-deep in development of their first computer built around a graphical user interface. Steve Jobs wanted to call this new computer the “Lisa,” after his daughter, but the legal folks at Apple quickly discovered that Sierra was offering an assembler for the Apple ][ with the same name. Trademark law being what it is, Apple couldn’t easily go ahead and use the “Lisa” name in the computer field without risking infringing on the trademark Sierra already owned.
So one day between 1981-1982, Ken Williams got a call from Apple. In exchange for the rights to use the “Lisa” name on Apple’s newest computer, Ken would receive several brand-new Lisa computers—each worth $10,000 at launch in 1983—as well as a few prototypes of a top-secret new machine in the works at Apple (this turned out to be the Macintosh). The Lisas arrived, and a few months later came the promised Mac prototypes, which Sierra used to develop Frogger, among other early Mac games.
Along with four Mac prototypes came a Picasso-style “Macintosh” lamp, usually designated for Mac dealers. Ken was not exactly known for keeping a tidy office (I can relate), so the Macintosh lamp became one of many items scattered about his space. Several months later, Al was in Ken’s office, noticed the lamp on his floor, and inquired about it… only to receive the shock that Ken was just going to throw it away. Al was able to successfully begnegotiate for the lamp, which found a new home in his personal office, where it lived for almost 33 years.
Al offered up the lamp for sale in 2017, and given its history, I bought it. It was on my desk at work for a while but now lives in my office perched—appropriately—atop my Lisa.
I promised Al and myself that the lamp would stay in the games industry. So far I’ve kept my word, and I foresee no reason to break that promise any time soon.
It’s astounding to think back and consider how much technological progress has occurred in just the past 15 years. Most folks today carry a smartphone in their pocket everywhere they go, and a great many of those smartphones have powerful cameras built in capable of recording multiple hours in high definition. Pair this ability with low-cost video editing software—some of which comes at no cost at all—and far more people today have the tools to practice shooting, editing, compositing, and rendering professional-looking videos on a modest budget.
My personal experience with photography began around age 7 shooting on 110 film using a small “spy” camera I got as a gift. My dad’s Sony CCD-V5 was bulky, heavy, and probably expensive when he bought it around 1987, so he was reluctant to let me or my sister operate it under his supervision, let alone borrow it to make our own films by ourselves. As a consequence, my sister and I kept ourselves entertained by making audio recordings on much cheaper audio cassette hardware and tapes—we produced an episodic “radio show” starring our stuffed animals long before the podcast was invented. Though my sister and I took good care of our audio equipment, Dad stuck to his guns when it came to who got to use the camcorder, but he would sometimes indulge us when we had a full production planned, scripted, and rehearsed. Video8 tapes were expensive, too, and for the most part Dad reserved their use for important events like concerts, school graduations, birthdays, and family holidays.
I went off to college and spent a lot of time lurking the originaltrilogy.com forums. It was here that not only did I learn a lot about the making and technical background of the Star Wars films (a topic I could blog about ad nauseum), but I also picked up a lot about video editing, codecs, post-production techniques, and preservation. OT.com was and still is home to a community of video hobbyists and professionals, most of whom share a common love for the unreleased “original unaltered” versions of the Star Wars trilogy. As such, many tips were/are shared as to how to produce the best “fan preservations” of Star Wars and other classic films given the materials available, sacrificing the least amount of quality.
I bought my dad a Sony HDR-CX100 camcorder some years ago to supplement his by that time affinity for digital still cameras—he took it to Vienna and Salzburg soon after and has since transitioned to shooting digital video mostly on his iPhone. But the 8mm tapes chronicling my family’s milestones over the first 25 years of my life continued to sit, undisturbed, in my folks’ cool, dry basement. My dad has recordings on them going as far back as 1988 that I’ve found so far. These recordings are over 30 years old, so the tapes must be at least that age.
8mm video tape does not last forever, but making analog copies of video tape incurs generational loss each time a copy is dubbed. On the other hand, a digital file can be copied as many times as one wants without any quality loss. All I need is the right capture hardware, appropriate capture software, enough digital storage, and a way to play back the source tapes, and I can preserve one lossless digital capture of each tape indefinitely. The last 8mm camcorder my dad bought—a Sony CCD-TR917—still has clean, working heads and can route playback of our existing library of tapes through its S-video and stereo RCA outputs. This provides me with the best possible quality given how they were originally shot.
Generally with modern analog-to-digital preservation, you want to losslessly capture the raw source at a reasonably high sample rate with as little processing done to the source material as possible, from the moment it hits the playback heads to the instant it’s written to disk. Any cleanup can be done in post-production software; in fact, as digital restoration technology improves, it is ideal to have a raw, lossless original available to revisit with improved techniques. For this project, I am using my dad’s aforementioned Sony CCD-TR917 camcorder attached directly to the S-video and stereo audio inputs of a Blackmagic Intensity Pro PCIe card. The capturing PC is running Debian Linux and is plugged into the same circuit as the camcorder to avoid possible ground loop noise.
Since my Debian box is headless, I’m not interested in bringing up a full X installation just to grab some videos. Therefore I use the open source, command-line based bmdtools suite—specifically bmdcapture—to do the raw captures from my Intensity Pro card. I do have to pull down the DeckLink SDK in order to build bmdcapture, which does have some minor X-related dependencies, but I have to pull down the DeckLink software anyway for Linux drivers. I invoke the following from a shell before starting playback on the camcorder:
The options passed to bmdcapture configure the capture as follows:
-C 0: Use the one Intensity Pro card I have installed (ID 0)
-m 0: Capture using mode 0; that is, 525i59.94 NTSC, or 720×486 pixels at 29.97 FPS
-M 4: Set a queue size of up to 4GB. Without this, bmdcapture can run out of memory before the entire tape is captured to disk.
-A 1: Use the “Analog (RCA or XLR)” audio input. In my case, stereo RCA.
-V 6: Use the “S-Video” video input. The S-video input on the Intensity Pro is provided as an RCA pair for chroma (“B-Y In”) and luma/sync (“Y In”); an adapter cable is necessary to convert to the standard miniDIN-4 connector.
-d 0: Fill in dropped frames with a black frame. The Sony CCD-TR917 has a built-in TBC (which I leave enabled since I don’t own a separate TBC), but owing to the age of the tapes, there is an occasional frame drop.
-n 230000: Capture 230000 frames. At 29.97 FPS, that’s almost 7675 seconds, which is a little over two hours. Should be enough even for full tapes.
-f <output>.nut: Write to <output>.nut in the NUT container format by default, substituting the tape’s label for <output>. The README.md provided with bmdtools suggests sticking with the default, and since FFmpeg has no trouble converting from NUT and I’ve had no trouble capturing to that format, I leave the output file format alone.
Once I have my lossless capture, I compress the .nut file using bzip2, getting the file size down to up to a quarter of the original size depending on how much of the tape is filled. I then create parity data on the .bz2 archive using the par2 utility, and put my compressed capture and parity files somewhere safe for long-term archival storage. 🙂
My Windows-based Intel NUC is where I do most of my video post-production work. It lacks a PCIe slot, so I can’t capture there, but that’s fine because at this point my workflow is purely digital and I only have to worry about moving files around. My tools of choice here are AviSynth 2.6 and VirtualDub 1.10.4, but since AviSynth/VirtualDub are designed to work with AVI containers, I first convert my capture from the NUT container to the AVI container using FFmpeg:
The options passed to FFmpeg are order-dependent and direct it to do the following:
-i <output>.nut: Use <output>.nut as the input file. FFmpeg is smart and will auto-detect its file format when opened.
-vcodec copy: Copy the video stream from the input file’s container to the output file’s container; do not re-encode.
-acodec copy: Likewise for the audio stream, copy from the input file’s container to the output file; do not re-encode.
<output>.avi: Write to <output>.avi, again substituting my tape’s label for <output> in both the input and output filenames.
A note about video containers vs. video formats
Pop quiz! Given a file with the .mov extension, do you know for sure whether it will play in your media player?
Files ending with .mov, .avi, .mkv, and even the .nut format mentioned above are “container” files. When you save a digital video as a QuickTime .mov file, the .mov file is just a wrapper around your media, which must be encoded using one or more “codecs.” Codecs are small programs that can encode and/or decode audio or video. These codecs must be specified at the same time as when you save your movie. QuickTime files can wrap among a great many codecs: Motion JPEG, MPEG, H.264, and Cinepak just to name a few. They’re a bit like Zip files, except that instead of files inside you have audio and/or video tracks, and there’s no compression other than what’s already done by the tracks’ codecs. Though Apple provides support in QuickTime for a number of modern codecs, older formats have been dropped over time and so any particular .mov file may or may not play… even using Apple’s own QuickTime software! Asking for a “QuickTime movie” is terribly vague—a QuickTime .mov file may not play properly on a given piece of hardware if support for a containing codec is missing.
AVI, MKV, and MP4 are containers, too—MP4 is in fact based on Apple’s own QuickTime format. But these are still just containers, and a movie file is nothing without some media inside that can be decoded. Put another way, when I buy a book I’m often offered the option of PDF, hardcover, or paperback form. But if the words contained therein are in Klingon, I still won’t be able to read it. When asked to provide a movie in QuickTime or AVI “format,” get the specifics—what codecs should be inside?
Now that I have an AVI source file, I can open it in VirtualDub. Owing to its namesake, VirtualDub’s interface is reminiscent of a dual cassette deck ready to “dub” from one container to another. It isn’t as user-friendly as, say, Premiere or Resolve when it comes to editing and compositing, but what it lacks in usability it gains in flexibility. In particular, VirtualDub is designed to run a designated range of source video through one or more “filters,” encoding to one of several output codecs available at the user’s discretion via Video for Windows and/or DirectShow. If no filters are applied, VirtualDub can trim a video (and its audio) without re-encoding—great for preparing source footage clips for later editing or other processing.
Though the Sony CCD-TR917 has a built-in video noise reduction feature, I explicitly turn it off before capturing, because one of the filters I have for VirtualDub is “Neat Video” by ABSoft. It’s the temporal version of their “Neat Image” Photoshop filter for still images, which I used most recently to prepare a number of stills for Richard Moss’s The Secret History of Mac Gaming. It’s a very intelligent program that has a lot of knobs and dials to really tune in the noise profile you want to filter out, so I was equally delighted to find that ABSoft’s magic works on videos too. Luckily they offer a plugin built to work with VirtualDub, so I didn’t hesitate to buy it as a sure improvement over the mid-90s noise reduction technology built in to the camcorder.
Most of the aforementioned features can be done in high-end NLE applications such as Resolve—indeed I have used Resolve to edit several video projects of my own. What makes VirtualDub the “killer app” for me is its use of Windows’s built-in video playback library, and therefore its ability to work with AviSynth scripts. AviSynth is a library that can be installed on Windows PCs that grants the ability to interpret AviSynth “script” files (with the .avs extension) as AVI files anywhere Windows is prompted to play one using its built-in facilities. The basic AviSynth scripting language is procedural, without loops or conditionals, but it does retain the ability to work with multiple variables at runtime and organize frequently-called sequences into subroutines. Its most common use is to form a filter chain starting with one or more source clips, ending with a final output clip. When “played back,” the filter chain is evaluated for each frame, but this is transparent to Windows, which instead just sees a complete movie as though it’s already rendered to an AVI container.
Combined with VirtualDub, AviSynth allows me to write tiny scripts to do trims and conversions with frame-accurate precision, then render these edits to a final output video. Though AviSynth should be able to invoke VirtualDub plugins from its scripting language, I couldn’t figure out how to get it to work with Neat Video, so I did the next best thing: I created a pair of AviSynth scripts; one to feed to Neat Video, and one to process the output from Neat Video. The first script looks like this:
# Invoke Neat Video
Absent of an explicit input argument, each AviSynth instruction receives the output of the previous instruction as its input. The Neat Video plugin for VirtualDub expects its input to be encoded as 8-bit RGB. VirtualDub will automatically convert the source video to what Neat Video expects if not already in the proper format. Since I’m not sure exactly how VirtualDub does its automatic conversion, I want to retain control over the process so I do the conversion myself from YUV to RGB using the Rec.601 matrix. I know that my source video is from an interlaced analog NTSC source; VirtualDub doesn’t know that unless I explicitly say so.
I render this intermediate video to an AVI container using the Huffyuv codec. Huffyuv is a lossless codec, meaning it can compress the video without any generational loss. Despite its name, Huffyuv is perfectly capable of keeping my video encoded as RGB. I can’t do further AviSynth processing on the result from Neat Video until I load it into my second AviSynth script, so I’m happy that its output can be unchanged from one script to the next.
Color encodings and chroma subsampling
Colors reproduced by mixing photons can be broken down into three “primary colors.” We all learned about these in grade school: red, green, and blue. Red and blue make purple, blue and green make turquoise, all three make white, and so on.
On TV screens, things are a bit more complicated. Way back when, probably before you were born, TV signals in the United States only came in black and white, and TVs only had one electron gun responsible for generating the entire picture. The picture signal mostly comprised of a varying voltage level per 525 lines indicating how bright or dark the picture should be at that point in that particular line. The history of the NTSC standard used to transmit analog television in the United States is well-documented elsewhere on the Internet, but the important fact here is that in 1953, color information was added to the TV signal broadcast to televisions conforming to the NTSC standard.
One of the challenges in adding color to what was heretofore a strictly monochrome-only signal was that millions of black and white TVs were already in active use in the United States. A TV set was extremely expensive even by the early 1950s, so rendering all the active sets obsolete by introducing a new color standard would have proven quite unpopular. The solution—similar to how FM stereo radio was later standardized in 1961—was to add color as a completely optional, but still integral, signal of monochrome TV. The original black and white signal—now known as “luma”—would continue to be used to determine the brightness or “luminance” of the picture at any particular point on the screen, while the new color stream—known as “chroma”—would only transmit the color or “chrominance” information for that point. Existing black and white TVs would only know about the original “luma” signal, and so would continue to interpret it as a monochrome picture, whereas new color TVs would be aware of and overlay the new “chroma” stream on top of the original “luma” stream to produce a rich, vibrant, color picture. All of this information still had to fit into a relatively limited bandwidth signal, designed in the early 1940s to be transmitted through the air with graceful degradation in poor conditions.
The developers of early color computer monitors, by contrast, needed not worry about maintaining backward compatibility with black and white American television nor did they need to concern themselves with adopting a signal format that was almost 50 years old by that point. It should be of little surprise then, naturally, that computer monitors generate color closer to how we learned in grade school. Computer monitors in particular translate a signal describing separate intensities of red, green, and blue to a screen made up of triads of red, green, and blue dots of light. This signal describing what’s known as “RGB” color (for Red, Green, and Blue) comes both from that aforementioned color theory of mixing primaries, but also historically from those individual color signals more or less directly driving the respective voltages of three electron guns inside a color CRT. Despite both color TVs and computer monitors having three electron guns for mixing red, green, and blue primary colors, the way that color information is encoded before entering the TV is a main differentiator.
Whereas RGB is the encoding scheme by which discrete Red, Green, and Blue values are represented, color TV uses something more akin to what’s known as “YUV.” YUV doesn’t really stand for anything—the “Y” component represents Luma, and the “UV” represents a coordinate into a 2D color plane, where (1, 1) is magenta, (-1, -1) is green, and (0, 0) is gray (the “default” value for when only the “Y” component is present, such as on black and white TVs). In NTSC, quadrature amplitude modulation is used to convey both of the UV components on top of the Y component’s frequency—I don’t know exactly what quadrature amplitude modulation is either, but suffice it to say it’s a fancy way of conveying two streams of information over one signal. 🙂
An interesting quirk of how the human visual system works is that we have evolved to be much more sensitive to changes in brightness than in color. Some verysmart people have sciencey explanations as to why this is, but ultimately we can thank our early ancestors for this trait—being able to detect the subtlest of movements even in low light made us expert hunters. Indeed, when the lights are mostly off, most of us can still do a pretty good job of navigating our surroundings (read: hunting for the fridge) despite there being limited dynamic range in the brightness of what we can see.
Note that having a higher sensitivity to brightness vs. color does not mean humans are better at seeing in black and white. It merely means that we notice the difference between something bright and something dark better than we can tell if something is one shade of red vs. a different shade of red. In addition, humans are more sensitive to orange/blue than we are to purple/green. These facts actually came in very handy when trying to figure out how to fit a good-enough-looking color TV signal into the bandwidth already reserved (and used) for American television. Because we are not as sensitive to color, the designers of NTSC color TV could get away with transmitting less color information than what’s in the luma signal. By reducing the amount of bandwidth for the purple/green range, color in NTSC can still be satisfactorily reproduced, though the designers of NTSC adopted a variant of YUV to accomplish this called “YIQ.” In YIQ, the Y component is still Luma, but the “IQ” represents a new coordinate into the same 2D color plane as YUV, just rotated slightly so that the purple/green spectrum falls on the axis with a smaller range. Nowadays with the higher bandwidth digital TV provides, we no longer need to encode using YIQ, but due again to the way our vision system responds to color and the technical benefits it provides, TV/video is still encoded using YUV, albeit will a fuller chroma representation.
What does all of this have to do with digitizing old 8mm tapes?
Each pixel on a modern computer screen is represented by at least three discrete values for Red, Green, and Blue. Though NTSC defines 525 lines per frame (~480 visible), being an analog standard means there really isn’t such a thing as “pixels” horizontally. However, most capture cards are configured to sample 720 points along each line of NTSC video, forming what we would call 720 “pixels” per line. But two important details must be noted:
Though 720 samples are enough to effectively capture the entire line, only 704 of them are typically visible and furthermore, NTSC TV is designed for a 4:3 aspect ratio. That is, if the picture is 480 square dots vertically, then it must be (480 * 4) / 3 == 640 square dots horizontally, or the picture will appear squished and everything will look “fat.” A captured NTSC frame at 720×480 will need horizontal scaling plus cropping to 640×480 to be displayed with the correct aspect ratio on a computer screen with square dots.
720 samples are enough to capture each line of the luma component. The chroma component is a whole other story.
Remember how the luma and chroma components are encoded separately, but that some of the chroma information can be discarded to save space and we’re not likely to notice? Turns out, computers can use that technique too to reduce bandwidth usage and save disk space. RGB is just how your computer talks to its display, but there’s no rule that says computer video files need to be encoded as RGB. We can encode them as YUV, too, and this is where the term chroma subsampling comes in.
While we always want to sample all 704 visible “pixels” of luma information, we can often get away with capturing 50% or even as little as 25% of the chroma information for a given line of picture. The ratio of sampled chroma data to luma data is called the “chroma subsampling” ratio and is indicated by the notation Y:a:b, where:
Y: the number of luma samples per line in the conceptual two-line block used by a and b to reference. This is almost always 4.
a: the number of chroma samples mapped over those Y dots in the first line. This is at most Y.
b: the number of times those a chroma samples change in the second line. This is at most a.
A chroma subsampling ratio of 4:4:4 means that every luma sample has its own distinct chroma sample; this ratio captures the most data and therefore provides the highest resolution. The value of a is directly proportional to the horizontal chroma resolution, and the value of b is directly proportional to the vertical chroma resolution. NTSC is somewhere between 4:1:1 and 4:2:1 (consumer tape formats even less), whereas the PAL standard in Europe is closer to 4:2:0 (half the vertical chrominance resolution as NTSC). As the values of a and b shrink, the overall chroma resolution decreases, and depending on your source picture it may be more or less noticeable.
As the chroma resolution on this photo of Annie is diminished, artifacts become apparent which should remind you of heavily-compressed JPEG images. This is no coincidence—the popular JPEG image codec and all variants of the popular MPEG video codec use YUV encoding and discard chroma data as a form of compression. This is one reason why encoding as YUV is popular with real-world images and video—an RGB representation can be reconstructed by decoding the often-smaller YUV information, much like a color CRT has to do to get a YUV-encoded picture on its screen through its RGB electron guns. A ratio of 4:2:0 is popular with the common H.26x video codecs (though H.26x can go up to a full 4:4:4), and several professional codecs indicate their maximum chroma subsampling ratios directly in the name of the codec. DNxHR 444 and ProRes 422 come to mind. What these numbers represent is how much chroma information can be preserved from the original, uncompressed image, but looking at it another way, they represent how much chroma information is discarded during compression.
Breaking it down further, we can see how various chroma subsampling ratios affect the chroma information saved (or not) in this close-up of Annie’s toys. Notice the boundary between the pink and green halves of her ball, and the rightmost edge of her red and white rolling cage.
With my analog NTSC 8mm tapes having chroma resolution of at most 50% of the luma resolution, I need to capture them with at least a 4:2:0 chroma subsampling ratio. That 50% could be in the horizontal direction, the vertical direction, or both, but fortunately bmdcapture grabs YUV video at a 4:2:2 chroma subsampling ratio from my Blackmagic Intensity Pro card by default, which is enough to sample all the chroma information in my NTSC source. This is fine for my 8mm tapes, but if I want to capture a richer video signal in the future (like, say, a digital source over HDMI), I would probably want to investigate capturing at a full 4:4:4 ratio.
I tell Neat Video to clean up my interlaced video with a temporal radius of 5 (the maximum) and a noise profile I generated in advance for the Sony CCD-TR917 I’m using. With these parameters, it takes about eight hours to clean up two hours of raw footage on my PC, so I usually start it in the morning and have it running in the background while I work during the day. Once it’s finished, I’m ready to feed its intermediate result to my second and final AviSynth script to get my final output:
AviSource("1988 Christmas NV.avi")
QTGMC(Preset="Placebo", MatchPreset="Placebo", MatchPreset2="Placebo", NoiseProcess=0, SourceMatch=3, Lossless=2)
# Trim the fat
# Crop to 704x480
Crop(4, 0, -12, -6)
For some reason, Neat Video produces two identical frames for every input frame I provide in the first script, so the first step in the second script tells AviSynth to disregard those duplicate frames and assume we’re dealing with a 59.94 Hz interlaced NTSC video. I then convert Neat Video’s intermediate RGB result back to its original YUV encoding, once again using the Rec.601 matrix corresponding to my video’s NTSC format.
YUY2 is a digital representation of YUV-encoded video, using 16 bits per pixel. When capturing with a chroma subsampling ratio of 4:2:2, each chroma sample is shared with two luma samples. YUY2 encodes this by reserving 8 bits for each of four luma samples (“Y”), then 16 bits for each of the two chroma samples (8 bits for each of the two components of “UV”). When you add it all up, 8 bits x 4 luma samples + 16 bits x 2 chroma samples == 64 bits. Divide that by the four luma samples and you get 16 bits per pixel.
Before calling ConvertToRGB in the first script, our original raw capture was encoded as uncompressed YUV 4:2:2. ConvertToRGB does a bit of math to map the YUV values into RGB space given the conversion matrix specified. If you are converting back to YUY2 from an RGB clip created by AviSynth, using ConvertBackToYUY2 can be more effective because it is allowed to make assumptions about the algorithm initially used to convert to RGB, given that both functions belong to AviSynth. In particular, ConvertBackToYUY2 does a point resampling of the chroma information instead of averaging chroma values as ConvertToYUY2 would, resulting in a chroma value closer to that in the original pre-RGB source clip.
After cleaning up the noise with Neat Video, to prepare these videos for playback on a progressive scan display such as a PC, mobile device, or modern TV, I still need to apply a deinterlacing filter. Each NTSC frame is split into two “fields”—the first field alternates every other line, and the second field has the other half. NTSC only has enough bandwidth to transmit 29.97 frames per second, so interlacing is a trick used to send half-sized “fields” at twice the rate. CRT TVs have a slower response time than modern digital TVs, so the net effect of interlacing on CRTs is that one field “blurs” into the next, producing the illusion of a higher rate of motion. Nowadays when TVs receive an analog NTSC signal they’re smart enough to recognize that it’s interlaced and apply a simple deinterlacing filter. But such filters are designed to work in real time, and without one, an interlaced source will appear to stutter when compared to its deinterlaced 60 Hz counterpart. A free third-party AviSynth script called QTGMC can do a much higher-quality job of deinterlacing ahead of time, with the caveat that I need to apply this script in advance and offline, that is, not in real time.
Here’s something to know about me. When it comes to filtering and rendering media offline, it doesn’t matter to me all that much how long it takes. The only deadline I have with this stuff is the eventual deterioration of the source media (and my own mortality, I suppose). For the moment, I have time and computing cycles to burn. As with the eight-hour Neat Video process, I have no qualms about backgrounding a rendering queue and letting it run overnight or even well into the next day. So when QTGMC and other tools advertise “Placebo” presets, I use them. I sleep better at night knowing there’s nothing more I could do to get a better result out of my tools, however miniscule. 🙂
I finally trim the start and end of my raw capture, leaving only the actual content behind. The last step crops some “garbage” lines from the bottom (VBI data I don’t need to save, but my Blackmagic card captures anyway), along with some black bars on the left and right sides of the image. Not surprisingly, the final dimensions of the video come out to 704×480: the visible part of a digitally-sampled NTSC frame. I render the whole thing out as a lossless Huffyuv-encoded AVI once again, saving the final lossy encoding step for FFmpeg.
At this point, I have a couple of directions I can take. I can prepare my final video for export to DVD, or I can split my video into one or more video files suitable for uploading to my Plex server. I can do both if I want. In either case, I need to prepare chapter markers so FFmpeg knows how to split the final, monolithic video into its constituent clips. I use Google Sheets for this; I have one sheet with columns for “Start time,” “Duration,” “End time,” and “Label.” I put the “Start time” column in the HH:mm:ss.SSS format that FFmpeg expects for its timecodes, and I express the “Duration” column in seconds with decimals. When I’ve marked all the chapters (relying on VirtualDub’s interface for precise frame numbers and timing), I pull the “Start time” and “Duration” columns into a new sheet and export to a CSV file.
I wrote a handful of Windows batch scripts that take this CSV file, a path to FFmpeg, and a path to my final video, and split my final render into a collection of H.264-encoded MP4 files, or a series of MPEG files encoded for NTSC DVD. In the DVD version, I also generate an XML file suitable for import into dvdauthor. I found that neither my iPhone nor my Android devices like H.264 files encoded with a chroma subsampling ratio higher than 4:2:0, so I wrote a script specifically to encode videos destined for mobile devices. These scripts are longer than make sense to post inline here, so here are links to each one if you’re curious and would like to use them:
Capturing and digitizing my family’s movie memories will help preserve these moments so they can be enjoyed in the years to come without wearing out the original tapes. The biggest cost to this digitization process is time, both in capturing from tape (which is always done in real time) and the offline clean up and encoding to digital files. I still have more than a couple dozen tapes to sort through, so this blog post doesn’t signify the end of this project, but now that I have a reliable process that results in what I consider a marginal improvement over the source material, I like to think I can get through the remaining tapes on a sort of “autopilot.”
I hope this post will inspire someone else to preserve their own memories.
Bonus postscript: field order
As I mentioned above, an interlaced NTSC frame is split into two “fields,” with each field alternating between even and odd lines of the full frame. But which field is sent first? The even field, or the odd field?
The answer, as one might expect, is “it depends.” For my purposes here, the fields in a 480i NTSC signal are always sent bottom field first. But a 1080i ATSC signal—as is used with modern digital TV broadcasts—sends interlaced frames top field first. If you crop the source video before deinterlacing, you have to be careful to crop in vertical increments of two lines, or you run the risk of throwing off the deinterlacer’s expected field order. For example, if line 12 is the first visible line of the first field of my picture, it’s reasonable to expect it to be the bottom field of the frame. After cropping 13 lines from the top of the frame, though, the deinterlacer will assume that line 13 is the bottom field, when it really is the top field. The deinterlacer will then assume that the previous frame is the current frame and will get confused, resulting in motion that appears jerky.
OK, I’ll admit that the Retroquest Super Retro Castle isn’t really a “Pippin Classic,” but it sure looks the part, and when I learned of its existence back in April, I figured it had to have at least as much power as a stock Pippin. Its advertised specs reminded me of the Raspberry Pi, leading me to speculate that maybe the Super Retro Castle is nothing more than a Raspberry Pi in a fancy Pippin-inspired case:
Two USB controller ports
DC power jack
Odds seemed good to me that it runs either a variant of Android or mainline Linux. Therefore, couldn’t I turn it into a sort of miniature Pippin console?
With a working knowledge of Linux and a little bit of hardware hacking, maybe I can. 🙂
Coaxing the Super Retro Castle into running user-provided code is certainly easier than cracking its inspiration. Booting it quickly reveals that it’s running a mostly-stock build of RetroArch on Linux, and booting it with the bundled microSD card inserted reveals that it’s configured to search the card for supplemental configurations. A-ha, a vector! Without making any modifications to the console itself, it is trivial to add new ROMs and even new libretro modules for emulators not built in to the Super Retro Castle. Just make sure the desired modules are built for the “armhf” architecture and the respective .lpl playlist file is configured with the correct path to the .so file under the microSD card’s /storagesd mountpoint, and you’re off to the races.
Bonus points: it runs RetroArch as root.
This is great if you don’t mind the Super Retro Castle’s built-in software and interface and are content to use it as an cheap emulator box. Don’t get me wrong, it’s great at what it does, but 1) I can’t read Japanese and 2) I’d like to be able to replace its built-in software with something newer, or something different entirely. To do that, I’d have to break out of its RetroArch jail and get access to its filesystem, ideally via a shell.
Rooting the Super Retro Castle, though similarly easy, requires a bit more elbow grease. Step one is getting the case apart, which is pretty straightforward: Just remove the four Phillips-head screws located underneath the rubber pads on the underside of the case:
Right away we notice that most of the console’s weight is concentrated in the metal heatsink affixed to the bottom of the case. But the other thing we see is that the Super Retro Castle is not a Raspberry Pi or derivative at all; it appears to be a custom logic board built around the Amlogic S905X SoC.
Wikipedia tells me quite a bit about the S905X: It sports a quad-core ARM-based CPU clocked at 1.2 GHz, a Mali-450 GPU, and support for decoding H.264-encoded video at up to 1080p60 in hardware. Quite the chip. When implemented on the Super Retro Castle board, the S905X is supported by two Samsung K4B461646E-BCMA chips for a total of 1 GiB of RAM, and one Samsung KLMBG2JETD-B041 eMMC chip for 32 GiB of onboard storage. The latter definitely explains the wealth of preloaded ROMs available for selection right after bootup. 😉
A handful of pads next to the S905X chip suggest a JTAG interface, but what piques my interest is this small set of vias next to the microSD card slot:
GND, TX, RX, and VCC? That sure looks like a UART serial interface to me, and if the Linux image used is only slightly tweaked from defaults, I should expect to see a console on this interface at 115200 baud. But first, let’s solder on a header:
Next, route a short cable out through the rear vent holes…
… reassemble it…
… and now it almost vaguely resembles a Pippin dev/test kit. 😛
Side note about UARTs
It is tempting to think of a UART interface such as the one found on the Super Retro Castle as the same interface used by oldschool PC serial ports (RS-232 / RS-422). But beware: they are not the same. Not only are the voltage ranges different—between -15 and +15 V for RS-232 and -6 to +6 V for RS-422—but UARTs are TTL devices (transistor-transistor logic) that expect voltages between 0V for logic low, and either 3.3V or 5V for logic high. Attempting to drive a UART by naively wiring it to a USB serial adapter (as I initially did) therefore runs the risk of frying the UART, which in the instance of the Super Retro Castle is built in to the S905X SoC. Fortunately the Super Retro Castle has some protective circuitry somewhere, so while my initial efforts produced garbage data regardless of baud rate, I was lucky in that I could try again with a proper USB UART adapter. I picked up a μART from Crowd Supply for this purpose and I really like it so far. 🙂
Getting root at this point just required wiring the GND, TX, RX, and VCC pins to my μART adapter, connecting the adapter to my PC, and opening up a PuTTY instance over the COM port it provides. At 115200 baud, I get a full boot log confirming four CPU cores each clocked at 1.2 GHz, U-Boot as the bootloader, and a build of Lakka running from the internal eMMC chip mounted read-only, followed by a root shell prompt. The initial syslog also seems to indicate the presence of a network interface—this follows from the description of the S905X chip on Wikipedia, but I don’t see any unused pads on the main board suggesting the relevant pins are routed from the SoC. Next steps for me will be backing up an image of the internal filesystem before attempting to remount it read-write so I can augment it with my own provided software.
I owe a lot to Mark of the Unicorn’s Professional Composer. Had my dad not encountered this program around 1985 and subsequently adopted it (and its corresponding Mac hardware) for himself two years later, I would not have grown up with Macs, possibly even computers. I certainly wouldn’t have become as familiar with music notation software, let alone music theory, as I am today. My dad tells the origin story thusly:
After Graduate School (1985 or so) I became familiar with the notation program called Professional Composer™. The program was housed in the ISU [Illinois State University] computer lab where it was run on several Macintosh 512 computers. I’ve always been something of a visual learner when it came to things like this and found I could navigate this program rather easily without having to read a manual. I began by doing easy arrangements of trumpet quartets. Early on, these programs ran on 3 1/2″ floppy disks which meant that your files couldn’t be very big before you’d need another disk. This gave way to hard drives but even then you were still limited as to how big your files were or how many files you had on the drive.
I remember asking upper-level administration in District #131 that if I bought the hardware (in this case a Mac Plus), would they buy me the software for this program? They agreed and the rest is pretty much history.
MOTU discontinued Professional Composer (hereafter referred to by the nickname “ProCo,” courtesy of my friend and fellow hacker Josh Juran) sometime after its final 2.3M revision was released in 1990. My dad used this program almost every day for 20 years(!), after which I had convinced him to crossgrade to MOTU’s Composer’s Mosaic. The latter offered better MIDI playback and print layout capabilities, plus could import his by this time extensive library of ProCo files. However, neither ProCo nor Mosaic files can be fully imported into any modern music notation program. It therefore fell upon me as the family’s computer expert—and continues to even now—to ensure that the hardware powering my dad’s favorite music software continues to run despite all other advancements. As a matter of course, I have an intimate knowledge of the capabilities, requirements, and quirks of these two programs. (I have a lot of sympathy for banks and government institutions tasked with similar mandates.)
By the time ProCo 2.3M came out, hard drives were common and the ProCo application itself had long since outgrown its original home on a 400K (later 800K) boot disk. So 2.3M shipped on an 800K disk offering the option to install or remove itself to or from an attached hard drive, respectively, if launched from the master disk. Installing to the hard drive decrements an “install count” on the master disk, allowing the user to use one master disk to install ProCo to one hard drive at a time. Merely copying the ProCo application to a hard drive isn’t enough; if the application isn’t properly installed by the master disk, launching the program from hard drive prompts the user to insert the master disk if not already present. I remember accidentally wiping out at least one of these hard drive installations from my dad’s Mac as a curious tinkerer in my youth (for which Dad was not pleased), leading Dad to request/beg MOTU for one final backup master disk some time in the mid-90s. It is a testament to the quality of 3.5″ floppies back then along with how well my dad takes care of them that the disks remain usable, some 30+ years later.
Eventually Dad got me my own Mac(s), where I could hack away safely—safe from his expensive software, at least. But as I grew as a hacker and programmer, acquiring and installing various software packages of my own, encountering—and defeating—assorted authentication schemes, going deep down the rabbit hole of the inner workings of Mac OS, and even dipping my toes into the field of software preservation, the music notation program that started it all continued to elude me. Why can’t Disk Copy or DiskDup produce a working substitute for the ProCo master disk? Why is it so difficult to duplicate the master disk using, for example, a KryoFlux? Why can I install ProCo to an emulated HD20 via my Floppy Emu, but not to a mounted Disk Copy disk image? Why does ProCo crash when After Dark kicks in? Why does its installer only appear when launched from the original master disk? And how does ProCo know that it has been properly installed?
I’m determined to finally find out.
ProCo has a minimal About dialog, displaying the name of the software, the version, its copyright years, and credit only to “Mark of the Unicorn, Inc.” I therefore have no real idea who wrote it, despite having asked MOTU via emailandTwitter for the source code multiple times over the years. Ultimately my goal here is to develop a conversion utility that brings ProCo files into the 21st century, so I’d even be satisfied with documentation of its file format, but I would be surprised if there is anybody left at MOTU who is even aware of Professional Composer, let alone familiar with a product they haven’t supported in over a quarter century.
So off to the disassembler we go. 😉
A Short Primer on Memory Management and Launching 68K Applications on the Mac
Much of the Mac’s software is split into chunks of data and code called “resources” that can be swapped in and out of RAM as needed. In order to maximize the use of the sparse amount of memory on the original Macintosh, its designers traded a tiny bit of speed for greater efficiency when building the Memory Manager. When asked to load a resource from disk, the Resource Manager returns a “handle” to the loaded resource, which is a pointer to an OS-controlled “master pointer” pointing to a relocatable block within the heap. The Memory Manager can then be allowed to move or “compact” relocatable blocks in the heap, or even remove/”purge” such blocks when available RAM is running low. This relieves applications from some of this management burden and is leveraged throughout the Mac System Software. In addition to allocating their own handles and non-relocatable blocks, applications may mark existing handles with various attributes to guide the Memory Manager in its housekeeping; for example marking a handle “purgeable” allows the Memory Manager to free its associated block, and likewise “locking” a handle prevents it from being moved or freed.
Each time you double-click on a 68K application to launch it from the Finder, a carefully orchestrated sequence of events takes place:
Finder calls the _Launch trap with the name of the application.
The Segment Loader opens the resource fork of the file passed to _Launch and immediately preloads ‘CODE’ resource ID 0. ‘CODE’ 0 is a specially formatted ‘CODE’ resource. It contains the parameters necessary to set up a non-relocatable block of memory near the top of the application’s memory space containing application and QuickDraw globals, any parameters passed to it from the Finder, and the application’s jump table. Following these parameters is the jump table itself: a list of tiny eight-byte routines that each load a ‘CODE’ resource, or “segment,” and jump to an offset within that segment.
Using the parameters at the start of ‘CODE’ 0, the Segment Loader allocates space for globals pivoting around register A5, which is eventually passed to the application. This is known in Mac programming parlance as the “A5 world” and is unique to each running application.
The jump table is copied above the Finder’s application parameters in the A5 world and the ‘CODE’ 0 resource is released.
The first entry in the jump table is executed, and the application takes control.
The relative jump instructions of the original 68000 processor are limited to signed 16-bit offsets, so branches or subroutine calls are limited to 32K offsets in either direction from the current program counter. In order to accommodate programs with more than 32K of code under the memory constraints of the original 128K Macintosh, the Segment Loader was invented which manages applications split into ~32K code “segments.” Code within each segment can make intra-segment jumps (branches or subroutine calls), but once a subroutine is needed outside a particular segment, a call must be made to the jump table which in turn loads the necessary segment. New segments are returned as handles to relocatable blocks just like any other resource, so as they are loaded the Memory Manager automatically compacts the heap and/or frees purgeable handles to make room in RAM. Recall that the jump table is copied to a known location relative to the A5 register, so applications always have easy access to it. But since new code segments are created at locations on the application heap unknown at compile time, this also means that all code segments are invoked assuming position-independent code, meaning all branches and subroutine calls are relative.
All 68000 processors support branches to absolute addresses utilitizing the full usable width of the address bus. The 68020 and later processors support larger relative branch offsets, so segments are not necessarily limited to around 32K. Well-behaved applications check at launch that the host Mac has their necessarily capabilities, and exit early if not. But for maximum compatibility, some applications built with compilers such as CodeWarrior are generated with a table of offsets to absolute branch instructions within each code segment. These instructions are compiled as jumps to offsets within the segment relative to zero—sure to crash the Mac if executed as stored. But in a small bit of “preflight” code, these absolute branches are fixed up to point within the segment, providing larger branch offsets to all Macs. This is how the ‘rvpr’ 0 resource was compiled for the Pippin.
The first thing we notice is that ProCo’s jump table contains one valid entry and then… a lot of nonsense. This is a likely sign of an encrypted jump table—Epyx’s Temple of Apshai along with Winter Games also uses this obfuscation trick to scare off casual hackers. In fact, almost all of ProCo’s ‘CODE’ resources look to be encrypted! If I hope to make any sense out of ProCo’s file format by looking at its code, we’ll need to derive the algorithm that decrypts the rest of this.
MOVE.W #$0029,-(A7) pushes the ID of ‘CODE’ resource 41 onto the stack prior to jumping into it via _LoadSeg. Once there, we start by allocating a couple of memory blocks to use, starting with a 384-byte block of memory that we’ll call the “environment” block. We stash the stack pointer into offset 28 of that block, push a pointer to our environment block onto the stack, then stash the value of the ScrDmpEnb global into offset 32 of our environment block.
ScrDmpEnb is short for “screen dump enable” which originally meant whether the screen shot feature is enabled via Command-Shift-3 on Macs, but grew to include other FKEYs as well. One popular third-party FKEY available to hackers was the “Programmer’s Key,” which drops into the installed debugger when invoking e.g. Command-Shift-7, providing a way to drop to MacsBug without a physical programmer’s switch installed on the side of the machine. But the same functionality could be had by simply writing your own equivalent FKEY, following instructions provided in the official MacsBug manual. MOTU certainly couldn’t have made it easy for most would-be crackers to just conveniently drop into the debugger during the startup process of their precious software, so ScrDmpEnb is set to zero, effectively disabling all FKEYs.
What the FKEY?!
The Apple ][, Lisa keyboard, original Macintosh keyboard, and most keyboards that later shipped with 20th-century Macs, did not feature what we know as “function keys”: the F1-F12 (and beyond) keys that adorn the top of most keyboards today. These devices more closely mimicked typewriter keyboard layouts that many users were familiar with at the time of their respective introductions. But on the Apple ][, there are a few reserved keyboard shortcuts that are always available to the user: Control-Reset breaks out of the currently running program, and Control-Open Apple-Reset resets the computer, for example.
These shortcuts are hardcoded into the ROM and not easily modifiable by the user. On the original Mac, since localization (in particular, keyboard layouts) fell out of the disk-based System Software’s foundation built on resources, it’s only natural then that shortcuts be handled by the OS in a modular way as well. So Apple made up for the lack of physical function keys by providing several “virtual” function keys bound to Command-Shift-numbers. When invoked, these shortcuts run tiny programs stored as ‘FKEY’ resources in the System file, which is why they are known as “FKEYs.” Programmers quickly discovered that they could write their own tiny FKEY programs and install them into the System file assigned to otherwise unused numbers.
The original set of FKEYs as shipped in 1984 are as follows:
Command-Shift-1: eject the first/internal floppy disk, if present
Command-Shift-2: eject the second/external floppy disk, if present
Command-Shift-3: take a screenshot and save it to disk
Command-Shift-4: take a screenshot and print it
The first two ejecting FKEYs went away with the introduction of Mac OS X in 2001, as Apple had stopped shipping Macs with floppy drives by then (though macOS continues to support external drives natively). But Command-Shift-3 lives on as the assigned shortcut for saving screenshots to disk—one of the few remaining holdovers from the original 1984 System Software.
We then allocate a handle to a new locked 82-byte “context,” pop the pointer to our environment block into offset 48 of our context, then push our newly-allocated context’s handle onto the stack. Next we pass the pointer to the top of the ‘CODE’ 41 resource we’re executing from to _RecoverHandle to get our ‘CODE’ resource’s handle. We store this handle at offset 0 of our context, then store its master pointer at offset 4. We finally recover our context’s handle from the stack before passing it to the first real “stage.”
Stage 1: Front Line Disassembly
Stage 1 is fairly simple and calls three subroutines before launching into Stage 2, looking roughly like this when decompiled back to pseudo-C code:
So let’s break it down, function by function. We start with initContext, which looks like this:
void initContext(Ptr contextPtr)
for (short i = 0; i < 8; ++i)
contextPtr->stagePtrs[i] = stageInfoCmd(
contextPtr->stageGlobalsPtr = contextPtr->stagePtrs;
contextPtr->code41Size = _GetHandleSize(contextPtr->code41Handle);
contextPtr->code41End = contextPtr->code41Ptr
+ contextPtr->code41Size - 1;
initContext initializes a block of eight pointers to the results of stageInfoCmd, which looks like this:
SET_OFFSET = 0,
GET_OFFSET = 1,
GET_OFFSETS_ARRAY = 2,
static short offsets =
0x0086, // offset to Stage 1 from top of 'CODE' 41
0x0388, // offset to Stage 2 from top of 'CODE' 41
0x0D24, // ?
Ptr outPtr = nullptr;
offsets[index] = (short)(stagePtr - codePtr);
outPtr = codePtr + offsets[index];
outPtr = offsets;
So initContext initializes a block of eight pointers in our context to point to eight different areas of 'CODE' resource ID 41. The first of these pointers points to the Stage 1 code we're currently executing, and the second of these pointers points to the encrypted block of 'CODE' 41 immediately following the bits of Stage 1 that are recognizable as executable code. This eventually becomes the Stage 2 code that we jump to later. (It's interesting that stageInfoCmd is only ever called with the GET_OFFSET selector, making the rest of that function dead code.)
Next we initialize a field in our context that's used to store an aggregate of all computed checksums. We then call updateStage2Checksums which looks like this:
updateStage2Checksums in turn makes a couple of calls to calculateChecksum, passing each call the blocks leading up to, and following Stage 2, respectively.
long calculateChecksum(Ptr startPtr, Ptr endPtr, long seed)
long size = endPtr - startPtr;
long cksum = seed;
long sizeInLongs = size / sizeof(long);
Ptr ptr = startPtr;
if (sizeInLongs != 0)
size -= sizeInLongs * sizeof(long);
for (long i = 0; i < sizeInLongs; ++i)
cksum += *((long*)ptr)++;
if (size > 0)
for (long i = 0; i < size; ++i)
cksum += *(byte*)ptr)++;
One thing that sticks out immediately to me is the use of the longword 'PACE' as an initial checksum seed. 'PACE' is likely a reference to PACE Anti-Piracy: a company founded in 1985 that's still around today. MOTU adopted PACE's protection code for ProCo starting with version 2.1, released in late 1987. Indeed, pirated versions of ProCo existed as early as 1985; some are still available on the Internet. With 1.0 and 2.0 unencrypted, "sharing" these was much easier than with later versions. My dad acquired his Mac Plus / ProCo combo in the summer of 1987—possibly June of that year—so with 2.1 having a creation date of September 11, 1987 and assuming MOTU shipped the latest version with new orders, it follows that 2.0 is the earliest version my dad owns. MOTU periodically shipped new master disks containing updated versions of ProCo to registered owners as they became available, free of charge—a practice I commend them for. 🙂
Now that we have our checksums, we're ready to head into decryptStage2. The decryption code takes the checksum of the code to be decrypted as the "key." This makes binary patching the ProCo application an involved process, since the remainder of the code would need to be reencrypted for decryption with its new checksum to succeed. One thing is for sure about this protection: it is designed to be resilient against quick-and-dirty patches.
void decryptStage2(Ptr contextPtr)
void decryptStage2Block(long key, Ptr startPtr, Ptr endPtr)
long size = endPtr - startPtr;
long sizeInLongs = size / sizeof(long);
Ptr ptr = startPtr;
if (sizeInLongs-- != 0)
long longsLeft = sizeInLongs;
long rotCount = key & 0x0F;
if (rotCount == 0)
rotCount = 1;
key = rotateRight(key, rotCount); // Ror.L in 68K
*((long*)ptr)++ ^= key;
size -= sizeInLongs * sizeof(long);
if (--size >= 0)
long bytesLeft = size;
long rotCount = key & 0x0F;
if (rotCount == 0)
rotCount = 1;
key = rotateRight(key, rotCount); // Ror.L in 68K
*((byte*)ptr)++ ^= (byte)key;
With Stage 2 now fully decrypted, we can jump right in by loading contextPtr->stagePtrs into A0 and JMPing right to it.
That wasn't so hard, was it? 😛
There is still at least another stage of this protection to get through, and we're hardly that much closer to decrypting the remaining 'CODE' resources or the jump table. Knowing PACE's reputation, this is likely a small victory in what will ultimately be a long battle. ProCo is legendary in some Mac circles for its copy protection, so if what I've heard of it is true, then I'm surely in for a ride. PACE even makes a bold claim on their own website:
We know it sounds like an unrealistic boast to say our anti piracy software cannot be cracked. Our goal is to stay ahead of the curve and hacking trends. We avoid giving known hooks or patterns that they recognize, and we pepper our anti piracy solutions with methods that we know are time consuming and difficult, if not impossible, to remove.
I’ve been busy. The Pippinizer is going to take me longer than I expected to put together into a releasable form, so I wrote a small utility that should tide folks over until that’s ready.
Introducing Pippin Kickstart. This is a small, carefully-crafted boot disc for the Pippin that circumvents the console’s built-in security and instead offers the choice to boot from an unsigned volume. It works on 1.0, 1.2, and 1.3 Pippins (so, every known retail Pippin ROM out there as of the time of this writing) without any modification.
To use it, simply download the Pippin Kickstart disc image available here, burn it to CD, and use that disc to boot the Pippin. Pippin Kickstart will identify what ROM and RAM it detects, eject itself, and then immediately begin searching for a bootable volume candidate. The Pippin will boot from CD-ROM using only its internal drive, but other types of removable media may work as well assuming that they can boot a regular Mac without special drivers. It also has been tested working using an external hard drive.
“But Keith, I thought 1.3 Pippins don’t do the authentication check at startup. Why would I use Pippin Kickstart with a 1.3 Pippin?” While it is true that ROM 1.3 does away with the signing check, it is still hardcoded to boot only using the Pippin’s internal CD-ROM drive. Pippin Kickstart offers owners of 1.3 Pippins the ability to boot from other media sources such as a hard drive, providing itself as a sort of “launch pad.”
The Pippin Kickstart disc is a hybrid HFS/ISO image containing the source code, a short README, and– just for fun– a few extra “goodies” that I found useful during its development:
UPDATE (20190702): Pippin Kickstart hasn’t even been out for 48 hours and I’ve already got an update prepared. Version 1.0.1 is available here (I’ve updated the rest of this post as well) and improves the accuracy of the RAM detection by calling Gestalt instead of reading MemTop directly.
Exploring the Pippin ROM(s), part 2, in which I discover that the Pippin’s boot process loads an ‘rvpr’ resource of ID 0 during the Start Manager’s phase of locating a bootable volume from the Pippin’s internal CD-ROM drive
Exploring the Pippin ROM(s), part 3, in which I skim ‘rvpr’ 0, question its multitude of seemingly similar subroutines, and show how it patches itself in place before jumping to main
Apple’s public key for verifying the authentication data on a Pippin boot volume is: E0 E0 27 5C AB 60 C8 86 A3 FA C2 98 21 79 54 A8 9F D1 B9 DC 8A BA 84 EF B1 E7 C9 E2 1B F7 DD D7 DC F0 E4 4A BB 79 51 0E 7C EB 80 B1 1D
How did I find this? Strap in and let’s go for a ride. 🙂
Quick recap: I want to unlock homebrew on the Pippin. Every time a Pippin tries to boot from an HFS volume on CD-ROM, it loads the ‘rvpr’ resource with ID 0 from ROM and executes it as code, passing as arguments that volume’s ID and two blocks of data found elsewhere in ROM. ‘rvpr’ 0 locates and reads an RSA-signed “PippinAuthenticationFile” from the volume. It contains a 128-bit digest for each 128K block in the volume, along with a 45-byte signature at the end of the file. If the signature cannot be verified or any of the digests don’t match what ‘rvpr’ 0 calculates from the volume in its main loop, the Pippin (r)ejects the disc.
Last time I looked at ‘rvpr’ 0, I examined and broke down what the main loop does at a high level. Outside of main, there are ten non-library function symbols in the resource, six of which I’ve identified their usage. Reading through these, I determined the format of the PippinAuthenticationFile and how its data is fed to the rest of the main loop. The remaining four functions—VerifyDigestInfo, VerifySignature, CreateDigest, and InitRSAAlgorithmChooser—form what I conjectured to be the “meat” of the authentication process. I elected to pore over these at a later time.
‘rvpr’ 0 is over 35K in size, with almost 34K of that comprised of 68K assembly code. Unfortunately, what I looked at in part 6 only touches about 3K of that—not even 10% of the whole. What’s worse is that the remaining 31K/90+% of code is almost completely lacking symbols, save for an occasional call to T_malloc, T_memset, T_memcpy, or T_free. Without human-readable symbols to guide what the remaining memory locations, values, and functions mean in the context of ‘rvpr’ 0’s greater purpose, I would be “flying blind” without a safety net. If I was to attempt to grok this code to the same degree as I currently understand main and its (named, mind you) auxiliary functions, I would have a long road ahead of me especially if I used the same static analysis technique of stepping through the code offline on paper.
I decided that the best way to figure out the rest of this code was to use dynamic analysis; that is, to examine it while it’s running. There were just too many subroutines and too much data being pushed around to keep it all straight in my head. I needed a computer to help. I don’t have any hardware debuggers for the Pippin that would allow me to step the CPU and examine the contents of RAM, and no working software emulators exist for the Pippin (yet). What I found does exist is a suite of 68K assembly tools—a code editor, binary editor, and, crucially: a simulator—called EASy68K. If I wanted to look at ‘rvpr’ 0 in something that even somehow resembled a debugger, I’d have to build a working version of ‘rvpr’ 0 that could be run outside of a Pippin, without first understanding how the code works in the first place.
EASy68K’s simulator provides a hypothetical computer system featuring a 68K CPU, 16MB of RAM, and rudimentary I/O facilities. Luckily, ‘rvpr’ 0 is pretty self-contained, which allowed me to quickly “port” it to EASy68K. I was correct in that this technique significantly accelerated my understanding of the digesting and verification process, but as I hope to elucidate later in this post, that pursuit required very little actual parsing of code. 🙂
The first step to building an ‘rvpr’ 0 replica in EASy68K was to adopt the syntax EASy68K likes. I prefer FDisasm for disassembly because it’s part of the Mini vMac project and as such, it’s at least a tiny bit aware of the classic Mac OS API, known as the Toolbox. FDisasm can replace raw A-traps (two-byte 68K instructions beginning with the hex digit $A, which typically map to commonly-used subroutines) with their corresponding human-readable names according to official Mac documentation, which is a nice time-saver especially in large blocks of 68K Mac code. I also like FDisasm’s output formatting, which is the basis of how I list 68K assembly in these blog posts.
Cloning ‘rvpr’ 0 in EASy68K serves two purposes. First, I can step through it using a real signed Pippin CD, observe what its code does, and document it. After I reverse-engineer the authentication process though, this functional copy will serve a second purpose: to verify that my own authentication files are crafted properly. Since we know from part 2 and part 6 that the main loop will return zero in register D0 if the verification process succeeds, we should be able to observe that in the simulator. Using our own ‘rvpr’ 0 binary that’s as close as possible to what’s in ROM on an actual Pippin should assuage doubt as to whether a proof-of-concept will pass the console’s tests or not. Plus, since it’s all simulated in software, it saves me from having to burn a ton of test CDs. 😛
Converting my (annotated) disassembly from FDisasm’s syntax to EASy68K’s was easy—regular expressions to the rescue. Assembling the result produces code identical to what’s in the original resource—yay, the assembler works, and we have a byte-for-byte clone of what’s in the Pippin ROM. But making this new replica functional required a little bit of creativity.
On a real Pippin, ‘rvpr’ 0 is loaded from the ROM’s resource map into an area on the system heap in RAM. The relocation code at the beginning of ‘rvpr’ 0 patches each subroutine jump by offsetting them relative to where the code resides in RAM (discussed in part 6). It keeps track of whether this is done by storing this offset in a global when relocation is complete. Recall from part 3 that this global has an initial value of zero when ‘rvpr’ 0 is first run. If this code is executed a second time, it subtracts this global from its base address in RAM and, if the result is zero, it doesn’t need to do relocation again since the jumps already have valid destination addresses.
The simulator comes with no bootloader at all but starts up fully initialized, so in essence the contents of ‘rvpr’ 0 form the “ROM” of our virtual computer. We thus boot directly to ‘rvpr’ 0’s entry point, at the start of the simulator’s memory space. But since ‘rvpr’ 0 now always starts at address 0, the difference between that initial global and our base address… is zero. So the relocation code never runs in the simulator; it doesn’t have to because those unpatched jumps are already relative to zero. 😉
By the time ‘rvpr’ 0 executes in a real Pippin’s boot process, many subsystems on the console have been readied: the QuickDraw API for graphics, the Device Manager for device I/O, and the HFS filesystem package to name a few. These APIs, having been designed and built by Apple for the Mac, only exist on a Mac-based system and therefore naturally aren’t present in the fantasy system we get from EASy68K. We are in luck though in that ‘rvpr’ 0 only makes calls to a grand total of nine Toolbox APIs. Four of these calls are used in the “prologue” code discussed earlier that relocate all the jumps before main is even called. Since that relocation code doesn’t run in our simulator, that leaves five Toolbox APIs essential to the main loop: _Read, _Random, _NewPtr, _DisposePtr, and _BlockMoveData. We need equivalents to these routines if we are to expect ‘rvpr’ 0 to work properly.
_BlockMoveData is an easy one. It copies D0 bytes from (A0) to (A1):
I took a shortcut with _Random: my implementation simply returns a constant value. I did this partially because I’m lazy but also because _Random is only called once, albeit in a loop: to determine which 128K chunks to digest. By controlling the values returned, I can selectively and deterministically test chunk hashing.
I took similar liberties with _NewPtr and _DisposePtr: I keep a global pointing to the next unused block, and _NewPtr simply returns the value of that global and then advances it by the requested size. _DisposePtr is implemented as a no-op. Why did I do this? Well, again, part of it is because I’m lazy and didn’t want to write a proper heap allocator for this, but also because it affords me the ability to inspect memory blocks used even after they’ve been “freed.” I don’t care about memory leaks in this case—in fact, here they’re a feature! 🙂 Since ‘rvpr’ 0 is roughly 36K, I set aside the first 64K of memory for it (and any additional supporting code/data I add, like these replacement Toolbox routines). With register A7 initially pointing to the top of memory for use by the stack, the rest of RAM starting at $10000 I designate for my “heap.”
Finally we come to _Read. EASy68K may be pretty bare-bones, but it does come with some niceties allowing for basic interactions with its host PC. In this case, I needed a way for my “virtual Pippin” to have random-access readability from a virtual CD-ROM drive. Fortunately, EASy68K provides this in the form of the Trap #15 instruction. My version of _Read only does the bare minimum of what ‘rvpr’ 0’s main loop requires: it opens an HFS disk image on the host PC, seeks to the offset specified in the ParamBlockRec passed on the stack, reads the requested amount of bytes into the specified buffer, then closes the file.
Now that we’ve got functional replacements for the necessary Toolbox routines, how do we refit the rest of the code so that our versions are called instead of Apple’s, which don’t exist? I already had the Toolbox API names substituted in my listing, thanks to FDisasm, so I could simply create macros with those names that execute a tiny bit of code in place of those calls. The easiest way, and the method I tried first, is to invoke each replacement with a Jsr instruction, which is short for “Jump to SubRoutine.” This was really straightforward to do and assembled without issue, but upon loading and running in the simulator, I quickly discovered why this approach wouldn’t work. Jsr is a four, sometimes six byte instruction, whereas the original A-traps they were to replace use only two bytes. Since these larger instructions are inserted in the main loop near the beginning of the code, this throws off hardcoded addresses used later. Needless to say, when I ran in the simulator, a hardcoded Jsr landed instead in an unexpected area of code and I crashed almost instantly.
However I was going to invoke my faux-Toolbox calls, they had to be done in only two bytes. I thought for a second about how I could write an A-trap exception handler and leave Apple’s original A-trap instructions as-is, but I didn’t do that either because 1) laziness and 2) I thought of an easier way. Remember the Trap #15 instruction I used to implement _Read?
On a 68K processor, the two-byte Trap instruction provides a way to jump to any of 16 different addresses stored in a predefined area of memory, known as “vectors.” These 32-bit vectors are all stored consecutively in a block of memory that always starts at address $80. ‘rvpr’ 0 normally executes code at address $80, but that’s part of the address relocation done only on a real Pippin, not in our simulator. It is therefore safe for us to replace that block of code with the addresses of our replacement Toolbox routines, starting with Trap #0 and ending with Trap #3. Recall that I’ve implemented _DisposePtr as a no-op—which is the two-byte opcode $4E71—so I don’t need to set aside a trap vector for it. EASy68K only sets aside trap 15 for itself, leaving traps 0-14 for us to use however we wish. The code we do care about executing in the simulator doesn’t start until after address $100, so our entire trap table easily fits inside this block of unused code. How lucky can you get? 🙂
My very own Pippin “emulator”
With my cobbled-together Pippin “emulator” now up and running, finally I could take a look at the before and after of the remaining functions called by the main loop. I decided to start with CreateDigest, as I had already figured out the inner workings of CompareDigests, so this seemed like a simple starting point. CreateDigest starts by creating a context object for itself, and initializes one of its buffers with a curious but predictable pattern of 16 bytes: 67 45 23 01 EF CD AB 89 98 BA DC FE 10 32 54 76. Along the way it checks to see if any of this setup fails, and if so, the entire authentication check is written off as a failure. But assuming everything is fine, it enters a loop which digests our 128K input chunk up to 16K at a time, by passing each 16K “window” to an unnamed subroutine. Each time we iterate through this loop, the aforementioned buffer which was initially filled with a predictable pattern now instead contains a brand new seemingly random jumble of numbers. I suspected that this 16-byte buffer was used as a sort of working space for whatever Apple chose as a hashing algorithm, since its size matched that of the digests in the PippinAuthenticationFile.
764 B883 Cmp.L D3, D4 ; chunk size > 16K?
766 6C02 BGE.B dontResetD3 ; if so, use 16K for window size
768 2604 Move.L D4, D3 ; otherwise we have <16K left, so use what's left as window size instead
76A 4878 0000 Pea.L ($0) ; push zero
76E 2F03 Move.L D3, -(A7) ; push window size (typically 16K until the end)
770 2F0A Move.L A2, -(A7) ; push working chunk buffer ptr
772 2F2E FFFC Move.L -$4(A6), -(A7) ; push ptr to our hash buffer
776 4EB9 0000 6516 Jsr Anon217 ; patched, creates hash? digest?
77C 2A00 Move.L D0, D5
77E 4A85 Tst.L D5
780 4FEF 0010 Lea.L $10(A7), A7 ; cleanup stack
784 6628 BNE.B createDigestCleanup ; if Anon217 failed, bail
786 9883 Sub.L D3, D4 ; subtract window size from chunk size
788 D5C3 AddA.L D3, A2 ; advance working chunk buffer to next 16K-ish window
78A 4A84 Tst.L D4 ; still more data to hash/digest?
78C 6ED6 BGT.B createDigestLoop ; hash/digest it
78E 4878 0000 Pea.L ($0) ; push zero
792 4878 0010 Pea.L ($10) ; push 16
796 486E FFF8 Pea.L -$8(A6) ; push address of local longword
79A 2F2E 0010 Move.L $10(A6), -(A7) ; push out buffer ptr
79E 2F2E FFFC Move.L -$4(A6), -(A7) ; push ptr to our hash buffer
7A2 4EB9 0000 654C Jsr Anon218 ; patched, copies hash out?
7A8 2A00 Move.L D0, D5
7AA 4FEF 0014 Lea.L $14(A7), A7 ; cleanup stack
Finally, CreateDigest makes one more call to an unnamed subroutine, passing it among other things the number 16 (presumably the digest size in bytes), a pointer to its context object, and a pointer to a 16-byte area on the stack filled with $FF bytes. After this call, the working buffer is once again reset to its initial pattern, and the area on the stack is filled with what looks like it could be a digest.
Wait a minute, upon closer examination...
... the output hash matches what's in the PippinAuthenticationFile! This makes sense, because all CreateDigest does after this is tear down and dispose of its context object. It then returns to the main loop, where the computed digest is passed along for CompareDigests to, well, compare. So clearly those two unnamed subroutines play a vital role in computing the digest, however that's done.
I dove right in to the routine called in the loop. It starts by doing several integrity checks of the data structures it's about to use, then goes right into a confusing routine that appears to add the amount of bits in our up-to-16K input window to a counter of some sort, optionally increasing the next byte if the counter rolls over. I suspected this was used for a 40-bit counter of bits hashed, its purpose not obvious yet. It then enters a loop of its own, dividing our input window into yet another sliding window, this time of 64 bytes in size. Each iteration of this loop, it passes these 64 bytes and CreateDigest's 16-byte working buffer to an unnamed subroutine with some very interesting behavior.
Looking at this new subroutine, I was fairly convinced that it does the actual hashing. It is an unrolled loop that does a number of bitwise operations before adding a longword (32-bit quantity) from our input window along with what appear to be magic numbers to one of four 32-bit registers. At the end of this function, the contents of these registers are concatenated and added to the existing contents of CreateDigest's 16-byte working buffer. In order to maybe recognize a pattern to what this hash function was doing and perhaps identify the algorithm, I converted this assembly back to C code, and then verified that my C version produced identical output. Unfortunately, the algorithm didn't look familiar to me at all—I assumed it was something Apple invented specifically for the Pippin. I feared the initial "salt" might not be constant and could change depending on where the input chunk exists in the volume. Perhaps I merely found one hash function, but the Pippin could switch between different hash functions depending on some heuristic? It would require more disassembly and careful analysis to verify whether or not this was the case and why. 🙁
A Fun Side Story
Last Friday was 4/26, known informally among fans as Alien Day. I’m a big fan of the Alien universe, and this year happens to be the 40th anniversary of the 1979 Ridley Scott classic. So on Friday I had a number of friends over to watch both Alien films. 😉 There was pizza, chips, my homemade queso (half a box of Velveeta + a can of Rotel chiles—nuke it in the microwave for five minutes, stirring occasionally), and everybody had a good time.
One of the folks who dropped by was my friend Allison, who wanted to leave me with a disc of early Xbox demos her wife Erica found for me. I’m interested in investigating the contents of this disc in case there’s anything of historical value on it, but Xbox discs cannot be mounted or copied using a run-of-the-mill DVD-ROM drive. I remember years ago burning one or two (homebrew, ahem) Xbox DVDs with my PC, so I know writing Xbox discs is possible, but I was curious why reading them posed such an obstacle.
After some Googling, I found that the Xbox employs its own scheme to verify and “unlock” a boot disc candidate (described by none other than Multimedia Mike—an intrepid hacker whose blog I recommend). As I read, I learned that the Xbox’s disc verification involves the host (in most cases, an Xbox) answering an encrypted series of challenges at the drive level. This process, which is unique to each Xbox disc, uses SHA-1 hashes and RC4 encryption. This is a pretty cool and fascinating way to hide Xbox game data from non-Xboxes—it’s definitely worth checking out the details.
As one does on a Friday evening, I found myself clicking through to the Wikipedia entry on SHA-1. Not much time later, I was deep in the Wikipedia rabbit hole, ultimately landing on the page describing the MD5 message digest algorithm. Those of you reading this who have at least a passing familiarity with cryptography might recognize where this is going based on my description of CreateDigest's behavior above. I did not. 😛
According to Wikipedia, MD5 was designed in 1991 by Ronald Rivest, one of the inventors of the RSA cryptosigning algorithm used by the Pippin. MD5 was designed to replace an earlier version, MD4, which traded better security for increased performance. At a basic level, MD5 takes a bitstring of arbitrary length—the "message"—and generates a 128-bit string that uniquely identifies this input, called a "digest." The input string is padded to a multiple of 512 bits by adding a 1 bit, a number of zero bits, and finally the size of the original message in bits, stored as a little-endian 64-bit value. The padded message is finally split into chunks of 16 longwords, and these 64-byte chunks are then passed into MD5's core hash function to be added to a final 16-byte digest. If that sounds confusing, here's a summary: MD5 takes an arbitrary-sized message and turns it into a unique fixed-size message. The same input will result in the same output, but no two distinct inputs will result in the same output (this isn't 100% true, but for the purpose of this discussion we'll pretend it is).
On the surface, MD5 sounded like it might be what Apple adopted as the Pippin's digest algorithm, but I knew I'd hit paydirt when I instantly recognized the "magic numbers" used in its reference implementation. What's more, the Transform function looked almost exactly the same as the C code I derived from that unnamed subroutine in 'rvpr' 0, and the MD5Update function likewise performs the same steps as the 68K routine that calls into Transform. I am confident that Apple licensed this particular implementation for use in the Pippin. It follows the MD5 specification to the letter, even going so far as endian-swapping the input longwords from the Pippin's native big-endianness.
Armed with the knowledge that MD5 is the message digest algorithm used in the Pippin authentication process, it is clear that the digests computed in CreateDigest, and the digests read from the PippinAuthenticationFile used in CompareDigests, are themselves not signed with RSA. In fact, RSA is not involved with verifying chunks of the disc at all. This tells me that the only thing RSA is used for in the authentication process is for verifying the signature at the end of the PippinAuthenticationFile.
The signature, to recap, is 45 bytes long and lives at the end of the PippinAuthenticationFile. Before entering the main loop, 'rvpr' 0 makes a call to VerifyDigestInfo, which in turn makes a call to VerifySignature. VerifySignature calls upon MD5 to digest the "message" portion of the PippinAuthenticationFile—everything but the signature. It then must use RSA to decrypt and verify the signature against that MD5 digest. If it does, we know the chunk hashes therein can be trusted, so RSA is no longer needed. Otherwise, we know the PippinAuthenticationFile has been tampered with in some way.
Let's say for illustration's sake that the PippinAuthenticationFile is 64K and the last 1K is the signature. When the signature is decrypted, it should contain a digest of that first 63K. If we digest that first 63K ourselves and the two match, we're verified. The whole process is... pretty modern, actually, when you consider this was 1995. 🙂
Using the "Macintosh on Pippin" CD (a.k.a. "Tuscon") as a test case, I stepped through VerifySignature to obtain the MD5 digest of the authentication file's "message:" AE 1A EC AE A4 C5 11 68 2E 38 7D D1 48 F0 55 C2. With this in mind, I set out to test my hypothesis and hopefully find our computed MD5 digest of the message portion somewhere in memory. If I could find this, I could work backwards and reveal how the signature is decrypted. Apple indicated in a Pippin technote that RSA was licensed for their authentication software library. Whether this means Apple used the library as-is, or licensed the code to augment for their own needs, I wanted to verify this one way or the other. VerifySignature makes two unnamed calls before cleaning up and returning a result:
668 B883 Cmp.L D3, D4 ; is the remaining bytes to hash greater than 16K
66A 6C02 BGE.B (* + $4) ; then hash another 16K
66C 2604 Move.L D4, D3 ; else hash the remaining bytes (which will be < 16K)
66E 4878 0000 Pea.L ($0)
672 2F03 Move.L D3, -(A7) ; push size as bytes to hash (typically 16K until the last chunk)
674 2F0A Move.L A2, -(A7) ; push ptr to start of chunk to hash
676 2F2E FFFC Move.L -$4(A6), -(A7) ; -$4(A6) -> hash object
67A 4EB9 0000 816C Jsr ($816C).L ; create digest of 16K chunk in hash object
680 2A00 Move.L D0, D5 ; (don't know what is returned in D0, I think a size?)
682 9883 Sub.L D3, D4 ; remaining bytes -= how many bytes we just hashed (typically 16K)
684 D5C3 AddA.L D3, A2 ; A2 -> next chunk to hash
686 4FEF 0010 Lea.L $10(A7), A7 ; cleanup stack
68A 4A84 Tst.L D4 ; are there any remaining bytes left?
68C 6EDA BGT.B (* + -$24) ; keep hashing
68E 4878 0000 Pea.L ($0)
692 4878 0000 Pea.L ($0)
696 2F2E 001C Move.L $1C(A6), -(A7) ; $1C(A6) == $2D (size of signature?)
69A 2F2E 0018 Move.L $18(A6), -(A7) ; $18(A6) -> second longword in data block after hashes in auth file (start of signature?)
69E 2F2E FFFC Move.L -$4(A6), -(A7) ; -$4(A6) -> hash object
6A2 4EB9 0000 81A2 Jsr ($81A2).L ; probably decrypt the signature?
6A8 2A00 Move.L D0, D5
6AA 4FEF 0014 Lea.L $14(A7), A7 ; cleanup stack
I knew that the first call at $816C computes the MD5 digest of the PippinAuthenticationFile. I intuited that in order to determine whether verification succeeds, whatever occurs in the second call at $81A2 must accomplish that. Therefore, the public key must exist in memory at some point during $81A2's execution. In addition, the signature bytes must exist in memory at the same time. If I drill down into $81A2 until I find the signature bytes in RAM, I should find clues as to what data is used to decrypt it based on the proximity of what's changed to what hasn't, thanks to how I implemented my "heap."
$81A2 eventually makes its way to a subroutine at $1B0E, wherein I found the following:
1B58 2F0B Move.L A3, -(A7) ; push nullptr?
1B5A 4878 0000 Pea.L ($0) ; push 0
1B5E 2F04 Move.L D4, -(A7) ; push size of signature?
1B60 2F05 Move.L D5, -(A7) ; push address of signature?
1B62 42A7 Clr.L -(A7) ; push 0
1B64 486E FF78 Pea.L -$88(A6) ; push ptr to area on stack
1B68 4878 0000 Pea.L ($0) ; push 0
1B6C 486A 0028 Pea.L $28(A2) ; push ptr to $10F0 in hash object?
1B70 4EB9 0000 12E0 Jsr ($12E0).L ; jump somewhere that copies the signature to a working buffer
1B76 2600 Move.L D0, D3 ; D0 == result code
1B78 4FEF 0020 Lea.L $20(A7), A7
1B7C 6600 00AE BNE (* + $B0) ; if it's nonzero, bail
1B80 2F0B Move.L A3, -(A7) ; push nullptr?
1B82 4878 0000 Pea.L ($0) ; push 0
1B86 4878 0040 Pea.L ($40) ; push 64
1B8A 486E FF98 Pea.L -$68(A6) ; push ptr to somewhere on stack
1B8E 486E FFA8 Pea.L -$58(A6) ; push ptr to somewhere else on stack
1B92 486A 0028 Pea.L $28(A2) ; push ptr to $10F0 in hash object?
1B96 4EB9 0000 13D8 Jsr ($13D8).L ; jump somewhere, get processed data we care about on stack
$12E0 copies our signature from the PippinAuthenticationFile into a working buffer shortly after a copy of part of the data block in ROM passed to 'rvpr' 0 upon initial invocation. Could the "processed data" coming from the subroutine at $13D8 be our decrypted signature? I took a look at the memory before and after the call, and...
Look at memory location $20451.
When I saw it, I gasped. There it is. There's our decrypted digest.
I wasn't as lucky with the RSA code as I was with MD5—neither the reference implementations 1.0 nor 2.0 of RSA have portions that appear in this code, but they do answer the question of the signature format. The bytes appearing before our decrypted digest are a header consisting mostly of magic numbers and some $FF padding bytes, but with a lonely "05" byte at address $2044C, or offset 24 into our decrypted signature. This byte's value indicates that the digest is an MD5 digest, just like the reference implementation specifies.
That completed my understanding of the format of the PippinAuthenticationFile, leaving only one final piece of the puzzle: what and where the public key is. The public key must come from somewhere, but at this point I hadn't yet determined the purpose of the data passed in from ROM to 'rvpr' 0...
RSA (in which I dust off my math minor)
The RSA algorithm for cryptosecurity, invented in 1977 by Ronald Rivest, Adi Shamir, and Leonard Adleman, is built upon the notion that factoring large semiprime numbers is considered a hard problem. Not impossible, but very hard. Finding these primes can take a computer or cluster of computers a significant amount of time, proportional to the size of the semiprime to factor. Mathematically, it involves a few steps but can be implemented using basic algebraic concepts.
First, find two numbers and such that they are both prime, meaning that both and can only be divided neatly by themselves and one. Let's use and as examples.
Next, compute . in our example case. Call this .
Now we need to calculate . We do this by computing . .
We need to choose a value for such that and are coprime; that is, is not neatly divisible by . The smallest value that works here is 5, so we'll use that in our example case, but typically a larger value is used with a small amount of 1 bits in its binary representation.
Finally we need to find the value of . can be found by solving the equation . We do this by using the extended Euclidean algorithm.
All we do here is integer divide by and also take ; that is, we divide by with remainder. Then repeat this process with the results until the remainder equals one: divide the divisor by the remainder from the previous calculation. For example, start with :
Take the divisor from that and divide by the remainder:
One more time:
Once we have a remainder of one, we've found the greatest common divisor and so we need to build back up to find our value for . Do so by substituting our results until we have a linear combination of and (288 and 5, respectively):
Since we need to satisfy , we ignore 's coefficient here, leaving with a coefficient of -115. equals 288 (as we calculated earlier), so is 173, giving us our value for .
We now have everything we need to sign and verify messages. Our "private key" is our values for and —messages signed with the private key can only be verified by someone with our "public key". Our "public key" is our values for and —only our public key can verify messages signed by our private key, assuring our recipient that the signed message indeed comes from us and can be trusted.
Let's say we want to send someone the answer to the ultimate question, but we want to sign it first in case our message gets intercepted by Vogons. 😛 Call the original message with the value 42, and here we'll calculate the signature . We do so using our values for and :
When our recipient receives our message, they will need to verify our signature in order to be sure it can be trusted and that it has not been tampered with by any poetic aliens. 😉 They do so using our values for and :
If the signature matches the original message (as it does here in our example), the message arrived safely intact.
Notice that the values for , , , , , and are all relatively small—the largest of these, , only needs nine bits for its binary representation. You could therefore say that we used a 9-bit key in our above example. Furthermore, notice that our message 42—and its signature 111—also fit inside of nine bits. This is a property of RSA: a key of bit length X can only operate on a message with a maximum bit length also of X.
The signature size as defined in the PippinAuthenticationFile is 45 bytes, suggesting that the Pippin's public RSA key is at least 360 bits long (45 * 8 bits per byte). Recall from earlier that although the RSA public key is still unknown, it must come from somewhere in the ROM, and it is still at this point unclear what purpose the blocks of data passed to 'rvpr' 0 serve.
I found part of one of the blocks in RAM near where the decrypted signature is: 45 bytes, same as the signature. I also found a nearby value of 0x10001, or 65537, which seems to be a popular choice for the value of . Hmm. Interesting.
I found another block of memory also nearby containing the same data, but reversed by 16-bit words. Hmm. Interesting.
Wonder what the odds are... 😉
It didn't work. The "decrypted" signature didn't match at all. Garbage in, garbage out. Cue sigh of disappointment.
I had one data block left, and with little hope remaining...
I'd found it.
Apple's public key for verifying the signature on a PippinAuthenticationFile is: E0 E0 27 5C AB 60 C8 86 A3 FA C2 98 21 79 54 A8 9F D1 B9 DC 8A BA 84 EF B1 E7 C9 E2 1B F7 DD D7 DC F0 E4 4A BB 79 51 0E 7C EB 80 B1 1D
... and I didn't even have to look at very much code. 😀
I just have to crack RSA now, right?
Fortunately, available tools make that a much less daunting prospect than popular media contemporary with the Pippin suggested. There was even an ongoing RSA Factoring Challenge for a while until 2007. Back then though, it was a different story. The Open Source Initiative had yet to be founded. Prime factoring was done mostly in isolation by dedicated teams with access to massive amounts of computing power (for the time). A 364-bit decimal number took a two-person team and a supercomputer about a month to factor in 1992.
But this isn't 1992 anymore. The computers on our desks and in our pockets have more than enough number-crunching power to factor the Pippin's public key. Today, with some freely available open source software and a typical desktop PC, a 360-bit key can be factored in a matter of hours. And thanks to the efforts of several open source projects within recent years, we have a little tool to help us called msieve. 😀
msieve is very user-friendly. 😉 You pass the number you want to factor as its only command line argument and it just goes. It even saves its progress to disk, just in case it's a Really Big Number and something terrible happens like a power outage or something.
msieve took 18 hours, 34 minutes, and 4 seconds on my i7 Intel NUC to find two prime factors and of the Pippin's public key:
Let's plug these into our RSA formulas from above and find Apple's private key, shall we?
0F 2D 25 BF 3C 5B 70 28 72 6E 49 75 3F D5 62 67 11 37 38 94 51 EF D7 0E D1 47 5D E1 92 41 28 59 2C 4B 3E 47 4E 5F C1 23 1F 1B AF A0 D8 2B 0x10001 E0 E0 27 5C AB 60 C8 86 A3 FA C2 98 21 79 54 A8 9F D1 B9 DC 8A BA 84 EF B1 E7 C9 E2 1B F7 DD D7 DC F0 E4 4A BB 79 51 0E 7C EB 80 B1 1D (this is the public key—we know this from stepping through 'rvpr' 0 and examining memory) E0 E0 27 5C AB 60 C8 86 A3 FA C2 98 21 79 54 A8 9F D1 B9 DC 8A BA 66 F1 44 CA AB F4 6A A7 12 3D 48 3D 5D 26 F9 51 1C B8 28 A7 8D E9 1C 01 1C D3 AD E7 99 86 67 D6 E9 E2 17 11 DB EC 33 07 B6 0E 4D 6D 03 26 20 77 5D DB 9B 3B 64 CF 22 B2 0E 4A F3 2F 07 40 EE B0 6F 85 F2 A0 1D
Its first four bytes indicate a message size of $FD4F, or 64847 bytes. The MD5 digest of those first 64847 bytes is AE 1A EC AE A4 C5 11 68 2E 38 7D D1 48 F0 55 C2.
It has the following signature: 5A 90 36 69 DD 06 F5 15 EF 7A A2 04 5D 24 C2 CA 3C DD 2E C3 85 7D BB B8 9C 53 78 24 65 CC F0 0A 52 09 20 76 E1 9D F7 CC B3 C6 6D 7E AF
Notice that the last 16 bytes match our computed MD5 digest.
Finally, if we take the decrypted signature and re-sign it using what we think is Apple's private key, we get: 5A 90 36 69 DD 06 F5 15 EF 7A A2 04 5D 24 C2 CA 3C DD 2E C3 85 7D BB B8 9C 53 78 24 65 CC F0 0A 52 09 20 76 E1 9D F7 CC B3 C6 6D 7E AF
... which matches the original signature found in the PippinAuthenticationFile.
The RSA keys used in the signing and verification of a PippinAuthenticationFile are 360 bits long.
Apple’s public key for verifying the PippinAuthenticationFile is: E0 E0 27 5C AB 60 C8 86 A3 FA C2 98 21 79 54 A8 9F D1 B9 DC 8A BA 84 EF B1 E7 C9 E2 1B F7 DD D7 DC F0 E4 4A BB 79 51 0E 7C EB 80 B1 1D
Apple’s private key for signing a PippinAuthenticationFile is: 01 1C D3 AD E7 99 86 67 D6 E9 E2 17 11 DB EC 33 07 B6 0E 4D 6D 03 26 20 77 5D DB 9B 3B 64 CF 22 B2 0E 4A F3 2F 07 40 EE B0 6F 85 F2 A0 1D
There you go, Internet. We now have all the information we need to sign and boot our own Pippin media.
It’s been a while. I haven’t lost interest, I’ve just been busy with work and other things. Life has a funny way of sneaking up on you. 😉
The wait is worth it, though. Buckle up; this one’s a doozy.
Apple hasn’t documented the Pippin’s authentication process beyond what developers needed to know. There exists a technote that was distributed via the SDK(s) that gives an overview of what developers were expected to do to get their discs signed before final mastering and duplication. The Pippin’s authenticated boot process hinges upon the presence of a specially-crafted, RSA-signed file unique to each disc called the “PippinAuthenticationFile.” Since the Pippin platform was abandoned and subsequently cancelled in 1998, Apple no longer signs Pippin discs nor have they made available the means for third parties to do so. To my knowledge, most of the specifics of how the PippinAuthenticationFile plays a role in the Pippin’s boot process have never been documented outside of Apple.
That changes today.
This post is pretty dense, so I highly recommend (re-)reading parts 1through4 for some background before getting too deep. Otherwise, here’s a quick recap: during every boot, a retail Pippin console locates a potential boot volume on CD, loads an ‘rvpr’ 0 resource from ROM, then calls the code therein in order to verify that the target volume passes an authentication check allowing it to boot the system. (An aside: Previously, I asserted that while I found identical copies of ‘rvpr’ 0 in the 1.2 and 1.3 ROMs, I couldn’t find an entry for it in the resource map, therefore it must either be dead code or called some other way. This conclusion turned out to be incorrect—the resource map is not contiguous in ROMs 1.2 and 1.3, which made manually searching it more difficult, but it does indeed contain an entry for ‘rvpr’ 0. The authentication process is therefore identical between ROM 1.0 and 1.2.) When I last looked at ‘rvpr’ 0, I was stymied by a routine called upon entry which, absent of any symbols to help point me toward its purpose, I conjectured used a complex block of data at the end of the resource to “decrypt” the code therein. After taking a closer look a few days ago, I was delighted to find that its purpose is much simpler—it exists to patch the absolute memory locations in the code so they are relative to the buffer where ‘rvpr’ 0 is loaded. Without these patches, the code would crash the Pippin on boot practically every time!
The way this routine accomplishes this is kind of elegant. We initialize a cursor pointer to the beginning of our buffer where ‘rvpr’ 0 is loaded. The offset table starting at offset $8A47 from the start of ‘rvpr’ 0 begins with a 32-bit longword defining the size of the table. Then, the table itself is compressed: a byte with bit 7 set means it’s a relative sign-extended 7-bit offset from our cursor position, a byte with bit 6 set means it along with the next byte form a sign-extended 14-bit offset from our cursor position, but if both bit 6 and 7 are clear, then combine the next three bytes to form a 30-bit absolute cursor position. Multiply these offsets by two before applying them (because 68K opcodes are always at least two bytes), add the address of our ‘rvpr’ 0 buffer to the 32-bit longword pointed to by our cursor, then repeat the process until we’ve exhausted the offset table. Easy peasy.
5E 1218 Move.B (A0)+, D1 ; grab the next byte into D1, we'll call it the command byte
60 1001 Move.B D1, D0
62 0240 0080 AndI #$80, D0 ; is bit 7 set?
66 670C BEQ.B @checkBit6 ; then handle the bit 6 case
; command byte bit 7 is set, so
; D2 += signExtend(D1 * 2) as a byte (* 2 because alignment)
68 D201 Add.B D1, D1
6A 1001 Move.B D1, D0
6C 4880 Ext D0
6E 48C0 Ext.L D0
70 D480 Add.L D0, D2
72 6028 Bra.B @gotOffset
; else, command byte bit 7 not set...
74 1E81 Move.B D1, (A7) ; put D1 into the highest byte of temp
76 1F58 0001 Move.B (A0)+, $1(A7) ; grab the next byte into the 2nd byte of temp
7A 1001 Move.B D1, D0
7C 0240 0040 AndI #$40, D0 ; is bit 6 of D1 set?
80 670C BEQ.B @get32BitOffset ; yes? then
; command byte bit 6 is set, so
; our address offset is only 14 bits
82 3017 Move (A7), D0 ; grab the new temp into D0
84 E548 LsL #2, D0 ; D0 <<= 2
86 E240 AsR #1, D0 ; D0 /= 2
88 48C0 Ext.L D0 ; sign extend it
8A D480 Add.L D0, D2 ; D0 is the found offset * 2 (because alignment), add to our current offset
8C 600E Bra.B @gotOffset ; apply it
; bit 6 not set...
8E 1F58 0002 Move.B (A0)+, $2(A7) ; grab the next byte into the 3rd byte of temp
92 1F58 0003 Move.B (A0)+, $3(A7) ; grab the next byte into the 4th byte of temp
96 2417 Move.L (A7), D2 ; D2 is a brand new offset!
98 E58A LsL.L #2, D2 ; D2 <<= 2
9A E282 AsR.L #1, D2 ; D2 /= 2
; D2 == the offset we want to apply to argument 2
; D6 == the offset we want to apply to the longword found there (typically @start)
9C DDB1 2800 Add.L D6, $0(A1,D2.L) ; add D6 to the longword at (@start + D2)
A0 5385 SubQ.L #1, D5
A2 4A85 Tst.L D5 ; are we out of longs to patch?
A4 6EB8 BGT.B @loopBody
Now that we've "unpacked" the code of 'rvpr' 0, let's dig into it. 🙂
main starts off initializing a number of globals, first by calling InitRSAAlgorithmChooser and then a handful of other subroutines. It then initializes some local variables on the stack: previous A4, values related to the PippinAuthenticationFile, and a ParamBlockRec for calls to _Read. In addition, among those locals is a 16-byte temporary buffer for digests created during the main loop.
Recall from part 2 that the Pippin ROM passes as input to 'rvpr' 0 the following arguments: two pointers to some as-of-yet-unknown data in ROM shortly preceding the callsite, the ID of the boot volume candidate, and the refNum of the candidate's disk driver. After we've initialized our variables, we hit the ground running by calling GetVolAuthFileInfo to fetch the offset to and size of the PippinAuthenticationFile. Note that if at any point during 'rvpr' 0 one of its internal subroutines fails, the entire process is reported as having failed the authentication check.
24C 41EE FFAA Lea.L -$56(A6), A0 ; A0 -> temp var for created digests
250 2E08 Move.L A0, D7 ; D7 == ptr to temp digest
252 486E FFCA Pea.L -$36(A6) ; pass size out address
256 486E FFBE Pea.L -$42(A6) ; pass offset out address
25A 3F05 Move D5, -(A7) ; $10(A6) is dqDrive passed in from ROM
25C 3F2E 0012 Move $12(A6), -(A7) ; $12(A6) is dqRefNum passed in from ROM
260 4EB9 0000 03E6 Jsr GetVolAuthFileInfo
266 3600 Move D0, D3
268 4A43 Tst D3
26A 4FEF 000C Lea.L $C(A7), A7
26E 6600 0142 BNE @mainCleanup ; if GetVolAuthFileInfo returns nonzero, fail
272 202E FFBE Move.L -$42(A6), D0 ; D0 = offset from start of HFS volume to PippinAuthenticationFile in allocation blocks
276 7209 MoveQ.L #9, D1
278 E3A8 LsL.L D1, D0 ; D0 = offset from start of HFS volume to PippinAuthenticationFile in bytes
27A 2D40 FFC6 Move.L D0, -$3A(A6) ; save the offset into -$3A(A6)
27E 202E FFCA Move.L -$36(A6), D0 ; D0 = size of the PippinAuthenticationFile in bytes
282 A11E _NewPtr
284 2648 MoveA.L A0, A3
286 200B Move.L A3, D0
288 4A80 Tst.L D0
28A 660E BNE.B @gotAuthBuffer ; if _NewPtr returns null, clean up the stack, and fail
28C 554F SubQ #2, A7
28E 3EB8 0220 Move (MemErr), (A7)
292 301F Move (A7)+, D0
294 3600 Move D0, D3
296 6000 011A Bra @mainCleanup
29A 3D6E 0012 FFE6 Move $12(A6), ioRefNum(A6)
2A0 3D45 FFE4 Move D5, ioVRefNum(A6) ; dqDrive
2A4 2D4B FFEE Move.L A3, ioBuffer(A6)
2A8 2D6E FFCA FFF2 Move.L -$36(A6), ioReqCount(A6) ; size of the PippinAuthenticationFile
2AE 3D7C 0001 FFFA Move #fsFromStar, ioPosMode(A6)
2B4 202E FFBE Move.L -$42(A6), D0 ; offset from start of HFS volume to PippinAuthenticationFile in device blocks
2B8 7209 MoveQ.L #9, D1
2BA E3A8 LsL.L D1, D0 ; get the offset in bytes by multiplying by device block size (512 bytes)
2BC 2D40 FFFC Move.L D0, ioPosOffset(A6)
2C0 41EE FFCE Lea.L -$32(A6), A0
2C4 A002 _Read
2C6 3600 Move D0, D3
2C8 4A43 Tst D3
2CA 6600 00E6 BNE @mainCleanup ; if _Read returns something other than noErr, fail
Following this call, we allocate enough space for the file by calling _NewPtr. We then call _Read with our local ParamBlockRec filled with the disk driver refNum, the volume refNum, the pointer to the buffer we just allocated, our buffer's size, and the byte offset to the authentication file on our candidate. Armed with the contents of the PippinAuthenticationFile, we then pass them to VerifyDigestInfo to verify the signature contained therein. If that succeeds, we're clear to start verifying the candidate's contents against this file, so we allocate temp space large enough to load a single "chunk" of data from the candidate to be hashed and verified.
Every Pippin except the KMP 2000 shipped with a built-in 4x speed CD-ROM drive. A 1x CD-ROM drive can read data at a rate of 150KB/sec, which is the speed necessary for smooth playback of audio CDs. A 2x drive doubles that rate to 300KB/sec, a 4x drive quadruples it to 600KB/sec, and so on. At 600KB/sec, it would take a Pippin almost a full minute to read just over 35MB, and nearly 20 minutes to read the entire contents of a 700MB CD-ROM. Even the KMP 2000 with its 8x drive would take almost 10 minutes to do the same. Hashing the entire contents of a CD during every boot would be unacceptable at this speed, and since the Pippin only takes a couple seconds to verify a disc at startup, it's clearly not verifying the whole thing. So what does the Pippin do?
320 4A86 Tst.L D6
322 6604 BNE.B @pickRandomChunk
324 7800 MoveQ.L #0, D4
326 6016 Bra.B @readChunk
328 202B 004C Move.L $4C(A3), D0 ; longword after the 128K size field at $48, appears to be number of entries in table
32C 5380 SubQ.L #1, D0
32E 2F00 Move.L D0, -(A7) ; upper bound == total number of chunks
330 4878 0001 Pea.L ($1) ; lower bound == 1
334 4EB9 0000 0814 Jsr RangedRand ; patched
33A 2800 Move.L D0, D4 ; D4 == pseudorandom integer between [1, <total number of chunks>]?
33C 504F AddQ #8, A7 ; clean up stack
33E 2004 Move.L D4, D0 ; D0 == pseudorandom integer in the lowword, probably
340 2205 Move.L D5, D1 ; D1 == 128K? 0 x 0002 0000
342 4EB9 0000 0116 Jsr _D0timesD1 ; patched, does some weird multiplication, returns in D0
348 2D40 FFFC Move.L D0, ioPosOffset(A6) ; D0 == the pseudorandom integer * 128K? D0 == offset to random 128K chunk in disc?
34C 41EE FFCE Lea.L -$32(A6), A0
350 A002 _Read
352 3600 Move D0, D3
354 4A43 Tst D3
356 665A BNE.B @mainCleanup
Put simply, the Pippin randomly spot-checks the candidate volume's contents every boot. The PippinAuthenticationFile isn't just a key, it isn't just a single hash—it is in fact a collection of hashes corresponding to as many 128K chunks of data that make up the boot volume. main enters a loop that iterates six times: the first check, it loads the first 128K of the volume containing important metadata about the HFS filesystem into our temporary buffer, and then verifies that data against its corresponding digested hash previously loaded from the PippinAuthenticationFile. The remaining five checks, it does the same, but on randomly selected other 128K chunks of the volume. This way, the Pippin only has to load and verify 768K—a process that takes less than a couple seconds on its 4x CD-ROM drive. But because this loop selects five of those six input chunks at random each run-through, the PippinAuthenticationFile still needs digests of the entire volume. For it's not known ahead of time which five chunks will be verified and furthermore, they rarely will be the same five chunks.
Examining several PippinAuthenticationFile examples with this code in mind quickly reveals how this file is structured. Both the chunk size and the total number of chunks in the volume are stored in a common header. This loop uses that information to determine the upper bound of which chunks to select at random and how large. Following these two fields is a table of digested 128-bit hashes corresponding, in sequential order, to the chunks in the volume. Finally, there is a signature near the end of the file, which gets verified in the call to VerifyDigestInfo before entering the loop. The process by which a PippinAuthenticationFile is created, therefore, is essentially as follows:
Get the size of the target volume.
Integer divide this size into 128K chunks. Call the total number of chunks N.
Allocate 80 bytes for a file header.
Multiply N chunks by the 16-byte size of each digest (N * 16). Call this table size T. Allocate T bytes for digests of each 128K chunk.
Allocate 16 bytes for the signature size S.
Pad the signature size until it is a multiple of 16 bytes (((S + 16) % 16) *16). Call this padded size P. Allocate P bytes for the signature itself.
Pad additional bytes until the total file size is the next multiple of the device block size (512 bytes). The total file size therefore should be ((80 + T + 16 + P + 512) % 512) * 512.
Preallocate a blank version of this file on the target volume.
Starting at offset 80, compute and store a 16-byte (128-bit) digest for each of the N sequential 128K chunks. Note that to compute the digests for the entire finalized volume correctly, this file must already exist in the filesystem. It is therefore necessary to compute the size of this file in advance, preallocate a "dummy" version of it on the target volume, then compute the digests and overwrite the file in-place.
At offset 80 + T + 15, store the signature size S as a byte (only one byte at the end of this space is actually used, the rest are zeroes).
At offset 80 + T + 16 + 3, store the signature itself. The signature always seems to be 45 bytes long, placed such that it ends on a 16-byte boundary, explaining the extra 3-byte offset.
Fill in the file header at the beginning of the file:
offset 0 (4 bytes): offset to signature size byte (80 + T + 15)
offset 4 (4 bytes): longword equal to zero (version?)
offset 72 (4 bytes): chunk size longword equal to 128K, or $20000
offset 76 (4 bytes): chunk count longword equal to N
Apple probably provided a tool that automated this process for stamping houses. Said tool presumably would have named the aforementioned file "PippinAuthenticationFile" with type/creator 'PpnV'/'PpnA' and saved it to the filesystem root. I imagine that this same tool likely would have filled the file's contents in-place with the signed version received from Apple. However, I have never seen such a tool in the wild so this is pure speculation on my part.
Incidentally, the name, placement within the folder hierarchy, and type/creator codes of the authentication file itself are inconsequential. The Pippin makes no HFS calls to locate the PippinAuthenticationFile—it could technically be buried within a nest of folders or named "FoobarAuthenticationFile." The verification code does not care. Instead, it fetches the Master Directory Block—512 bytes located at byte offset 1024 from the start of the boot volume. The "logical" MDB is a data structure 161 bytes in size and found immediately at the start of this "physical" MDB. However, that leaves 351 bytes unaccounted for. For Pippin CD-ROMs, Apple chose to set aside two 32-bit longwords at the end of the physical MDB for the purpose of locating the PippinAuthenticationFile at the block level. The first of these longwords defines the offset, in 512-byte blocks from the start of the volume, to the contents of the authentication file. The second of these longwords define the authentication file's size in bytes.
(As an aside, this mechanism is one reason why deleting the PippinAuthenticationFile and naively replacing it with a new version at the filesystem level is not likely to work. The new file would likely reside starting at a different allocation block in the volume; the offset in the MDB would still point to where the deleted file was/is, and HFS wouldn't know to patch it up—why should it?)
One important component of creating the authentication file, and verifying against it, is the concept of chunk "cleansing." Once the loop selects and loads a chunk, it passes it to CleanseInputChunk to optionally "cleanse" it. What does that mean in this context?
358 2F2E FFCA Move.L -$36(A6), -(A7) ; size of the PippinAuthenticationFile in bytes
35C 2F2E FFC6 Move.L -$3A(A6), -(A7) ; offset from start of HFS volume to PippinAuthenticationFile in bytes
360 2F05 Move.L D5, -(A7) ; 128K
362 2F2E FFFC Move.L ioPosOffset(A6), -(A7) ; the computed offset
366 2F0A Move.L A2, -(A7) ; working chunk buffer
368 4EB9 0000 049A Jsr CleanseInputChunk ; patched, remove the auth file from this chunk if we happened to land on it
The digested hashes contained within the authentication file do not include hashing the authentication file itself, for obvious reasons. Similarly, certain fields in the MDB should not be hashed because they change upon writing the final signed authentication file to the volume. "Cleansing" solves this problem by zeroing out these areas before hashing and digesting them. While creating an authentication file, the blocks of the volume containing the file itself should be zeroed out when hashing, and likewise most of the MDB. Upon loading a chunk, the verification loop checks its offset to see whether the chunk overlaps the MDB or the authentication file. If so, it zeroes out that data so that the digested hash matches the corresponding one stored in the authentication file.
Finally, the loop passes the chunk to CreateDigest, and compares its result byte-for-byte with the digest in the authentication file by calling CompareDigests. If all six digested chunks match what's found in the authentication file, we pass the check (and can boot from this volume! Yay!). Otherwise, we return -1 to indicate failure.
Astute readers may notice that I have provided sparse details about the functions related to dealing with digests and signatures so far. Those are next on my list but will also be the hardest to grok because I'll be more or less "flying solo" without any symbols whatsoever to guide me. Fortunately I have a passing familiarity with the RSA algorithm so I have a vague idea of what logic to look for. I have already found functions for 32-bit multiply and 32-bit modulo, both of which are essential for RSA.
In part 4 I explored hacking the Apple Partition Map to load custom disk drivers before the authentication check takes effect. I was not successful with that experiment and decided not to pursue it further, but my discoveries here reveal a possible alternate avenue to explore: additional partitions. The authentication check is performed on the boot volume, and only the bootvolume (emphasis on each word). The partition map is not included in the check, nor are any other partitions. It should be totally possible to take an existing signed Pippin disc with unpartitioned free space available (for example, the "Tuscon" disc) and graft an additional HFS partition containing whatever apps or documents one might want. The OS should mount the other partition as it would normally, without performing any checks beyond what would be done on a real Mac.
I leave that as an exercise for the reader. I'm going after the big fish: authoring an entire homebrew disc from scratch.
According to the NetBSD/macppc FAQ, Open Firmware “is part of the boot ROMs in most PowerPC-based Macintosh systems, and we use it to load the kernel from disk or network.”
Turns out, “most PowerPC-based Macintosh systems” happens to include the Pippin. If you have the rare keyboard/tablet (or an ADB keyboard via the AppleJack dongle) attached and hold down Command-Option-O-F at startup, the Pippin boots to an Open Firmware prompt. However, you won’t see anything on screen because it outputs to a serial console by default; specifically, all console I/O is handled through the GeoPort. My Mac Plus happens to sit next to my Pippin, so tonight I temporarily switched my ImageWriter II’s cable over, booted both machines, and fired up ZTerm.
The following is what I discovered.
Open Firmware, PipPCI.
To continue booting the MacOS type:
To continue booting from the default boot device type:
0 > dev / ls
0 > dev /openprom ok
0 > .properties
model Open Firmware, PipPCI.
0 > printenv auto-boot?
auto-boot? true true
0 > printenv use-nvramrc?
use-nvramrc? false false
0 > printenv real-base
real-base -1 -1
0 > printenv load-base
load-base 4000 4000
0 > printenv boot-device
boot-device /AAPL,ROM /AAPL,ROM
0 > printenv boot-file
0 > printenv input-device
input-device ttya ttya
0 > printenv output-device
output-device ttya ttya
0 > printenv nvramrc
0 > printenv boot-command
boot-command boot boot
0 > bye
This dump was generated on my @WORLD Pippin with ROM 1.2. Some observations:
OF doesn't report a version number, instead reporting "PipPCI" in its place. Searching the ROM for strings reveals "June 28, 1996" as the latest date I could find, so whatever Apple was using in its Power Macs at that time I imagine is what is running here.
The ROM is located at 0xFFC00000, which follows what I've seen from hardcoded addresses I've found.
"taos" is the video hardware starting at 0xF0800000. I'm not sure offhand if that address is the base of video memory, but I do know from the Pegasus Prime code that taos does allow for writing directly to VRAM.
There is a TFTP package(!)-- wonder how it works?
The Pippin has a SWIM III chip onboard. There is an official floppy drive expansion dock and an unofficial floppy drive expansion board, both of which appear to be "dumb" hardware that merely connect a drive directly to pins of the Pippin's X-PCI connector on the underside of the system. The drive itself is powered and controlled entirely by hardware already built in to the Pippin. However, as far as I know, the SWIM II and later floppy controllers (including the SWIM III) lack the low-level access necessary for HD20 support, so large drives emulated by hardware such as the Floppy Emu will not work.
In part 1 of this series of posts, Daniel suggested taking a look at the role the SCSI Manager plays in the Mac’s– and by extension, the Pippin’s– startup process. I took a break from examining the auth check code to dissect how the SCSI Manager behaves at startup on the Pippin, with the ultimate goal to discover if an exploit was even possible. After all, from my initial search of the ROM it’s clear that Apple had patched part of the boot process to only allow starting up from the internal CD-ROM drive– who’s to say there aren’t further patches deep in the SCSI code?
The tl;dr is that, after careful examination, I have determined that the SCSI code in the Pippin’s boot process is for the most part unchanged compared to a Mac of the same era. However I’m not yet sure if a “patch partition”-based exploit is possible.
Every SCSI-equipped Mac for the most part follows the same procedure for booting from a SCSI device:
It looks for a valid Driver Descriptor Map in the first 512-byte block
Loads the device’s driver(s) according to the information found in the DDM
Mounts the disk’s HFS partition
Loads the boot blocks from the startup volume and executes them
This has been the case since SCSI was introduced to the Macintosh line with the Mac Plus in 1986. Previously, Macs had primarily booted from floppy disk and occasionally from a non-SCSI hard drive such as the HD20 or HyperDrive (but usually with help from a boot floppy). In either event, the Mac had no concept of drive “partitions” or even “volumes” in the modern sense– the MFS file system consumed the entire disk, and that was it.
The Apple Partition Map, introduced at the same time SCSI arrived on the Mac, changes this behavior by placing the primary file system in its own partition. On a high level, the Mac sees the volume as its own big disk, but the APM also allows device drivers to live in their own partitions rather than in INIT files as was done with the HD20. This moves their load time to much earlier in the boot process independent of the System Software and– critically– eschews the need for a separate boot disk. The SCSI code in ROM recognizes the presence of an APM, loads the driver(s) directly from disk, then locates and boots a startup volume. In this way, Macs can boot directly from a hard drive, a CD-ROM drive, or any SCSI-capable storage medium so long as it has the proper drivers preconfigured.
What does all of this have to do with the Pippin? Well, the Pippin’s only internal storage is its 128KB of non-volatile Flash memory, and even if the Pippin ROM was written to look there for an OS, 128KB is barely enough to boot the original Macintosh– never mind the System 7.5-based OS that Pippin titles shipped with. Instead, the Pippin boots from its internal (SCSI) CD-ROM drive, presumably calling into the SCSI Manager to load any device drivers followed by mounting the HFS partition, verifying the contents of the “PippinAuthenticationFile,” then continuing to boot the OS if everything checks out.
Wait a minute…
Before searching for an operating system, the SCSI Manager is supposed to load device drivers directly from disk according to the Driver Descriptor Map. It does a basic checksum verification, then… executes them! They run before the auth check!
At least, in theory.
I took a look at the “Pippin Network CD” (product ID BDB-002) that originally shipped with the 1.0 Atmark units in Japan. I chose this particular disc because of two things: 1) It has an Apple Partition Map and 2) It contains a very small device driver partition with plenty of extra room in which I could hack a “proof-of-concept.” I rustled out the original driver from an ISO image of the disc with a hex editor, then added some code that does the equivalent of while (!Button());. I then recalculated my hacked driver’s checksum using a quick program I wrote, poked the driver (and its proper checksum) back into the ISO, burned it to disc, booted my Pippin with it, and…
It booted directly to the login screen.
This tells me that one of two things might be happening:
The Pippin’s version of the SCSI Manager in ROM does not in fact load device drivers from disc. The SCSI Manager does a number of checks before it determines that it needs to load drivers for a particular device, and it’s possible that the Pippin fails one of those checks, possibly intentionally. In this case, it may be that the driver lives in ROM and any drivers on Pippin discs exist for the benefit of mounting on actual Macs, where the hardware isn’t fixed and device drivers would be more necessary. Also, only a handful of Pippin discs have partition maps; the majority of them are formatted as a flat HFS volume. What does the SCSI Manager do in that case? Where do the requisite device drivers come from?
I screwed up hacking my driver code into the disc image before burning it. 😛
Unfortunately, when I mount this disc on my Power Mac G3 in Mac OS 9.2.2, my driver code doesn’t run there either. There must be some condition(s) under which the SCSI Manager will skip loading drivers from a CD-ROM mastered with them.