Lightweight Continuous Documentation Build

You know this fixation we all have with the "document"? Yeah, that's the one, the one where we get all precious and don't want to show anyone the "document" until it's all nicely polished.
Well nuts to that!! I've been casting around for some time now to get a nice simple workflow that combines a Git repo with a lightweight wiki engine. I've finally got it up and working and happy. It's a bit of a sweet thing.
Here's how it works:

1. Write your stuff - you guessed it - in Markdown. Store the files in a Git repo.
2. Don't bother creating the "document". Instead break up what you are writing into manageable chunks and put each of those chunks into a file on its own.
3. Every now and then when you reach a point of wanting to go for a walk or running out of puff on what you are writing, do a commit in your repo. If you're not ready to show the world all of it then do a stash of the not-ready bits. But don't be too precious, this is about the conversation.
4. When you commit a post-commit hook runs. It does two things:
1. It issues a command "markdoc build". More on Markdoc in a minute.
2. It copies the resulting html files to the htdocs folder on the webserver.
5. The webserver serves your wiki content to the web.

It's really simple, really lightweight and really robust. Markdoc is the key component. It is a small Python script that takes the Markdown files and converts them to html and provides a tiny bit of structure. It can cope with a directory structure. It gives you back your html files along with a very simple directory listing "home page". Markdoc does this all with a two word command issued in Terminal on the Mac. It has an equally lightweight web server as part of the distro. Simply type "markdoc serve" and it will tell you where you can find the web pages with your browser.

I've chosen to ignore the built in server and instead I just copy the files across to the Mac server. That has a number of advantages. I can put the files for multiple wikis, each in their own directory, within the web root directory and serve from the web root directory. When I browse to the page I just get a directory listing, each of which is a wiki in its own right. When I open a wiki I get Markdoc's lightweight home page. It's very nice. Secondly I can use the Mac server's web hosting system to create a realm that requires access control. That means the wikis can be public or private as I please.

If I want to collaborate with someone on the writing, I don't do it on the wiki. That's only static html. Instead I get them to clone the Git repo and then push their changes back to the remote repo.

So why is this important? Well it means that we can share our progress with our team, our stakeholders and our collaborators. We don't get locked into "you'll have to wait for the next version of the document to see what's in it". Instead our team can see where we are up to, what we are thinking and where we have issues. It fosters discussion, collaboration and team work. It means we can't go off down a blind alley for too long before somebody notices; it means that people have a chance to see where we are going with our writing (and hence with the project) and to offer thoughts and ideas about how we might progress next.

It brings us one step closer to real-time collaboration. One step closer to breaking down the barriers that we've created, inadvertently, with the "document" and with our cumbersome tools. It allows us to step back in time using Git to reach a previous place if we realise we've drifted off track. It allows us to create a branch and work on an idea without fear of losing our work or getting confused. And all the while we are sharing our progress and sharing our ideas.

Very neat, it creates a very good working environment.

If you want some more details of the scripts and hooks let me know and I'll put them somewhere.

iPad at Work - Writing

This is the first of a series of posts about using the iPad at work. I'm not going to be writing about the iPad as a calendar tool or as a way to look at your email. That's just so fundamental that we'll take it as read that you already do that.
What I want to talk about are the more interesting uses of iPad at work.

Writing

This is pretty fundamental as well. We nearly all need to write at work and many users of iPad will be heavy users of the writing capability. So here are some thoughts about writing with the iPad.

Get a Keyboard!

Seriously you can't write more than a quick note without a keyboard. There are lots of options out there from keyboard cases to Apple's very nice wireless keyboard. Whatever your choice get it and make sure that you are comfortable touch typing on it before you shell out your hard earned.

Go Markdown!

On the iPad most of the tools for working with Word are, in a word, kludgy. They are overweight and limited in their functionality. It's just not worth it.
Markdown is the only sensible way to write anything other than notes on your iPad. Don't be put off by the first paragraph saying that "Markdown is a text-to-HTML conversion tool..." it's much more than that. It's the way you want to do your documents because:

• It's trivially simple to learn. It will take you 10 minutes to master it if you are as stupid as me. Most people take less time than that. There are cheat sheets available too. Here's a visual one. Download one and carry it with you for a while. Soon you won't need it at all.
• It is brilliant for re-purposing what you write. You can write in Markdown and output to docx, odt, html, LaTeX, ePub...the choices are endless.
• It is lightweight and human-readable. You can understand a Markdown document really simply.
• It's text so even as fads and file formats come and go it's text and you can open a Markdown file in almost any editor since the beginning of the computer age. That also means it has probably the best chance of future proofing and it frees you to use the tools that you want, not some heavyweight, expensive, proprietary tool.

Get Dropbox

It doesn't have to be Dropbox, however Dropbox support is becoming pervasive amongst the tools you are going to want to use. Whatever you choose you need a way to simply get things into a position where they are accessible on your iPad, on any web browser and on your desktop/laptop. Dropbox ticks all of those boxes.

Choose a Markdown Editor...or three

There are lots of good Markdown editors for iPad. I like Writeup for most stuff (I particularly like its version management and preview of document contents) and I also like Drafts, particularly because of its Actions and URL Actions. I use Drafts for quick notes and the like.
In addition you'll probably want to get TextExpander, it will improve your productivity on iPad.
On Mac you can choose anything you like, as long as it supports UTF-8. I use a variety of tools including Textwrangler, Markdown Pro and Byword. Textwrangler isn't a Markdown editor, but there is a language module available for Markdown and if I'm using this I use the very capable Marked to preview and output. Windows has similar tools available. Just don't use Notepad...it doesn't save as UTF-8.
For me, part of the beauty of using Markdown is that I can pick and choose tools for the particular writing task. If I'm writing something with lots of code snippets or code listings in it I use Textwrangler because its programmers' tools make jobs like that easy. If I'm writing a short note I use Drafts...The results are interchangeable.

Output

Most of the tools we've talked about provide output to at least PDF and HTML. Some of them have many more tricks than that.
What we are going to talk about in the next post is a trivially simple way to output anything you like from Markdown. But before that...

Workflow

Let's just be clear about one thing when you are using Markdown: Formatting doesn't matter...bear with me and don't start getting grumpy because...formatting doesn't matter. Here's why.
Word and other tools like it have fundamentally stuffed up the way we do documents. They have encouraged us to conflate two unrelated concepts: What a document element is; and How that document element should look. Those are two separate and unrelated concepts.
The only time that those two concepts come together is when we choose a particular presentation mode and a particular presentation tool. For example in Markdown this is a level one heading: #Heading One
Now obviously that's not how we want it to look. But how do we want it to look? What will it look like in a Word doc? What about on a web page? What about in that LaTeX document for our Uni assignment where it has to be in accordance with the APA style sheet? Each one of those will be different and can change as we modify the style for that element. The only thing that's important is that that element is a level one heading.
So having got that out of the way and having already talked about the way Markdown is human-readable...that means that we don't convert from Markdown to anything else until the end of our workflow. At least not until the end of our iPad workflow. We do all our revisions in Markdown, we share the document around in Markdown and when we are happy about it, then and only then, we transform it to whatever our desired output mode(s) are.
OK, I'm with you already: What about track changes you say. Well for me this is a beautiful thing. I hate Word's track changes functionality with a passion. It is kludgy and dense and after you've been modifying a document a few times it gets plain horrible. We don't do any of that with Markdown. We don't have track changes...and we don't need it. I just say to somebody who is reviewing a document for me "just make whatever changes you like". That's because I use Git to version all of my documents and I use Git's diff tools to tell me what's changed and I then decide to accept, reject or modify those proposed changes. Now most people don't use Git, but they can use the very beautiful Kaleidoscope tool or one of a number of other tools around to do the same job...better.
The only other thing to talk about is tables. These are a bit of a pain in Markdown. There is syntax for tables but I find them a bit painful to do manually. So you have three options: either you create your tables as objects (pictures) and embed them in Markdown using the image syntax; or you do them right at the end in Word; or if you can type and you can find the Terminal app in Mac or it's equivalent on other platforms you can use these tools. More on those and on output in general in a future post though.

Markdown to Word

So here's the use case:
You are increasingly working on your iPad at work. It means you don't have to transcribe hand written notes into some other document and you can write documents completely on your iPad. The problem is that the iPad and Microsoft Word are poor partners. The best way to write on the iPad is to use a Markdown editor. It's a simple markup language, anyone can learn it in 10 minutes and it allows the documents to be re-purposed to Word, HTML, LaTeX, ePub, PDF or whatever. And everybody in the business world wants something in MS Word.
The further problem is that whilst Markdown is the bees knees, it's not trivial to do tables in Markdown. In fact writing tables by hand in Markdown is a pain.
This is a real scenario that's been troubling a couple of people I know. So I've been working on a solution. Here's what the solution and tool chain look like:

1.  iPad combined with a Bluetooth keyboard. If you are going to write on the iPad then do yourself a favour and get a Bluetooth keyboard so you can touch type.
2. A Markdown editor. I like Writeup on the iPad but there are heaps of others.
3. iSSH on the iPad to access the server via ssh.
4. Dropbox. You have to get stuff on and off  your iPad and there's no better way than Dropbox.
5. Pandoc on an accessible Mac or Linux or Windows server. More on this below.
6. R statistical analysis package. Stay in your seat...it will be alright I promise.
7. These scripts.

Here's the workflow.

1. You create a Markdown document on your iPad and store it in Dropbox. Where you want a table in the document you put a marker of the form @filename.extension@ into the document on a line of its own.
2. Create the table data in MS Excel and export it as tab delimited into a separate named file for each table. The filenames must align with your markers from the step above, but without the @ at either end. Any strings with spaces in them in the table need to be quoted: "Jane Smith". Put these files in the same Dropbox folder as the document with the markers in it.
3. Use iSSH to generate and transfer a key to your server and set up the server to accept that key. That way you can create a session on the server without passwords. Set up iSSH to run a script on login that calls the Main.sh script from step 7 above. This is a one time setup.
4. Login using iSSH and choose GenerateTables.sh and run through selecting your table data and the output file name once for each table.
5. Now from Main.sh choose InsertTableMaster.sh and tell it which file has the markers in it and what you want the document called when it has the tables in it. This will replace the markers with the actual tables.
6. Now from Main.sh choose GenerateWord.sh and it will process your document into a Word document.

If this sounds complex it's not. One of the people using this has never used the Terminal and never wants to but they happily generate Word docs with tables in them. It is very quick. The table insertion process takes seconds to run and the conversion to Word takes under a second to run.
The package of scripts has a detailed Readme.md file with it and you can ask any questions on that site or in comments here. That link takes you to the overview page where you can read the documentation and learn about the very simple pre-requisites. From there you can either do a git clone if you are that way inclined or simply download the files as a zip.

Please give me feedback and suggestions in the comments.

Note 1: Dropbox doesn't update on a computer unless you are logged in. Therefore you need to have a user logged in to the console of the remote computer and that user has to at least have shared the Dropbox folders that you want to use. Otherwise you save a file to Dropbox on the iPad and when you ssh into the server it isn't in Dropbox.
Note 2: Setting up keys for ssh can take a moment but it's worth doing. If you have trouble say so in the comments and I'll write a post on it.

Drafts and Dropbox Sync

I like Drafts on iOS. It’s simple, quick and efficient. However the way it appends to Dropbox annoys me.In my Dropbox I have one directory called nvALT which is where I keep all my miscellaneous text files. I rely on nvALT to find stuff in those files and nvALT will only act across a single directory. I have a scratch file in nvALT that I use for all text clipping I do. I use a LaunchBar shortcut to append selected text to that file.
Drafts will append text to a file in Dropbox also. The problem is that it always puts it into /Apps/Drafts/Journal.txt and that appears to be a hardcoded path.
I don’t want two sets of text clippings, and I don’t want one of them outside of my nvALT directory. This sent me on a path to see if I could somehow move all the text into my scratch file in nvALT. I’m a rotten coder but I managed to hack something up that does the job. It’s in two parts. The first part is a bash script that checks to see if the /Apps/Drafts/ directory content has changed and then checks if the Journals.txt file has changed. If it has it appends the contents to scratch file.
The second part for Mac users is a plist that loads the script and keeps it chugging away.
As always, use these scripts at your own risk and I’d love to hear all the ways that you find to make them smarter, better, more elegant and more robust. I told you I’m a crap coder.

#!/bin/bash

#This script is designed to combine two scratch files that I use for holding snippets of text.
#They are both in ~/Documents/Dropbox. The first is in the /nvALT directory and is called ScratchX.txt I use it for clipping text to on Mac
#The second is used by Drafts on iOS and is in the /Apps/Drafts directory and is called Journal.txt
#I create a dummy JournalB.txt and use diff to check whether it and Journal.txt remain the same. If they differ we copy the diff to ScratchX.txt
#It requires a change in the /Apps/Drafts directory to trigger anything further.

#Create a directory listing file
touch ~/tmp/dirb.tmp


while true; do
  #List the directory and put it into dira.tmp 
  ls -l /Users/criticalold/Documents/Dropbox/Apps/Drafts > /tmp/dira.tmp
  #Check whether it differs from the last directory listing and tell us if it does
  diff /tmp/dira.tmp /tmp/dirb.tmp || echo 
  #Then do a diff on the files and send the diff to grep to get rid of some numbers that diff creates then append to ScratchX.txt.
  diff /Users/criticalold/Documents/Dropbox/Apps/Drafts/Journal.txt /Users/criticalold/Documents/Dropbox/Apps/Drafts/JournalB.txt | grep '^<' >> /Users/criticalold/Documents/Dropbox/nvALT/ScratchX.txt
  #Do our housekeeping for next time by making the *b files the same as the *a or the Journal file so that we do a valid comparison.
  cp /Users/criticalold/Documents/Dropbox/Apps/Drafts/Journal.txt /Users/criticalold/Documents/Dropbox/Apps/Drafts/JournalB.txt
  cp /tmp/dira.tmp /tmp/dirb.tmp
  #Wait for 25 seconds before checking again.
  sleep 25 
done

Now for the plist

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.critical.dropbox</string>
    <key>Program</key>
    <array>
        <string>/Users/criticalold/Applications/Utilities/Terminal.app</string>
    </array>
    <key>ProgramArguments</key>
    <array>
        <string>/Users/criticalold/Documents/ShellScripts/checkdirectory.sh</string>
    </array>
    <key>KeepAlive</key>
        <true/>
    </dict>
</plist>