#emacs | #GNU | #org | #programming

June 15, 2020

Emacs Org mode source blocks

Post updates

  • [2020-06-23] New section. "Combine source blocks"

Intro

It's time to start documenting how I use Emacs Org mode (from now on org) and it will contain a serie of posts. This is the first post and it will cover source blocks. Some of the content assumes basic knowledge about org. If you are new to org I can recommend this two videos; hrs and kitchin.

One of the features I've been using very frequently is source blocks. The power of evaluating code directly from a block and get the results back into the document is really convenient. While reading The AWK Programming Language I took notes and did all of the awk programming withing source blocks. This makes me able to have notes, code and execution results in the same document. And as everything is just text, it's easy to version control!

This is the structure of a source block. Org mode's documentation on source blocks can be found here.

#+BEGIN_SRC <language> <switches> <header arguments>
  <body>
#+END_SRC

Header arguments differ for different languages and applications. Look in the documentation for your specific language/application to find out what and how to use it.

Versions

… of Emacs and Org mode.

#+BEGIN_SRC emacs-lisp
  (org-version)
#+END_SRC

#+RESULTS:
: 9.3.7

#+BEGIN_SRC emacs-lisp
(emacs-version)
#+END_SRC

#+RESULTS:
: GNU Emacs 26.3 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.20)
:  of 2020-05-19

Emacs config

Without config Emacs don't know how to execute source blocks. This following snippet provides Emacs with the ability to understand a set of languages/applications. The set of languages/applications need to be available in the underlying OS. To get a complete list of languages/applications available in org. A complete list can be found here: repo. Support for source blocks are available in files prefixed with ob-.

(setq org-confirm-babel-evaluate nil)

(org-babel-do-load-languages
 'org-babel-load-languages
 '((emacs-lisp . t)
   (shell . t)  ; in my case /bin/bash
   (scheme . t)
   (python . t)
   (ledger . t)
   (sed . t)
   (awk . t)
   (clojure . t)))

Bash echo

This small example uses bash to evaluate the echo command. Evaluation is done by moving the pointer to the source block and press the key sequence C-c C-c.

What shell am I using?

#+BEGIN_SRC shell
  echo $SHELL
#+END_SRC

#+RESULTS:
: /bin/bash

Echo input variables

Arguments can be passed to source blocks from the document. This example passes 2 :var, ARG1 and ARG2. The command get the arguments and uses them in the command.

#+BEGIN_SRC shell :var ARG1="FOO" ARG2="BAR"
  echo $ARG1 $ARG2
#+END_SRC

#+RESULTS:
: FOO BAR

Document variables

Lets elaborate on argument to source blocks even further. Org documents can contain variables that holds data. This example contains a org table with the variable name my-list. my-list is passed as an argument to the Python source block via the variable lst. When executed by Python the lst data structure is a vector with vectors.

#+NAME: my-list
| A | 1 |
| B | 2 |
| C | 3 |

my-list is a variable in this org document and can be passed in as data to a
source code block.

#+BEGIN_SRC python :results output :var lst=my-list
  print(lst)
#+END_SRC

The data structure that will be passed to the source block is a vector or
vectors.

#+RESULTS:
: [['A', 1], ['B', 2], ['C', 3]]

Just like with a normal Python application you can manipulate the data.

#+BEGIN_SRC python :results raw :var lst=my-list
  return [[chr((ord(x)+1)), y+1] for x,y in lst]
#+END_SRC

#+RESULTS:
[['B', 2], ['C', 3], ['D', 4]]

Result output

Results output can be in many different shapes. Raw output, tables, as code and many more. Output results are described with the :results switch in the source block header. This post won't go more into details but more information can be found here.

More on document variables

This example provides another org document variable. In this case the text-example text block. The text in that block is passed as an argument to the awk source block. awk is a programming language with good features to manipulate text. It can take text from stdin. That's exactly what is happening here. Often you see awk used in shell commands, similar to this: cat /etc/passwd | awk -F":" '{ print $1 }'. Here awk gets stdin data from the pipe. But in our source block it gets stdin data from the org document.

#+NAME: text-example
#+BEGIN_EXAMPLE
  Item1 100
  Item2 200
  Item3 50
#+END_EXAMPLE

Sum field nr 2.

#+BEGIN_SRC awk :results table :stdin text-example
  BEGIN {OFS="|"}; { sum+= $2}; END { print "Sum", sum}
#+END_SRC

#+RESULTS:
| Sum | 350 |

File from file system

This example is very similar to the one in "Echo input variables". It passed a variable to the source block that tells a file path to awk. awk is then called with that :in-file as an argument.

Get system users that have =/bin/bash= access.

#+BEGIN_SRC awk :results table :in-file /etc/passwd
  BEGIN { FS=":"}
  $7 == "/bin/bash" { print $1}
#+END_SRC

#+RESULTS:
| root |
| john |

Combine source blocks

It's possible to combine text from different source blocks with the :noweb yes header argument. This example shows a ledger that's split up into income and expenses source blocks. They are then combined in the balance source block. Ledger is a plaintext accounting tool and can create a balance report from our assets, income and expenses. The :noweb header arguments in not something that is specific for ledger.

Docs on :noweb header argument can be found here. Docs on ledger can be found here.

#+NAME: income
#+BEGIN_SRC ledger
  2020-06-24 * Sold old computer
      Assets:Bank:Card                         5000.00
      Income:Secondhand                       -5000.00
  2020-06-25 * Salary
      Assets:Bank:Card                        25000.00
      Income:Work:Salary                     -25000.00
#+END_SRC

#+NAME: expenses
#+BEGIN_SRC ledger
  2020-06-23 * Rent
      Expenses:Rent                            3000.00
      Assets:Bank:Card                        -3000.00
  2020-06-24 * Gas
      Expenses:Cas:Gas                         3000.00
      Assets:Bank:Card                        -3000.00
#+END_SRC

#+NAME: balance
#+BEGIN_SRC ledger :noweb yes
  <<income>>
  <<expenses>>
#+END_SRC

#+RESULTS: balance
:                24000  Assets:Bank:Card
:                 6000  Expenses
:                 3000    Cas:Gas
:                 3000    Rent
:               -30000  Income
:                -5000    Secondhand
:               -25000    Work:Salary
: --------------------
:                    0

Combine results and source blocks

Now lets use the result from one source block as argument to another. The example requires some files with a .awk ending to be present in /tmp. We use bash to get data about our file system. That result is then feed to awk that creates a good looking table. That table is then sent to Python for further processing.

There is nothing different here from the other examples except that in this case we use #+NAME: on source blocks instead of a text blocks or a tables. The results from the named source blocks will then be available as input to other source blocks.

List files ending with .awk in /tmp/ dir.

#+NAME: ls-output
#+BEGIN_SRC shell :dir /tmp :results output
  ls -la *.awk
#+END_SRC

#+RESULTS: ls-output
: -rw-r--r-- 1 john users 492 Jun 15 11:46 awk-play.awk
: -rw-r--r-- 1 john users 396 Jun 15 13:10 copy-line-on-confirm.awk
: -rw-r--r-- 1 john users 412 Jun 15 12:25 filter-google-chromepasswords.awk

Take only size and filename and return a table.

#+NAME: ls-output-table
#+BEGIN_SRC awk :results table :stdin ls-output
  BEGIN { OFS="|" }; { print $5, $9}
#+END_SRC

#+RESULTS: ls-output-table
| 492 | awk-play.awk                      |
| 396 | copy-line-on-confirm.awk          |
| 412 | filter-google-chromepasswords.awk |

Process the table with Python.

#+BEGIN_SRC python :results output :var table=ls-output-table
  print(table)
  print(max(table))                              # Biggest awk file
  print(min(table))                              # Smallest awk file
  print(sum([x for x,y in table]) / 1000 , "kb") # Sum of all awk files
#+END_SRC

#+RESULTS:
: [[492, 'awk-play.awk'],
:  [396, 'copy-line-on-confirm.awk'],
:  [412, 'filter-google-chromepasswords.awk']]
: [492, 'awk-play.awk']
: [396, 'copy-line-on-confirm.awk']
: 1.3 kb

Nice things I didn't cover

Org have ways to execute code on remote servers via source blocks. This is awesome and can be used for DevOps. I have used it in cases where an Ansible playbook is overkill. There is a really nice blog post and video about this stuff here.

Source blocks can also be written to the file system via the header switch :tangle. This is also very useful. More info about that can be found here.

Conclusion

I think org is truly an amazing piece of software. With very little effort from the user it can provide a very rich and interactive experience with a lot of applications and languages. From the document view everything is just text so version control a org document is very simple.

Similar articles:

Powered by Hugo & Kiss.