"Linux Tool Box" - What's this about?

In the modeling of weather and climate, huge amounts of data are created, and need to be processed, reshuffled from time to time, visualized and presented. A main tool, or system, used in many institutes are linux/unix based computers, which might not be familiar to everyone. (For you Linux guys, however, there's probably nothing new here...) The list of tools here is far from comprehensive, and asking different people you most likely get different answers as for what tools to use. These ones are my very personal favorites and frequently used - maybe you'll find them helpful, too!


Cheers, and enjoy!
Verena

Plotting data

  • Gnuplot - this is simple and easy to use program to have a quick look into your data. Files and functions can be plotted, also data can be processed. You have an interactive window, and can zoom and turn your plots to look at them. For more complicated things, scripts can be created and executed. However, to my knowledge gnuplot doesn't support gridded data and binary formats, and also making nice and shiney plots with right sizes of labels and so on might be a little challenging (web). There's also a Windows version of the program.

  • Ncview - nice little program with many buttons to look at above mentioned gridded binary data formats (especially netCDF, web).

  • NCL - my personal program of choice to create nice plots. It can easily handle grib and netCDF files, also ascii files, though the latter take some effort to get the format right. Pretty much everything in the plot styles can be adjusted, data processed, many functions are included. It is especially powerful in visualizing (atmospheric) model data.It has no shiney buttons but relies on scripts, but the syntax is easy to learn - especially since the documentation is GREAT and you find about a gazillion examples on the webpage. There, it is also freely available. 

 

 

 

Scripting

Often you will have to do the same kind of plots or data processing again and again. Scripts will make this process easier and can do things automatically for you once programed. Some things I found useful here are listed here. Again, this list is far from comprehensive! 

  • shell scripts - basically shell scripts are little "programs" you write and which you then can execute in a terminal. You can write any command you might use in the terminal in there, and it will be executed in the order they appear in the script. Additionally, loops and conditions can be used, variables read in and so on.  (...work in progress...)

  • echo - prints out text or variables on the screen, either use `echo "Hi there!"` or for exampe echo $PATH to print out the path you are in. ($HOME for home path, $HOST to get the host you are working on and so on...). Despite for printing information, it is especially useful to deliver text or values to other programs. For example `echo "2*3" | bc -l` will give you "6" as an answer, since the calculator (see below) evaluates the 2*3. Piping (that is the `|`) to awk is always a useful thing to do as well (see even further below! ;-)

  • bc - useful little calculator. bc alone gives you integer values, bc -l also floats. Knows a couple of functions as exponential, sine, cosine and so on. Interactively usable in the shell for quick calculations or for example together with the echo command to calculate variables.

  • awk - let's call it a powerful little "spreadsheet program" usable in your terminal. Used as scripting language you can do very pretty calculations and data processing with it. Also, as a simple example, you can extract columns from files, or do calculations on the columns. Add column one and two from file1 in one line in your terminal would look like this: `awk '{print $1+$2'}' file1`. If you want it in a file2, use `> file2` again. It also works with characters, for example you can extract parts of a file name with it by defining a delimiter and print the respective field. (...work in progress...)

  • cat - combines for example two files. `cat file1 file2 > file12` will give a combined output with the X lines of file1 and the Y lines of file2 in file12. `>`writes the output in a file, if you leave that away you'll see the output in your terminal. Also, you can cat files in your script, start with `cat > yourfile << EOF` and put an `EOF` (no space behind!) in the line at the end of the to be written text. Everything inbetween will be written to the file. This comes in useful if you want to write different variables for loops in your script in an ncl or little fortran program you want to use on the data.

  • paste - combines files, too, but column based. If file1 and file2 have one column each, `paste file1 file2 > file3` will give you file3 with two columns.

  • sed - can be used to replace characters/text in a file.

Most of these tools have a man page or help pages. Try for example "sed --help" for a quick overview about the command, or "man sed" for a longer manual. With the pipe "|" you can use the programs in combination and can do several operations in one line without storing intermediate information in a file or in a variable.