._s to deal with temp and scratch files

One of the things which has always appealed to me about Linux or *NIX in general is that pretty much everything is a file. The command line tools make it easy to work with files in almost every conceivable way. That makes creating temporary and scratch files very attractive.

Thus, I create a lot of temp and scratch files. Often it’s just so I don’t have to query a source system, making the temp files act as a local cache. Other times it allows me to compare and look at things side by side. The reasons are pretty much endless.

All those temp files can add up and at some point they need to be cleaned. That can be a challenge if you don’t have a plan. It easily turns into tons of manual work and other times you end up committing things to version control that really don’t belong there.

I had experimented with a few different approaches to help me make cleanup easier. Among the things I tried were:

  • a single directory for all scratch files (i.e. cat > $TEMP/new_scratch_file)
  • sticking to a specific name (i.e. dump out and adding a number suffix)
  • prefixing all files with a date-stamp

None of those approaches worked as well as I wanted them to. For example the directory approach was tidy, but it also made me add a lot more characters every time I created a file. Then, when trying to work with the files, I would have to cd or add the directory path potentially several times to a command. Likewise the specific name quickly became an issue when I wanted to compare a half dozen files and the name lacked any context. Adding more complex names wasn’t going to make cleanup necessarily easier, since there could be legitimate files matching a broad wildcard. The date-stamp idea was interesting, but again the wildcarding to make cleanup easy would have been tedious and error prone. I wanted the cleanup to be as easy as the creation of the files.

Then I stumbled on an approach that let me keep creating temp and scratch files anywhere (including inside a repository), with meaningful names, and the ability to clean up nice and quick.

In a nutshell, I started adding ._s as the suffix or extension to all of my temp and scratch files.

That is a pretty short string that clearly tells me the file is temporary, while allowing me to use a clear and meaningful name for the file such as instance_list._s or prod_tables_before._s.

The cleanup ends up being a simple rm *._s or perhaps a find . -name "*._s" if I littered those little files throughout a directory tree.

Speaking of directories, the suffix also works for those as well, which brings even more flexibility.

A final benefit, due to the simple wildcard, is that you can add that *._s to your .gitignore file and avoid committing those files.

All in all, I’ve been very happy with that approach. I get to keep placing temporary files as needed anywhere I deem fit and I’ve not committed one of those files to version control since I adopted this approach. It’s the cleanup that is the best part. Nothing else seems to be using ._s as as file extension and I can confidently blow those files away.

P.S. I originally shared this idea during my lightning talk at DevOpsDays Rockies. Big shout out to Ryan C. Koch (@ryanckoch on twitter) who encouraged me to write this up.

Leave a Reply

Your email address will not be published. Required fields are marked *