So normal output redirection is handled with just the > or >> operators (depending on whether you want to overwrite or append the target file), like so:

dir >> myfile.txt

The operator outputs the stdout stream by default. But it can be modified to output the stderr stream by just specifying that stream with the operator. Remember, stderr is stream 2. So, you would do like so:

dir 2>> myfile.txt

This would write only the error output to the file.

At first glance, you'd conclude the process would work much the same way: Loop through the individual directories in the large repository, create smaller repositories for each one, then dump and import the contents of each directory into its small repository.

The tricky part is that the Subversion dump command dumps everything in the repository, by revision. In order to pull just a single directory, you must filter a complete dump with the "svndumpfilter" command. This blog post by AllMyBrain.com basically explains how to accomplish this in Linux. I usually have to work on a Windows box on the job, so I wrote up a script to accomplish this in a Windows batch script.

The strategy is the same as the Linux script, though. We're going to use "svnadmin dump" the large repository, then use "svndumpfilter" to filter by just the directory we want, then "svnadmin load" the results into the newly created repository. All of this can be combined into a single statement via piping:

This will make a little more sense when we look at the full script. Let's just put it out there and then go through it.

First, we're setting our paths. "SmallRepoPath" will be the directory holding all of the small repositories we'll be creating. "PathToRepo" and "UNCToRepo" point to the big repository as DOS and UNC paths, respectively. "PathToChkout" points to a Subversion checkout of the large repository.

First, we check out the large repository with the "svn co" command. We do this just so that we can call the "dir /A:D /B" command, which says, "List just the directories in the checkout directory." We use that output to loop through each directory in the large repository.

Then, for each directory in the large repository, we create a corresponding small repository, then do our dump/filter/load combo. Again, we're dumping the contents of the large repository, using "svndumpfilter" to filter by directory, then loading that filtered dump into the new small repository.

Finally, we just do some cleanup by removing our temp files and the checkout directory.

There are a few caveats with this code.

First, it will import all of the large repository's revisions into the smaller repository. There are svndumpfilter arguments to prevent this, such as --drop-empty-revs and --renumber-revs, but I found the Windows Subversion binaries to be problematic with these arguments. The end result is that you have more revision numbers than needed, but only the relevant data is actually imported into the repository, and viewing logs on just the imported directory will still obviously show revision logs related to that directory, so there's really little harm done.

Second, the dump/filter/load action doesn't always work on a directory that has been moved (copied/deleted) from another location within the large repository. What's worse, it won't fail, it just won't load any data into the small repository. To address this, use the --revision argument on the "svnadmin dump" command to do a dump starting at a revision after the move took place. Doing so will give the "svndumpfilter" command something it can work with.

This process is certainly more complicated to explain, but ultimately there's not that much more going on. Hopefully this explanation is helpful to you.

That said, there are advantages to having one single repository. No big deal, that can be done after the fact with code.

Here is some Windows code to combine all the repositories in a directory into a single big repository:

There's really not much happening here; the process is simple. First, we create the new "big" repository with the svnadmin create statement. Second, we loop through the directory, processing each Subversion repository in the directory with a three-step process: (a) Dump the repository with the svnadmin dump statement into a temporary *.dmp file. (b) Explicitly add a new directory in the "big" repository for the current repository we're processing, with the svn mkdir statement. (c) Import the dump into the "big" repository with the svnadmin load statement. Really, the rest of the code is just looping, commenting, or cleanup code.

What have we produced? As you might expect, we now have one big repository that has all of the files and commits that were in all of the smaller repositories. The big repository will maintain its own revision numbering, so the revision numbers in your smaller repositories will not match the big repository's revision numbering, although the original commit dates will be preserved. This can be really handy for searching or similar actions that you might do from a more global perspective.

However, this approach is not without its caveats. During the import process, one entire repository is imported at a time. All of a particular repository's revisions will be "grouped" together in the big repository. As a result, revision numbers in the big repository will change every time you recreate it, if there was any new activity in the repositories it contains. For instance, revision #1050 in the big repository may parallel revision #500 in Repository X, but if a commit was added to a repository that is imported before it and the big repository is recreated, that revision would now be #1051. Additionally, although all history and dates are preserved in the revisions, the big repository has commits that are not in chronological order since the import was processed by repository. This inconsistent date/commit ordering can be confusing to some repository reporting tools and may actually render those tools useless to you when they are reporting by date.

Filtering by revision. Note that my svnadmin dump statement includes the -r argument, which specifies the beginning and ending revisions to dump. By default, I'm using "0:HEAD", which basically means "dump every revision", or "dump from the first revision to the HEAD, or latest, revision". Changing the beginning and ending revisions can be useful, especially when used with dates instead of actual revision numbers. For instance, you could change the value to {2007-01-01}:{2007-12-31} to only dump revisions that were committed in 2007.

Combining all of your smaller repositories into one big repository after the fact isn't a perfect solution, but it can be handy, and it's really easy to do when you have a script like this ready to run.

Let's set this up. First, we have a data file named SomeAccounts.txt:

Josh
Mary
Suzy
Amanda
Trisha
Ben

Then, we have ProcessAccounts.bat, which we want to just loop through the accounts in the text file, tell us what they are, and tell us the first letter of the account name (just to have something to do):

Except when you do this, you encounter a problem: All of the values from the FOR loop are the same! It's as if the for loop ran the proper number of times, but it just ran on the last record over and over again! See below:

My account, Ben, starts with B.
My account, Ben, starts with B.
My account, Ben, starts with B.
My account, Ben, starts with B.
My account, Ben, starts with B.
My account, Ben, starts with B.

What's actually happening is the FOR loop is indeed running over every line, and setting the variables as instructed, but the results of those variables being altered isn't echoed until the FOR loop is complete, so the last value of the variable is what displays. This wouldn't be a problem if you were just using your FOR parameter, in this case %%i, but any variables you set while in the FOR loop, like username, experience this "wait until you're out of the loop" phenomenon.

The fix is simple enough, if you know about it! But I've found the solution to be a bit elusive, which is the whole point of sharing it now.

The key is the setlocal EnableDelayedExpansion command. As explained at ss64.com, making this statement before your FOR loop will enable you to display variables as their value at the moment you're referencing them, or their "intermediate values" while in the middle of the FOR loop. In addition to calling the setlocal command, you then have to reference your variables with the exclamation point (!) rather than percent (%) to indicate that you want to use the intermediate value.

Your script will then look like this:

It will now happily act as desired, outputting these results:

My account, Josh, starts with J.
My account, Mary, starts with M.
My account, Suzy, starts with S.
My account, Amanda, starts with A.
My account, Trisha, starts with T.
My account, Ben, starts with B.

WinZip offers a WinZip Command Line Add-on free of charge--if you already own a copy of WinZip Pro!

You shouldn't have to pay for command-line zip. And you don't have to. Enter Info-ZIP. This workgroup has been maintaining free, portable, high-quality versions of zip and unzip. They have plenty of command-line arguments like you would expect from an open source project.

So, with this project's executables in your system path, you can write up a batch file that is executed as a Windows scheduled task. Maybe something like this:

This will zip the c:data directory. Arguments: -q to do it quietly, -S to include system files, -r to recurse into subdirectories. Finally, use -i to point to a file that indicates the exact files to include, by means of a carriage return delimited list.

You can alternatively use -x to specify only which files should be excluded. Perhaps something like this:

The command-line flags are all optional, of course. This tool is certainly a must-have for the Windows scripter.

The basics. You want to generate either a log file or a file that you will be executing after you're done generating it. You'll likely be using a combination of echo commands as well as some output from a handful of commands. Welcome to the > and >> operators. The first will send the output to a file and erase its existing contents; the second will append the output to the file.

Examples:

# This will send the dir command's output to a fresh file:
dir > myfile.txt

# Here I will create a fresh file with an echo command,
# and append the dir output to that:
echo Here is a directory > myfile.txt
dir >> myfile.txt

Redirection for logging. This is a great way to make log files, which can even have the date and time with a simple improvement over that last example:

echo %date%, %time%: Listing a directory. > myfile.txt
dir >> myfile.txt

Using this technique, you can log not just what your batch scripts are trying to do, but what the results are. For instance, let's say we're generating a batch script that automatically creates accounts and drops them in a group. If I'm doing this in AD, I might have code like this:

# Add user "joe" to the "MyGroup" group.
net group MyGroup joe /add

To log the result of this, I might try:

net group MyGroup joe /add >> AddAccts.log

Better error logging. When this command succeeds, it outputs, "This command completed successfully." It will output this to the log file, and you know the net group command succeeded. However, if there were errors, it will output, "The command completed with one or more errors." However, the standard output doesn't include the actual error messages, even though you can see them outputting to the screen. This is because there is a separate error stream, and by default, the redirection operator only streams the standard output.

This is where the >& operator comes in. This will pass the output from one "stream", or "handle" into another. This means you can take a command's error stream, and pass it into its output stream, so that both standard output and error output will be together.

Everything thus far was in Microsoft's documentation page I referred to earlier. But now how do you use the >& operator? I stumbled across Eric Jarvi's blog post, "Redirecting stderr to stdout on the command line", which has an example in the comments.

A quick preface: The standard output stream has a handle of 1, and the error stream has a handle of 2. That being said, take the above example, and modify it like this so that both stdout and stderr will stream to your file:

net group MyGroup joe /add 1>>AddAccts.log 2>&1

This code is effectively saying: "Perform the net group command, then output the stdout to AddAccts.log, but also redirect the stderr stream into the stdout stream at the same time."

As a result, you have stdout and stderr both in your file. Your log will now give you much more meaningful feedback on any errors your batch scripts might be encountering.

What about combining stdout and stderr when piping? It may not be readily apparent, but you can also accomplish this with the 2>&1 redirection. It's just important where you place it; the redirection must occur before the pipe. So something like this would do the trick:

mycmd 2>&1 | othercmd

This would properly send both stdout and stderr from "mycmd" into the stdin stream for "othercmd".

I find the above operators the most commonly used in my development. I hope you found this information useful.

What about sending just stderr? This is simple. Take a look at Redirecting Errors to a File in Windows Batch Scripts for a quick post on doing that.