3.x is being worked on!

Since my last post, countless bug fixes, features, etc. have been merged into the project. There’s also a wave of new contributors who I’m immensely grateful for!

Something I’ve been struggling with for the past few months is conflicting priorities to get 3.x out the door, and include all the new feature requests. All this while having very little time to actually work on clap. My paid day job has been monopolizing my time for 2017 in an unfair way, I’ve been doing more traveling this year than in nearly all of 2015 and 2016 combined!

I’ve decided it’s time to get to work, I’m implementing a feature freeze for clap in an effort to get 3.x out the door, at which point the refactoring made in 3.x should also make implementing these feature requests far easier.

What Will Be New?

3.x has few “killer features” that should be headlining, namely Custom Derive, the ability to use enums as argument keys (i.e. getting away from “stringly typed” APIs), and using serde as the serialization framework. There are then several primary feature requests that will be priority number one once 3.x is available.

Some of the 3.x features will be offered as new crates, provided inside the clap repository (similar to how ripgrep offers it’s various crates), other features are being pulled out of mainline clap and moving into external crates (such as the completion generation code).

This should make the projects less monolithic, and easier to contribute to.

Custom Derive

The goal of custom derive will be to provide two or three primary traits that allow one to take an arbitrary struct and derive a clap::App from said struct, or take a clap::ArgMatches struct and convert back into your arbitrary struct. This is similar to what’s currently provided by structopt, which I’m hoping to collaborate with, or move into mainline clap.

The end result should look like this on a nightly compiler:

 

Then running:

A stable compiler should be the same, minus the IntoApp trait, where one would still have to define the:

 

But both cases are leaps and bounds better than current 2.x clap.

Enums as Argument Keys

To go along with the custom derive, I plan on providing another derivable trait to allow using enums as argument keys. So this would be possible (which would also work with subcommands):

 

This has a huge benefit of not compiling typos. For example, currently if I type, matches.is_present("verbse") my code will happily compile, and it won’t be until I reach a runtime path which executes that code that I’ll find my mistake.

The same can be said for subcommands, and adding new subcommands:

 

 

This will fail to compile because I didn’t exhaust the match possibilities and include Push(..) in the match statement.

serde

Using serde should go without saying. All clap structs will derive and implement the appropriate serde traits to allow going to/from any serde supported formats 🙂

The Ability to Mutate Args (Finally!)

3.x should also contain the ability to mutate Arg instances even after they’ve been added to an App instance. This will greatly simplify most CLI definitions as one could use the Arg::from(&str) methods and make small changes to specific args that require additional details. The exposed APIs will be something along the lines of:

  • App::mut_arg
  • App::mut_group
  • App::mut_subcommand

More Lean

The codebase should also be much more lean and easy to follow for new contributors helping everyone get involved. All this new custom derive work will take place in the clap_derive crate, and all completion generation will move to the clap_completions crate. Both of which will be hosted in the mainline clap repository.

I’ve toyed with the idea of moving the macros into a separate clap_macros crate, but at this time I will leave them in mainline.

Much, Much More

There will be quite a few other changes. If you’re curious about what they may be, look at the clap issue tracker for feature requests. Nearly all feature requests which have been tagged will be implemented in the early days of 3.x!

Sequence of Events

I plan on releasing 3.0-alpha.1 in the next few months (time permitting) which will contain all the deprecations, and new features. Most 2.x code should still work. This will provide a time to file bugs, and slowly move to 3.x APIs (which will be listed in the deprecation warnings).

After 30 days of 3.0-alpha.1 and no major regressions (small fixes will be released as 3.0-alpha.x and will not reset the 30 day window unless a major regression is found near the end of the 30 days), I will release 3.0-beta.1 which will remove all 2.x deprecations. At this point, 2.x code will break. This will give a chance to file bug reports and make any last minute changes.

After 30 days of 3.0-beta.x and no major changes, I will release 3.0-rc.1. By this point all major bugs should have already been fixed.

After 30 days of 3.0-rc.x with no major regressions, I will put out 3.0.0!

It’s a long cycle, but there are many, many changes. Hopefully everyone will enjoy the ride!

Kevin

Using Shell Completions

It’s been quite a while since I wrote anything blog-ish about clap, or anything else for that matter. I’ve had quite a few ideas that I’ve wanted to talk about, sadly those will have to wait a while longer.

For now, I’d like to announce the new site https://clap.rs and obviously this associated blog https://blog.clap.rs

My goal is to write more about the neat things that are happening in clap and keep people in the loop on ideas or new features.

I have also started a re-make of the old video tutorials; the new ones can be found on youtube. This series aims to go far more in depth into each feature and how to use them. Currently there are only a handful of topics covered, but as I get time I will continue down the train.

Finally, something I’ve really wanted to complete is the “clap Book” which is essentially the video tutorials in written format using mdBook. This should be an easier and more informative read than simply browsing the API docs.

Of course, this is all competing with my limited time to actually work on clap.

Bottom line, I have big ideas and big plans – I’m hopeful that you’ll see more from me over the coming months!

As always, thanks to everyone who has contributed to clap, to include issuses, comments, suggestions, and questions!
In this post I talk about the new ZSH completion script generation in clap. If you’re only here for the snacks, jump on down to the good bits!

Some Back Story

My personal schedule has been absolutely insane over the past few months. I felt like I was falling behind on some of my hobby projects. Not only do I enjoy working on these projects, as they’re a form of mental stress relief (…mostly), but with projects like clap where there are actual users, I also feel a need to provide support for something that many people are counting on.

Over the past weekend, I’ve finally had some excellent, albeit short lived free time. Even though there’s a long list of things I’d love to accomplish with clap, one of the most impactful (after catching up on some bug reports/fixes of course) is completion script generation.

Completion Scripts

As you may, or may not, know shell completion scripts are perhaps the best thing since sliced bread. Even though I adore command line applications, I can very rarely remember which switches to turn on, what subcommands exist, etc. Seriously, I feel like a 95 year old man at times…

The wonderful thing about completion scripts is I don’t have to remember. And with ZSH at least, I get a helpful hint at what I’m actually doing. You’ll see what I mean shortly! Suffice it to say that when you’re running a command, or trying to remember what to type to invoke the command, you simply double-tap your <tab> key, and see a list of valid possibilities. In most shells this also a context sensitive list (more on that in a bit). You may see something like:

 

If you’re in the middle typing a particular command, hitting the <tab> key once will finish it for you, or print any possibilities if there are ambiguities

 

The problem with these completion scripts is that you have to write them. And they can be tedious, redundant, and arcane. All the worst things, rolled into one heated mess. For some relativity, the BASH completion script for rustup is 950 lines with no comments. (⊙_☉)

On top of that, if you ever change your CLI, say add an option, change the name of an old command, etc. you’ll have to re-write these scripts. It can be a nightmare.

This is why I originally added completion script generation to clap. It simply looks at your valid flags, options, commands, etc. and generates a script for you. Yay!

Completion Script Generation in clap

clap has supported generating completion scripts (at least Bash and Fish) for a little while now. But as an avid user of robbyrussell/oh-my-zsh with ZSH as my primary shell, I felt I was missing out on some completing goodness.

That’s why I’m proud to (finally) say that as of v2.16 ZSH support has been added.

Two Ways

So now that we know what completion scripts are, we can start to use them and bask in their all knowing light. Completion scripts can be generated in one of two ways in clap either at compile time, in which case they’ll be output alongside your binary, or at runtime. Actually you can do both…or none. So really four ways.

Let’s create a new project real quick to demo it.

 

Then add clap as a dependency of the project.

 

Next we mock up a very simple CLI. One of the requirements of generating completion scripts is that our CLI code be abstracted out into some kind of function because we’ll need to call it from a few places.

Our little CLI has a positional argument, a flag, an option, a subcommand which also has it’s own option. It might be time we start considering a completion script…

Compile Time

Using the compile time method is nice, if you already have a nice way to distribute these scripts. It’s a little bit more work to set up, but not much. The upside is that it doesn’t require any additional code at runtime.

We’ll do this with a Rust build.rs “Build Script” which allows us to run arbitrary Rust code at compile time without the need for a nightly compiler or procedural macros.

First, we tell our project that we are in fact using a build script:

Then we create our build script. The idea is that we’ll build our CLI, then generate the script and save it to a file.

 

That’s it! Now when you run cargo build you’ll have a new fake.bash-completion file sitting alongside your fake binary.

Now you might be asking why we have to specify the binary name manually? It’s because we aren’t actually running the application, so clap doesn’t have a chance to figure out what the end result binary name actually ended up being.

If you want to generate for more than one shell, it’s as simple as changing to:

 

Runtime

Compile time is all well and good, but that whole build.rs thing is a bit messy. Plus you have to distribute all those scripts to your end users, or make them available, and you don’t know why which shell they’re using, or where they install their completion scripts too!

Alas, there is another way. You can add an option, or subcommand to your CLI which generates the scripts and outputs them to stdout. This allows the user to pick which shell, and then simply redirect stdout to desired file/location of their choosing.

Let’s do that.

Delete the build.rs and remove the build = "build.rs" from your Cargo.toml.

Now all we have to do is add this option our CLI: Since we already have a subcommand, let’s add another called completions. Although it’d be just as simple to create a --completions <SHELL> option, or something similar.

 

Then we add the logic to our src/main.rs

 

And we’re done! It may look slightly different from the compile time code, but the gist is that we’re using the gen_completions_to which allows us to specify an io::Write object (such as stdout), and we’re just turning the &str value of <SHELL> into a clap::Shell enum variant since clap::Shell implements FromStr.

Also note, the unrwap() call here is actually safe, because clap ensures that only valid &str values are used, and that one such value has been used.

Now you could run something like:

Also, if you don’t like that completions subcommand messing up your CLI help message, you can always add the line setting(AppSettings::Hidden) which will make it a secret (⌐■_■)

More Demos!

I’ve recently submitted a PR to rustup which includes the ability to generate these scripts. But for a demo of both the Bash and Zsh shell completion I give you..

ZSH


or GIFV

Bash


or GIFV

Closing Comments

Some technical notes not covered in the demos, but the ZSH implementation supports things like:

  • Automatic conflict resolution (i.e. if --a conflicts with --b, and you’ve already typed --a then your completion list will not display --b)
  • Automatic possible value lists (i.e. in the above example, things like only being able to type zsh, bash, or fish)
  • Switch stacking (i.e. allowing you do things like -a -b -c becoming -abc)

Happy completing!

2.x is out

I’m [happy|relieved|excited] to announce the release of clap v2.0.0! I spent a good bit of time debating a major version bump. Now that it’s released I’d be hard pressed to tell you exactly what I was afraid of. This release, in my opinion, is a dramatic improvement, and feels like a valiant contribution to the Rust community…or at least the portion that uses clap.

If you’re only here for the free snacks and new features, jump on down to “v2 – Phew!”. Otherwise, walk with me, lets converse about this journey.

Prologue

If you haven’t heard of clap before – and I mean Command Line Argument Parser (Not anything else which unfortunately shares the same name. But hey, it’s easy to remember and naming things is like 95% of the battle, right?) – it’s exactly what the name implies an argument parser. But it started as a learning platform for me to try and wrap my head around this new hotness I’d heard about, called Rust.

The project has since grown from purely a learning platform into a very usable library for getting Real Work® done. In fact, I’m now only one of 27 contributors as of this writing (if you count @homu and @yo-bot… /r/botsrights). It’s been amazing to watch PRs come in from around the globe. This is something I truly love about the Rust community.

I’m not a programmer by day, but I do use programming to make my paid job easier, or at least automated (note, the two are not always synonymous). It also helps me feel less guilty about taking extended coffee breaks.

I do use programming to make my paid job easier, or at least automated (note, the two are not always synonymous).

Either way, command line utilities in general are of a particular use to me, and I find them fascinating. Granted, the fact remains that I can almost never remember the proper incantations to invoke their magic without a hefty amount of Googleage. Perhaps this is why I’m more comfortable writing my own than memorizing others. Add to this, there is a decent amount of horribly designed interfaces…yes, I’m looking at you git.

I kid, git ♥

I’ve learned so much by working on clap; about Rust, about myself, about OSS in general, and a good bit about parsing arguments.

v0 – Be Gentle

The initial offerings of clap were embarrassing to say the least. I was am still figuring out how to properly speak crab. This coupled with coming from a non-systems programming background meant it was a slightly uphill battle.

I struggled with something I believe many do in the OSS community…is my code good enough?

…is my code good enough?

Putting your best effort out in the open, in black and white, for all the world to see and judge can be an intimidating process. At times it’s difficult to separate criticisms of code, from criticisms of ones self – because coding can be such a personal thing that the two are often intermixed.

I can say, without a doubt; the Rust community is one of the friendliest and most professional communities I’ve ever been a part of.

the Rust community is one of the friendliest and most professional communities I’ve ever been a part of.

If there is anyone out there still holding on to a private code base for fear of critique, you needn’t worry; the water really is warm. I say thanks to all those who work so hard to ensure this is the case!

Even though clap started as a learning project, enough people gave it a shot or offered friendly advice and PRs that I began trying to make it into a releasable product. To all those who jumped on board from the beginning, I tip my top hat to you fine ladies and gentlemen.

In my opinion, for a library to be of real use, it needs to form a contract with it’s consumers regarding breakage. For clap this meant stabilizing the API and releasing a 1.0 with strong backwards compatibility guarantees. This is very important to me.

v1 – Share All the Things!

Another game changing aspect of Rust is the package management scenario. In fact, without something as amazing as cargo and crates.io, projects like clap wouldn’t get anywhere. To use clap in your own applications it’s literally adding a single line to your Cargo.toml (or better yet, installing the excellent cargo-edit command, and running a single $ cargo add clap).

install the excellent cargo-edit command, and run a single $ cargo add clap

Because this process is so simple, clap got a fair amount of use. Like any project, the more I and others used it, the more we began to see where certain decisions in v1 needed to be tweaked, namely:

  • The inability to handle invalid UTF-8
  •  Too many lifetimes!
  • Vecs everywhere!
  • The inability to support external subcommands (i.e. similar to cargo)
  • The public API began to grow unwieldy with settings
  • The from_usage parser had some unintuitive design flaws

This meant a v2 had to be cut. My motto, and the way I intend to keep running clap is to iterate as much as required to build the best library I can, all while maintaining a breaking changes contract with my consumers. If that means clap reaches v6 next year (very unlikely), so be it.

 v2 – Phew!

This new version of clap marks a huge step forward in the project. Let me point out some of the highlights by addressing the pain points above

Invalid UTF-8

Invalid UTF-8 is perfectly legal on POSIX compliant OSs where many of these little CLI utilities see use. clap’s official stance is argument names, flags, and help must be valid UTF-8 encodable Rust strings. The values provided at runtime to those arguments (such as file names, etc.) do not need to be valid UTF-8 encoded strings.

External Subcommands

If one wanted to create an application similar to cargo where it’s possible to take a portion of the runtime arguments and pass them to a child process process, it’s now possible and just as easy as dealing with internal subcommands thanks to Rust’s pattern matching. See the AppSettings::AllowExternalSubcommands setting in the documentation for details.

Lifetimes

While you didn’t always see the lifetimes in clap unless browsing the documentation or you needed to do some funky return type handling, when you DID see the lifetimes your eyes would bleed. The signature of the  App  struct was something like  App<'a, 'b, 'c, 'd, 'e, 'f>  Which has now been paired down to  App<'a, 'b>  without losing any functionality. The same applies to other structs as well, like Arg

 Iterators, Yay!

This one is a two’fer!

When building a CLI with clap it’s often common to want to specify physically grouped arguments all passed to your application at once, which previously required allocating an  Vec<Arg>  to pass in…no longer says I.

You can pass inline slices

 

The second part is when getting multiple values out of an argument. Before this meant creating a Vec&lt;&amp;str&gt; but has since changed to returning a Values struct which implements Iterator and lets you decide which end container to use. This prevents things like,

 

 So Many Settings

The public API began to grow unwieldy with “settings” options. I.e. one has the ability to tweak the behavior of clap with various true / false toggles. Each of these toggles was originally implemented as separate method of App, due to their relatively small number. This quickly changed as clap grew and new features were requested. The number of “settings” quickly began to pollute the public API. A solution was derived to use a  AppSettings  enum, which also made adding new settings a breeze.

Unfortunately, clap had already built up quite a few “settings” methods which couldn’t be removed due to backwards compatibility. v2 makes cleaning that API possible!

From Usage

The  Arg::from_usage  parser, which allows various argument settings by simply parsing a usage string, had several unintuitive design flaws. Such as prioritizing naming of arguments using the value name instead of the flag. I.e to access an argument that was created using  -c, --config=<FILE>you had to access, "FILE" …but what if you had more than one argument which accepted a file?! Plus, you couldn’t set value names in a usage string, which is a common thing you’d want to do.

To contrast, here is the difference between v1 and v2:

 

The usage parser now does the intuitive thing and uses the short or long to derive the name, using the value name as exactly that, a value name!
Non Breaking Changes

There have been a huge number of non-breaking changes as well. The ones I’m most proud of aren’t even code.

  • The documentation has been vastly improved
  • The regression testing is far better
  • Errors and error messages are much improved
  • Performance is getting even better!
  • A number of smaller yet key features have been implemented.

Most notably these “small yet key features” is the universal short/long forms for option arguments, and the ability to parse delimited values.

By universal short/long I mean all of these are valid forms for parsing options (assuming the short and long has been defined):

 

Which goes very nicely with delimited values. Spaces as a delimiter has always been around and will continue to be. But now there is an “additional delimiter” which defaults to comma ( , ). For example, the above examples with three values each using delimited values looks like:

 

And you even have the ability to change the delimiter! So as a silly example, a very quick implementation of something that accepts only valid IPs (assuming each octet could be 0-255):

 

Here’s the output from the above code:

 

And keep in mind, these are all valid permutations with no code changes!

 Wrap IT Up!

Bottom line, I’m excited to see where clap goes next! Check out the repository, or the documentation for all the details and examples.

If you’re looking for a project to test out some Rust, jump on in, I’d be more than happy to assist with anyone getting started!

I can’t say it enough, but thank you to all 27 contributors, as well as those who have contributed in so many other ways!

clap Contributors (In alphabetical order)

  • Alexander Kuvaev (Core Collaborator)
  • Alex Gulyás
  • Benjamin Sago
  • Brad Urani
  • Dabo Ross
  • Georg Brandl
  • Paul Blouët
  • grossws
  • Ivan Dmitrievsky
  • Homu
  • Huon Wilson
  • J/A
  • Jacob Helwig
  • James McGlashan (Core Collaborator)
  • Jimmy Cuadra
  • Kevin K. (Me)
  • Markus Unterwaditzer
  • messense
  • Nelson Chen
  • Ross Nelson
  • Sebastian Thiel
  • Severen Redwood
  • SungRim Huh (Core Collaborator)
  • Tshepang Lekhonkhobe
  • Vincent Prouillet
  • Vlad
  • Yo-Bot

PS.

Sorry, I lied about the snacks…unless you count the cookies being served.