This post has been brought to you by booking.com, one of the Platinum sponsors of the 2018 meta::hack conference.

About the Various PANs

When it comes to CPAN, Perl has a lot of related acronyms, many of which can be hard to understand. Let’s take a moment to discuss some of them now. This discussion will focus mostly on the PANs. Let’s start with the most popular PAN in Perl:

CPAN

The Comprehensive Perl Archive Network. CPAN is a repository (or collection) of files which various authors have made available. There is one canonical CPAN repository and all other CPANs act as mirrors of either the canonical CPAN or of an additional mirror. This means that CPAN is distributed geographically. When you install from CPAN, you are, in many cases, actually installing from one of the CPAN mirrors. Choosing one which is geographically closer to you may result in faster download speeds, however this is not always necessary. For instance, if you choose cpan.metacpan.org as your mirror, Fastly will handle figuring out the fastest way to get the file to you.

PAUSE

PAUSE is the “Perl Author Upload SErver”. This isn’t strictly one of the PANs, but it does belong in a discussion of the various moving parts associated with CPAN. PAUSE is the administrative layer which exists over CPAN. PAUSE controls the processes of assigning author names to users, accepting CPAN uploads and deciding (via 02packages.txt) which files are authorized (ie official) and which are not.

See this discussion of the PAUSE permissions system for more information on how PAUSE regulates uploads. (Note that the m permissions as referenced in this article no longer exists on PAUSE. Any m permissions have now been replaced by f.)

See also this post on viewing module permissions on MetaCPAN.

MetaCPAN

MetaCPAN is a layer on top of CPAN. Initially its sole purpose was to provide a web API which would parse CPAN module metadata and serve up information about the various modules. Hence the meta in MetaCPAN. The MetaCPAN API is the core of MetaCPAN‘s functionality.

Following the publishing of the API, a web front end was created. This search engine, now known simply as metacpan.org, relies exclusively on the MetaCPAN API to serve up its results. So, while most people may think of the search site as MetaCPAN itself, it’s really just one part of the project.

BackPAN

BackPAN is an aggregation of what currently is and what at one point was on CPAN. For various reasons, modules can be deleted from CPAN before they have outlived their usefulness to the general public. If you’re looking for a module which can no longer be found on CPAN, you can try looking on a BackPAN. There are various BackPANs and, unlike CPAN itself, there is no one canonical BackPAN. This leads to a confusing situation where not every BackPAN may have exactly the same files, so this bears keeping in mind.

If you use backpan.metacpan.org or cpan.metacpan.org as the mirror which your CPAN install from, you are actually using a BackPAN, so in general you should be able to find what you need there. backpan.metacpan.org is an alias to cpan.metacpan.org, so just use whichever name you prefer.

MiniCPAN

Sometimes you want a copy of CPAN for when you’re on the go, or just not able to get fast Internet access or for various other reasons. For such occasions, you can create a MiniCPAN, which is essentially a copy of all of the latest modules on CPAN. Since the older versions of modules are not included in a fresh MiniCPAN, it can be much more compact than a full CPAN mirror. I use a MiniCPAN when I’m in airplane mode — literally when I’m on an airplane — because there are often times when I’m working on something while in the air and I need to install a Perl module right now! See CPAN::Mini for more information.

DarkPAN

Many of us have heard of the DarkPAN, but nailing down exactly what it is can be difficult, so I asked on Twitter. Based on my non-scientific poll, most people felt that DarkPAN was a blanket term for Perl code which does not exist in the CPAN. A smaller group of people thought of the DarkPAN as a MiniCPAN which contains Perl code not in the CPAN.

To quote TUX, a DarkPAN is:

the modules and scripts used in the wild (including non-OpenSourced company stuff) that is not publicly available (through CPAN). It is the part of perl that we cannot index, cannot analyze and cannot test when we upgrade perl.

GrayPAN

Along with my Twitter poll, I also solicited thoughts on Reddit, which brought up the GrayPAN. For this part, I’m just going to quote KENTNL who says a GrayPAN

is publicly accessible, but published outside the cpan infrastructure, resulting in a codebase that is factually public, but functionally non existent from the perspective of CPAN, as things can’t really depend on it, and subsequently doesn’t get subjected to multi Arch testing with every release, and critical breakage is likely to go unnoticed until too late.

…

For example: GNU Autotools/Autoconf uses Perl substantially, and is widely used in OpenSource, but due to not being on CPAN, it doesn’t attract being tested as routine against developmental versions of Perl.

This means that when Perl makes a change that breaks Auto*, it won’t be until Perl ships ( or is nearing shipping ) that people will know Autoconf is broken, which will mean there’s little to no scope for avoiding breaking it.

Greypan software is extra useful here, for another reason: It demonstrates that the argument about “stuff on darkpan could break”, is not merely a strawman, but a reality.

NB: Autotools being broken by perl changes actually happened, and late-cycle changes had to be made to Perl to avoid breakage, the “oh, consumers can fix their code” argument just isn’t helpful when the consumer is both so widespread, and important, and yet, potentially have a stagnant release process.

Hence, it’s a darkpan that doesn’t immediately appear “dark”, but functions with most of the downsides of a darkpan

I think that’s a better summary than I could provide.

The Lower Case PANs

After much discussion about CPAN and friends. What about cpan, cpanp, cpanm and even cpm? These all have one thing in common. They are utilities for installing modules from CPAN.

cpan

Lower case cpan is the name of (probably) the oldest CPAN installer client which is still in use. Read the docs.

cpanp

cpanp is an iteration on the functionality which was brought to us by the cpan client. It was initially easier to set up than cpan and had more options.

TUX says:

cpanp is the user interface (CLI) to CPANPLUS, a module that once tried to extend and generalize cpan/CPAN but is now obsolete and removed from CORE.

Read the docs.

cpanm

cpanm is the cpanminus command line CPAN installer. The name is a play on CPANPLUS. Written by MIYAGAWA, this command line interface did away with much of the configuration involved in using both cpan and cpanp. It offered a faster and friendlier way for people to get up and running when they wanted to install CPAN modules. Read the docs.

cpm

cpm is an even faster version of cpanm. It has more dependencies, but it can install modules in parallel, making it potentially much faster at installation than any of the other CPAN installers. Read the docs.

In Conclusion

If you found any of this confusing, you are not alone. To truly understand all of these tools (and even their names) it helps to understand the historical context in which they came to exist. There’s a lot of developer and community effort which has gone into the development of these tools and resources. I’m very grateful for it as these tools save me countless hours of my own time every week. Hopefully this little walk down CPAN‘s memory lane has been helpful to you. In future, if a friend or colleague is confused as to the meaning of any of these terms, you’ll be well placed to bore them to tears with the details.

If you’d like to discuss this post, /r/perl might be a good place for that.

Occasionally I find myself running some random Perl script from a Github gist or dealing with some code from a colleague that doesn’t have proper dependency management (yet). It’s a bit painful to

• run the script
• wait for it to die on a failed dependency
• install the missing dependency
• re-run the script
• wait for it to die
• install the missing dependency
• rinse
• lather
• repeat

Being lazy

I was aware of Acme::Magic::Pony. It already solves this problem, but it uses cpan for the installation. lib::xi also solves this problem, but does this via cpanm. (I was only made aware of lib::xi later). I’ve lately been using App::cpm to install modules, so I wanted a solution that would also use App::cpm. I came up with something which I’ve been using for my own needs. I’m now ready for the general public to tell me how bad of an idea it really is. Drum roll please…

I give you lazy.

Let’s take an example script, called foo.pl:

If you are missing an installed module, your output might look something like:

Let’s try re-invoking this script, but this time with lazy:

perl -Mlazy foo.pl

This will attempt to install any missing dependencies in a place where they are globally available to you. If all goes well you’ll be installing dependencies and running your code all in one invocation of your script. Does it work? Let’s test it.

It did work and it was pretty painless. lazy noticed that a module (Open::This) was missing, so it installed it, loaded it and then allowed the script successfully to run to completion.

(Note that the above script is giving you the arguments which you could pass to an editor, like vim to open up LWP::UserAgent at the line where sub new is defined. I hope to get to Open::This in a subsequent blog post.)

That’s enough information to make you dangerous. At this point you can stop reading and get on with your life. If you’d like to see some more advanced use cases, read on.

Using an arbitrary local::lib

If you’d like to keep the new dependencies in a sandbox, you can do this:

perl -Mlocal::lib=my_sandbox -Mlazy foo.pl

This will install the missing modules to the my_sandbox folder and it will also use this folder to find them. Your install might look something like:

But wait, there’s more!

Using a default local::lib

If you’re too lazy to specify a local directory, but you still want to use a local lib, lazy will use whatever your first default local::lib install location is:

perl -Mlocal::lib -Mlazy foo.pl

In my case it looks like:

But wait, there’s more!

Passing args to cpm

If you want to pass some extra flags to cpm you can do this by passing them to lazy:

perl -Mlazy=-v foo.pl

This switches on cpm‘s verbose reporting. The first part of the output now looks like:

Now, if you also want to remove the colour from this reporting, that’s easy enough:

perl -Mlazy=-v,--no-color foo.pl

Maybe you want to install man pages too:

perl -Mlazy=-v,--no-color,--man-pages foo.pl

Check cpm --help for a more exhaustive list of available options.

Anything you can do via command line switches, you can also do directly in your code.

In Your Editor

This whole thing got me thinking about a vim integration. In my case I’ve added the following to my .vimrc:

nnoremap <leader>l :!perl -Mlazy -c %:p

This essentially means that in my editor, if I type ,l then vim helpfully takes the name of the file I’m currently working on (Foo.pm) and runs perl -Mlazy -c Foo.pm. Since lazy will still be invoked when run via -c we don’t even have to run code to install modules. We can essentially just run a compile check and this sets the module installation chain of events into motion. I can install missing modules without leaving my editor.

Now, this is where the pedants among you speak up and say “but -c does run code in BEGIN and CHECK blocks”. See more discussion at https://stackoverflow.com/a/12908487/406224. Essentially code could be run under -c and it’s helpful to be aware of this, for security reasons.

In Your Linter

As an https://github.com/w0rp/ale + vim user, I’ve been using SKAJI’s syntax-check-perl plugin for https://github.com/w0rp/ale. Since this linter runs your code via -c, adding a use lazy; to your code will install your CPAN modules on demand, every time your linter runs. So, it’s possible for your modules to be installed asynchronously as you happily go about writing your code. Just be sure to add a use lazy; to the code you are writing. This statement must be before any modules which you would like to be auto-installed.

Wrapping Up

You get the idea. I’d be tickled if you decided to give lazy a spin and then headed over to Github to tell me all the things that it doesn’t do correctly. 🙂

fields

Recently at $work we were discussing some of the behaviours of WWW::Mechanize when submitting forms. For instance, when you pass the fields parameter to the submit_form() method, Mechanize might take a very lax approach to submitting your data. Imagine the following form:

Now take the following code:

Mechanize will happily post all of these fields to the form for you, even though the form doesn’t contain an input with the field “C”. If there is no server side validation which checks for unknown fields, you’ll likely get a 2XX status code in your response and all will appear to be well. This can lead to some confusing and hard to debug situations, especially if you’ve done something as subtle as misspelling an input name. You could be banging your head against the wall for quite some time.

with_fields

You can protect yourself from this scenario via with_fields, which will only submit forms which contain all of the provided fields.

If you try to run this code, Mechanize will die with a message like “There is no form with the requested fields at…“. This is already a big improvement. (Note that form_id is optional in this case. If you leave it out then Mechanize will only look for a for which contains all of the fields provided by with_fields. If we provide form_id then Mechanize will want a form which matches the provided id and which also provides all of the required fields).

strict_forms

Can we do any better than this? You bet. We can supply a strict_forms parameter to submit_form. This switches HTML::Form’s strict behaviour on. From HTML::Form’s docs:

Initialize any form objects with the given strict attribute. If the strict is turned on the methods that change values of the form will croak if you try to set illegal values or modify readonly fields.

That’s above and beyond what with_fields brings to the table. Handy stuff. Try to do this wherever possible if you want to make your life easier.

If strict_forms finds a problem, your code will die with something like “No such field ‘C’ at…“.

Notice that the above example uses fields and not with_fields. I would encourage you to use fields with strict_forms whenever possible. The reason is that with_fields will throw an exception before strict_forms is able to perform any validation. So, in many cases you’ll end up with the less helpful error message — the one which doesn’t name the offending field(s).

If for some reason, you need with_fields for your form selection, keep in mind that strict_forms can still find additional issues (like trying to set readonly fields). My advice would be to use strict_forms whenever possible and to use with_fields only in cases where fields + strict_forms is not possible.

In summary, TIMTOWDI, but here’s the order of preference for incantations of submit_form:

• fields + strict_forms
• with_fields + strict_forms
• with_fields

autocheck

autocheck can save you the overhead of checking status codes for success. You may outgrow it as your needs get more sophisticated, but it’s a safe option to start with. Consider the following code:

This code doesn’t die or warn, since it assumes you will “do the right thing” by checking status codes etc. Now try this:

If you run the above code you’ll get something like “Error GETing foobar.comcom: URL must be absolute at…“, which can also save you a lot of heartache.

HTTP::CookieJar::LWP

You are encouraged to install Mozilla::PublicSuffix and use HTTP::CookieJar::LWP as your cookie jar. This allows you to take advantage of HTTP::CookieJar which is more modern than HTTP::Cookies and “adheres as closely as possible to the user-agent rules of RFC 6265“. See also https://metacpan.org/pod/HTTP::Cookies#LIMITATIONS

protocols_allowed

How about restricting the protocols your agent might follow? Let’s look at protocols_allowed: This option is inherited directly from LWP::UserAgent. It allows you to whitelist the protocols you’re willing to allow.

This will prevent you from inadvertently following URLs like file:///etc/passwd

protocols_forbidden

If you don’t want to whitelist your protocols, you can blacklist them instead. Unsurprisingly, this option is called protocols_forbidden: This option is also inherited directly from LWP::UserAgent. It allows you to blacklist the protocols you’re unwilling to allow.

This will prevent you from inadvertently following URLs like file:///etc/passwd

Creating a Stricter Agent

If we put together all of these together we get something like:

Using these settings in conjunction with strict_forms can help you with debugging, security and also your own sanity.

Making it Official

Much of what has been discussed here is now documented at https://metacpan.org/pod/WWW::Mechanize#BEST-PRACTICES. Edit: It has been pointed out to me that readers of this article may also benefit from my previous UserAgent Debugging Made Easy post.

See Also

If WWW::Mechanize does not fit your use case, see also WWW::Mechanize::Chrome, WWW::Mechanize::Firefox, WWW::Mechanize::Cached, LWPx::UserAgent::Cached and Mojo::UserAgent.

In my previous post, I talked about using Ale with vim for linting and syntax checking. Since that time, the Ale defaults for Perl have changed. Perl::Critic checks are still on by default, but you will need to enable the syntax and compile checks that are run via the perl binary itself.

The reasoning for the new default is described in the Ale docs. If you want to (re-)enable your Perl checks, you can follow the example in my dot files.

As a vim user, I’ve used Syntastic for a long time. It’s a great tool for syntax checking. However, I was recently introduced to Ale. Ale does a lot of what Syntastic does, but it does it asynchronously. The practical benefits are

• You should experience less lag when editing large files
• Ale flags problematic lines containing errors and warnings in a gutter, making it easy to find problems
• Detailed information about errors and warnings appear at the bottom of your buffer

I may actually be underselling it. Ale is almost a drop-in replacement for Syntastic. (At least it was for me). Try it out. I don’t think you’ll go back to Syntastic once you’ve tried Ale.

Ale Configuration

let g:ale_perl_perl_options = '-c -Mwarnings -Ilib -It/lib'

This is what I’m currently using. (Note the Ale defaults to -c -Mwarnings -Ilib). Often I’m working with a t/lib directory. Having this included by default means less chance of my code not compiling when it’s run by Ale. Of course, pass in any flags which you may require here. You’ll likely want to keep the -c since you want to compile the code rather than run it. Keep in mind that code in BEGIN blocks will still be executed under the -c flag, so there can be security implications to opening random Perl files with this setting. Explicitly enabling the warnings pragma at the command line will cover you for cases where you haven’t already enabled warnings in your code. Your needs may vary depending upon your environment, so keep in mind that your vimrc can source other files. You can add something like the following to your vimrc:

source ~/.local_vimrc

Perl::Critic Configuration


let g:ale_perl_perlcritic_showrules = 1


When this is enabled, you’ll be shown which Perl::Critic rules which have been violated by your code. This helps you to a) fix the issue or b) copy/paste the rule class name so that you can whitelist the code in question.  For example, you may be told that you’re violating Perl::Critic::Policy::Modules::ProhibitEvilModules.  If you feel you need to embrace the evil, this makes it easy to add a ## no critic (ProhibitEvilModules) to the code in question.   Trust me, this is much easier than digging around to figure out exactly which policy you’ve violated.

Lastly, the default behaviour of Ale’s Perl::Critic linting is to display all violations as errors. I find this confusing, because if something is not preventing my code from compiling, I do not consider this to be an error. In order to set Perl::Critic violations to be displayed as warnings, just add the following to your .vimrc:

let g:ale_type_map = {
\ ‘perlcritic’: {‘ES’: ‘WS’, ‘E’: ‘W’},
\}

The above is a map, so you can add config options for other Ale linters to this map as well.

Unfortunately, the current incarnation of Ale can’t tell the difference between a Perl error and a warning. I have started a pull request to try to improve this somewhat, but I got stuck on the Docker end of things and haven’t gotten back to it yet. If you’d like to pitch in, I wouldn’t mind the help. 🙂