Managing Your Travis CI Config Files with App::CISetup

Posted on by oalders

This post is brought to you by ServerCentral Turing Group in Chicago, a Platinum level sponsor of the meta::hack conference in 2018. For the past 3 years we've had the privilege of having meta::hack at the ServerCentral office space in Chicago.

ServerCentral Turing Group (SCTG) is a trusted provider of AWS consulting, managed clouds, cloud-native software development, and global data center services for startups, enterprises, and Fortune 500 companies.

Managing Your Travis CI Config Files with App::CISetup

If you write software, you (hopefully) write tests. If you write tests, it's easy to forget to run them. Using Continuous Integration (CI) for your software is one way around this. Many of us use Travis CI for CI. In this post we'll go over ways to make your Travis CI faster and hopefully better.

Setting Up a New Travis CI Config

Travis builds rely on a .travis.yml file which lives in the top level of your repository. There are several ways to set up a new config.

  • You can do it by hand
  • You can have Dist::Zilla create the file for you when you create a new distribution
  • You can use App::CISetup

I'm going to cover the last option here, since it's so easy and possibly the least known of the options mentioned above.

First, install the module: cpanm App::CISetup.

Then create a new config file: setup-travis-yml.pl --create --dir .

This will create a file which looks something like this:

Let's break down some parts of this file to see what we can learn.

addons

What's going on here? The issue is that some Pod spelling tests will skip running tests if they don't find the proper dictionary files. If your tests are not running in verbose mode, you may not even know that tests are being skipped. So, this part of the setup ensures that there is an English dictionary available for spell checking. Obviously you can add other languages if your Pod needs this.

cache

In many test builds, installing your CPAN dependencies can add a lot of extra time to the total build time. This tells Travis to cache your module installs. Note that each Perl version which you test as a part of your Travis build does its own module install. So, if you're running 6 different jobs as part of a Travis build, you'll actually have 6 different caches. You don't have to worry about a cache from Perl 5.26 contaminating a cache for Perl 5.24.

matrix

Now we're at the build matrix:

We are going to allow some failures:

In the case of blead Perl we do care if something breaks, but we probably don't want to prevent any code from being merged because of a test failure from blead.

If you're wondering what blead is, it's explained at dev.perl.org/perl5/source.html

The master branch, where the development takes places, is named blead

fast_finish

Enabling fast_finish allows us to fail faster. The Travis blog explains it nicely:

With fast finishing enabled, Travis CI will mark your build as finished as soon as one of two conditions are met: The only remaining jobs are allowed to fail, or a job has already failed. In these cases, the status of the build can already be determined, so there’s no need to wait around until the other jobs finish.

Basically, enabling fast_finish allows us to get faster notifications from Travis, if we have builds in the allow_failures section.

include

This brings us to the build matrix includes:

In this case we're saying that we want to run an additional build. We want to run an extra Perl 5.28 build that uses Devel::Cover.

If we are going to run a coverage build, we'll likely also want to use coveralls.io to track changes to our coverage. Setting up Coveralls is outside the scope of this post, but if you haven't started using it yet, I highly recommend taking a few minutes to get set up with this service.

env

Moving along, we can see that some global environment variables are being set.

The above configuration ensures that author and release tests are run on all jobs in your build unless you override or unset these variables. Personally, I prefer to run release tests on just one job per build, but that's just me. I may eventually change my mind about that.

before_install

The last configuration looks simple, but it does a lot:

The above ensures that your build uses the Travis Perl Helpers, which does so much handy stuff I'll leave exploration of that as an exercise for the reader. I should note, however, that --always-upgrade-modules is enabled here, so even though we are caching our CPAN modules, Travis will automatically refresh our Perl module cache if it has gotten stale.

__app_cisetup__

After that APP::CISetup tracks some of our choices as comments, so that Travis does not know or care about them:

Now, what's missing from our Travis config?

sudo: false

Remember when you added sudo: false to all of your Travis builds so that they would build faster? That's no longer a thing, so App::CISetup helpfully removes this for you.

Updating a .travis.yml file

In future you can run setup-travis-yml.pl --dir . as many times as you'd like. (Note that this is not exactly the same command that we used earlier, since we have dropped the --create.) This will increment your Perl versions as new versions become available. This will also remove sudo: false from any config files which currently contain this option. Try running this on your existing .travis.yml files to see what happens.

Doing it Manually

All of the above can also be achieved by making these configurations by hand. Using App::CISetup is not for everyone. You may, for example, not want your YAML to be rewritten by a script which is outside of your direct control. Maybe you want things ordered in a certain way. Maybe you don't want to test on the latest version(s) of Perl. However you decide to manage your config files, I hope you've found some useful information here.

Caveats

At some point after I began writing this post and before I finished Travis CI was acquired. I won't get into the specifics as I'm not close to the situation but as with any change of ownership, but it's something to be aware of.

Happy testing!

How I Spent My Perl Toolchain Summit v2019

Posted on by oalders

PTS

The Perl Toolchain Summit (PTS) is an annual event, held in Europe, where work on improving the Perl toolchain takes place.

I was fortunate to be able to attend PTS again this year. I'd like to thank my employer, MaxMind, for sending me to PTS and for once again financially sponsoring this event. PTS has become something which I really look forward to. It's a block of time to work, undistracted, on MetaCPAN. It's also a chance to see some friends whom I only see once or twice per year. Having the MetaCPAN core developers in one time zone and in one room allows us to enjoy an accelerated feedback loop and get things done much, much faster than might otherwise have been the case.

This year we gathered in the UK, outside of London, to work out of Bisham Abbey, while staying in the adjacent National Sports Centre, which is part of the same complex.

The Sponsors

This event simply wouldn't take place without the financial backing of our wonderful sponsors. I'd like to thank the following donors for making this event a reality:

The Attendees

These past few years, as the MetaCPAN contingent has grown, we've set up our own table at PTS. This year we had Leo Lapworth, Mickey Nasriachi, Graham Knop and Shawn Sorichetti and me. Doug Bell sat with us as well, while hacking on CPAN Testers. Sadly, Joel Berger was not able to attend this year or we would have forced him to sit with us as well. Shawn Sorichetti was a new addition to our PTS table. He had joined us at meta::hack and helped us get a lot done there, so it was great to have him hacking with us again. He has a lot of experience with Docker and Ansible, which we took advantage of.

The MetaCPAN table. Photo by Wendy: https://www.flickr.com/photos/wendyga/

Left to right: Mickey Nasriachi, Olaf Alders, Leo Lapworth and Shawn Sorichetti. Missing from this photo is Graham Knop. Photo by Wendy: https://www.flickr.com/photos/wendyga/

The Goals

This year we took the first couple of hours of the hackathon to gather together as a group and hammer out a plan for the 4 day event. We decided to address some pain points in our stack. Firstly, our Vagrant environment, although it works, is quite unwieldy and generally not enjoyable to work with. Secondly, our Puppet deployments are a bit confusing. Puppet has been a great tool up to this point and a lot of time has gone into its configuration, but we're ready to move on to the next tool.

The result of our conversation was a commitment to move to Docker for both local work environments and also for live deployments. As part of the shift to Docker we decided to start moving our automation to Ansible. We're all quite aware that neither Docker nor Ansible are going to solve all of our issues, but we felt that on the whole it would be a net positive for all of us and for the project.

Dockerizing the first site

We chose to start our Dockerization with Github meets CPAN. This was a good candidate because it's not a mission critical web service and it has enough moving parts (Mojo + MongoDB + an hourly cron job) that we'd get an idea of what might be involved in porting the other services. This site is now deployed and in production using Docker. This work included:

  • setting up a Dockerfile for the site
  • setting up a docker-compose.yml for the site
  • setting up a MetaCPAN organization on DockerHub
  • creating a base Docker image with a pre-installed Perl and some common Perl modules that multiple sites will be able to use
  • configuring Travis to upload new images to Dockerhub after successful merges to master

Various other things happened as part of this, like setting up Ansible, setting up keybase.io for shared secrets and making a plan for which production server(s) to convert to serving Docker containers.

Dockerizing more sites

As part of the above process, we spent some time on expanding the Docker configuration of metacpan-web and metacpan-api. These had previously been set up by some helpful folks outside of the core team, but internally we had not been using them in anger. This gave us a chance to see how they really perform and to tweak them for our needs. I was able to hack on metacpan-web and metacpan-api using Docker during PTS and I'm quite happy with the result.

Making new uploads discoverable faster

Moving on from Docker, I was also able to speed up the amount of time it takes to view new modules on MetaCPAN. When you upload a new release to CPAN there's a lag time before it shows up as the latest, indexed module. This is due to several things.

  • We need to rsync the module from PAUSE
  • We need to rsync an updated 02packages file from PAUSE
  • In some cases, we need to rsync an updated 06perms file from PAUSE

The module sync is quite fast, but 02packages and 06perms are a bit trickier, since they are created/updated by cron scripts on the PAUSE servers. For example, as I found out at PTS, the 06packages file is updated every 12 minutes, with the exception of a 24 minute gap once per hour. Because of the timing of your module being set as latest relies on all of the various files syncing first, we've traditionally just run a cron hourly to set the latest flag. Now that we're using a Minion queue, we're able to set up specific queue jobs to run at set intervals after each module has been uploaded. With this new change, if the stars align, you may find your module is discoverable within just a few minutes. In other cases, you will still need to wait longer, but hopefully not as long as it has taken in the past. In the worst case you'll have to wait just as long as you previously did, but hopefully this is rarely true. This code was merged at PTS. Hopefully you're already noticing a difference.

Deleting unused code

Mickey and Graham spent some time auditing the codebase and they identified some code (mostly utility scripts) which people weren't generally aware of. Since none of us were using these scripts, they will be deleted.

Reworking the indexer

Mickey spent a lot of time getting into the guts of the indexer. This code is non-trivial and quite difficult to refactor. He's working to simplify how it works and was able to start breaking this into smaller chunks for deployment. The change was too invasive to deploy at once, but this is a very important step for us to be able to upgrade to a newer version of Elasticsearch at some point.

Fixing PAUSE account connection

For a good while there has been a problem with our previous mail sender in getting messages delivered to @cpan.org email addresses. This is how we verify PAUSE authors on metacpan.org. FastMail solved this by becoming an email sponsor for us. (I should add that FastMail is a PTS sponsor as well.)

Shortly before PTS we were able to merge the code to switch mail delivery to FastMail. I'm happy to report that authors are now once again able to connect their PAUSE accounts to their MetaCPAN profiles. There are still some related issues to be worked out, but it's better than it once was.

Ripping out the guts of our OAuth

An issue that has plagued us for years is that some users have had issues linking services (like Twitter and GitHub) to their accounts on metacpan.org. The issues have been really hard to reproduce and we just haven't solved them. Since I finally had a block of uninterrupted time, I took a deep dive into our OAuth setup and I'm now in the process of simplifying how this works. The first step is that I'm converting our OAuth flow to use Mojolicious::Plugin::Web::Auth rather than the custom code we currently have in place for this. This allows us to handle the same OAuth flows but with much less code. I had looked into setting up Auth0 for this, but our problems are with our internal flows rather than with the OAuth flows themselves. The internal issues would need to be, fixed regardless. It was easier for me to swap out our current code with a tiny bit of Mojo configuration than to look into setting up a third party integration and potentially having to conform to their requirements etc. I'm not ruling out a future third party integration, but this is a logical first step in fixing this issue.

As far as deployment goes, we're going to deploy these new Mojo OAuth flows while leaving the existing flows in place, since they can co-exist and are agnostic about each other. Once we're satisfied that the new system is working, we'll flip the switch. This seemed like the most disruptive and least risky way of approaching the changes. Some code was prepared for deployment at PTS, but I'll need to keep working at this in order to finish it up.

Goodbye Facebook

A side effect of the OAuth work is we have removed Facebook authentication. It has been broken for many months and nobody has opened a ticket that I'm aware of.

Goodbye OpenID

As we discussed simplifying our authentication options, we decided that we'll be removing OpenID. It doesn't appear to be heavily used and this would reduce the complexity of our code.

Spoiler Alert

INGY is doing some really cool stuff with tab completion and CPAN module installation. I believe he'll have it all ready to go for you YAPC::NA, so watch for that.

RJBS introduced me to JMAP, which FastMail is using to make their UI so snappy. It looks really interesting.

Trivia

I learned that cpm means "CPAN Package Manager" in the spirit of "npm" for Node Package Manager.

The Other Stuff

Although I was able to make it to PTS this year, it was a bit harder, since I had a company summit at MaxMind scheduled for the same week. My manager was very supportive of me attending PTS, so I was able to fly from Toronto to Boston on a Monday and then fly from Boston to London the following Wednesday morning. I had to miss two days of company meetings in order to do this, but it meant having more productive time at PTS. I got to bed at 1:15 AM on Wednesday morning. I got up two hours later at 3:15 AM and got ready to head to Logan. I had no delays and I arrived at Heathrow Wednesday evening where Leo Lapworth was waiting for me. We waited for Dan Book to arrive and then headed down to Marlow. We took a wrong turn on the way down which added 30 minutes to our trip. It also meant we'd be arriving at the pub after the kitchen had closed. I texted Neil and he was kind enough to order and pay for our meals so that they were waiting for us when we finally arrived. He's a real diamond geezer. I was able to repay him in the currency of Thai food later in the week.

We left the pub after the bell for last call had been rung. When we arrived at the National Sports Centre we were told there was good news and bad news. The good news was that we had rooms and the bad news was that we had to share our rooms, due to an overbooking. So, Leo and I bunked together for the night. No troubles there. It reminded me of the old days of PTS where having a roommate was part of the arrangement.

A warm breakfast was provided at the National Sports Centre on Thursday morning. At breakfast I was able to connect with many of the attendees whom I had missed on the previous evening. After finishing up there we made our way down to Bisham Abbey in order to get to work. The venue itself is historic and quite well suited to this kind of event. Perhaps even more importantly, the seats were comfortable and the wifi was solid. If you have uncomfortable seats or unreliable wifi, it makes for a very long four days. Luckily this part of things was rock solid, which I was very happy about.

On the Thursday evening, a large group of us went out to a local Thai restaurant in Marlow. Neil met us there, with his long flowing locks, looking like he'd just stepped out of one of those blow dry salons or perhaps a TV advert for a luxurious shampoo and conditioner.

Not an actual photo of Neil Bowers.

One of the more interesting things about this restaurant was the low ceiling beams, which were a concussion waiting to happen. No concussions were witnessed however. This is likely due in no small part to the fact that low beams were well marked with the amount of clearance required to pass safely. After a fantastic meal we headed back to the venue on foot.

Marlow Giggling Squid

The Giggling Squid helpfully notes how much vertical clearance you need to get past various parts of the establishment.

On Friday morning several of us went to the gym for a workout. I used to work in a gym after my first year of university, so I figured I knew my way around the place. However, as I was struggling with one of the machines, Paul Johnson kindly pointed out to me that I was doing it backwards.

After a productive Friday, we had a really unique group meal in the Abbey. We sat in a room which was long enough to fit us all at one table (or a series of tables?) and a three course meal was served to us. It was the only meal where the entire group was together, sitting down at the same table and it was a really nice experience.

Photo by Lee Johnson: https://metacpan.org/author/LEEJO

Photo by Lee Johnson: https://metacpan.org/author/LEEJO

On Saturday morning several of us went outside for a hike. We ran into BINGOS in the lobby. He had been up all night and hadn't slept. We invited him to join us, which he did, since he wasn't sleeping anyway.

After a productive Saturday a group of us went to a really nice Indian restaurant in Marlow where we spotted various other pockets of PTS attendees.

On Sunday morning several of us once again took advantage of the gym. This is the first PTS I've been to where a gym was an option. It was kind of surreal to start the day seeing Mickey on a stair climber or Ingy running on a treadmill or RJBS doing leg presses.

After Sunday lunch, Leo took me and Shawn back to London where we had booked a hotel in Covent Garden. We met simbabque as we were literally heading out the door of Bisham Abbey. We didn't get a chance to chat, but at least now I know what he looks like.

As far as London goes, Shawn and I knew that the London Marathon had taken place earlier that day, but I just kind of assumed that would mostly be a done deal by the time we got into town -- just a bit of cleanup and maybe some stragglers. I had no appreciation of the scale of this event. When we walked down to Buckingham Palace around 5:30 PM there were still hundreds of people running (and sometimes walking) to the finish line. The official announcement was that 9,000 out of 41,000 people had yet to cross the finish line. We saw a man running while dressed as a rotary phone. Sadly, we missed seeing Big Ben.

The city was packed. We eventually made it to a neighbourhood which wasn't as busy and had some fish and chips. We had wanted to tick this particular box back at Marlow, but there was one standout Google Review that I just wasn't able to get my head around.

The London fish and chips were great and I'm happy to report that the restrooms had not been abused in the manner described above. We also stopped at a pub on the way back. When in Rome...

Nice little place. I have no idea what the Canadian flag was there for.

On Monday morning we got up and had some croissants and canelé for breakfast at Paul and then did some quick shopping. We had an early lunch at Wagamama (hat tip to ILMARI for the suggestion) and then I headed to the airport. For some reason I had a 3 hour layover at JFK on my way back to Toronto, but by early Tuesday morning I was home, just over a week after my initial departure, putting my bags away and getting ready to get back to $work in the morning. I was tired but well chuffed and you might even say I was gruntled. Also, I had written the bulk of this blog post on the plane, so I felt a particular sense of accomplishment. It has taken me two weeks to publish the post, but let's not get stuck in the weeds here. It was mostly done before I got home.

PTS was, as always, a great experience. I feel like this year we made particularly good progress. Some years I've gotten a bit frustrated with what I was working on, but this year it was surprisingly free of roadblocks and really felt like time well spent.

I want to thank Neil, Book and Laurent for all of their work in organising this event. I would say it's a thankless job, but I just thanked them, so that would be a lie. Having worked to organize meta::hack, which is quite small, I can only imagine how much work and organizational skills go into running a PTS that went as smoothly as this one. I look forward to future PTS events, hoping that I get invited again!

About the Various PANs

Posted on by oalders

This post has been brought to you by booking.com, one of the Platinum sponsors of the 2018 meta::hack conference.

About the Various PANs

When it comes to CPAN, Perl has a lot of related acronyms, many of which can be hard to understand. Let's take a moment to discuss some of them now. This discussion will focus mostly on the PANs. Let's start with the most popular PAN in Perl:

CPAN

The Comprehensive Perl Archive Network. CPAN is a repository (or collection) of files which various authors have made available. There is one canonical CPAN repository and all other CPANs act as mirrors of either the canonical CPAN or of an additional mirror. This means that CPAN is distributed geographically. When you install from CPAN, you are, in many cases, actually installing from one of the CPAN mirrors. Choosing one which is geographically closer to you may result in faster download speeds, however this is not always necessary. For instance, if you choose cpan.metacpan.org as your mirror, Fastly will handle figuring out the fastest way to get the file to you.

PAUSE

PAUSE is the "Perl Author Upload SErver". This isn't strictly one of the PANs, but it does belong in a discussion of the various moving parts associated with CPAN. PAUSE is the administrative layer which exists over CPAN. PAUSE controls the processes of assigning author names to users, accepting CPAN uploads and deciding (via 02packages.txt) which files are authorized (ie official) and which are not.

See this discussion of the PAUSE permissions system for more information on how PAUSE regulates uploads. (Note that the m permissions as referenced in this article no longer exists on PAUSE. Any m permissions have now been replaced by f.)

See also this post on viewing module permissions on MetaCPAN.

MetaCPAN

MetaCPAN is a layer on top of CPAN. Initially its sole purpose was to provide a web API which would parse CPAN module metadata and serve up information about the various modules. Hence the meta in MetaCPAN. The MetaCPAN API is the core of MetaCPAN's functionality.

Following the publishing of the API, a web front end was created. This search engine, now known simply as metacpan.org, relies exclusively on the MetaCPAN API to serve up its results. So, while most people may think of the search site as MetaCPAN itself, it's really just one part of the project.

BackPAN

BackPAN is an aggregation of what currently is and what at one point was on CPAN. For various reasons, modules can be deleted from CPAN before they have outlived their usefulness to the general public. If you're looking for a module which can no longer be found on CPAN, you can try looking on a BackPAN. There are various BackPANs and, unlike CPAN itself, there is no one canonical BackPAN. This leads to a confusing situation where not every BackPAN may have exactly the same files, so this bears keeping in mind.

If you use backpan.metacpan.org or cpan.metacpan.org as the mirror which your CPAN install from, you are actually using a BackPAN, so in general you should be able to find what you need there. backpan.metacpan.org is an alias to cpan.metacpan.org, so just use whichever name you prefer.

MiniCPAN

Sometimes you want a copy of CPAN for when you're on the go, or just not able to get fast Internet access or for various other reasons. For such occasions, you can create a MiniCPAN, which is essentially a copy of all of the latest modules on CPAN. Since the older versions of modules are not included in a fresh MiniCPAN, it can be much more compact than a full CPAN mirror. I use a MiniCPAN when I'm in airplane mode -- literally when I'm on an airplane -- because there are often times when I'm working on something while in the air and I need to install a Perl module right now! See CPAN::Mini for more information.

DarkPAN

Many of us have heard of the DarkPAN, but nailing down exactly what it is can be difficult, so I asked on Twitter. Based on my non-scientific poll, most people felt that DarkPAN was a blanket term for Perl code which does not exist in the CPAN. A smaller group of people thought of the DarkPAN as a MiniCPAN which contains Perl code not in the CPAN.

To quote TUX, a DarkPAN is:

the modules and scripts used in the wild (including non-OpenSourced company stuff) that is not publicly available (through CPAN). It is the part of perl that we cannot index, cannot analyze and cannot test when we upgrade perl.

GrayPAN

Along with my Twitter poll, I also solicited thoughts on Reddit, which brought up the GrayPAN. For this part, I'm just going to quote KENTNL who says a GrayPAN

is publicly accessible, but published outside the cpan infrastructure, resulting in a codebase that is factually public, but functionally non existent from the perspective of CPAN, as things can't really depend on it, and subsequently doesn't get subjected to multi Arch testing with every release, and critical breakage is likely to go unnoticed until too late.

...

For example: GNU Autotools/Autoconf uses Perl substantially, and is widely used in OpenSource, but due to not being on CPAN, it doesn't attract being tested as routine against developmental versions of Perl.

This means that when Perl makes a change that breaks Auto*, it won't be until Perl ships ( or is nearing shipping ) that people will know Autoconf is broken, which will mean there's little to no scope for avoiding breaking it.

Greypan software is extra useful here, for another reason: It demonstrates that the argument about "stuff on darkpan could break", is not merely a strawman, but a reality.

NB: Autotools being broken by perl changes actually happened, and late-cycle changes had to be made to Perl to avoid breakage, the "oh, consumers can fix their code" argument just isn't helpful when the consumer is both so widespread, and important, and yet, potentially have a stagnant release process.

Hence, it's a darkpan that doesn't immediately appear "dark", but functions with most of the downsides of a darkpan

I think that's a better summary than I could provide.

The Lower Case PANs

After much discussion about CPAN and friends. What about cpan, cpanp, cpanm and even cpm? These all have one thing in common. They are utilities for installing modules from CPAN.

cpan

Lower case cpan is the name of (probably) the oldest CPAN installer client which is still in use. Read the docs.

cpanp

cpanp is an iteration on the functionality which was brought to us by the cpan client. It was initially easier to set up than cpan and had more options.

TUX says:

cpanp is the user interface (CLI) to CPANPLUS, a module that once tried to extend and generalize cpan/CPAN but is now obsolete and removed from CORE.

Read the docs.

cpanm

cpanm is the cpanminus command line CPAN installer. The name is a play on CPANPLUS. Written by MIYAGAWA, this command line interface did away with much of the configuration involved in using both cpan and cpanp. It offered a faster and friendlier way for people to get up and running when they wanted to install CPAN modules. Read the docs.

cpm

cpm is an even faster version of cpanm. It has more dependencies, but it can install modules in parallel, making it potentially much faster at installation than any of the other CPAN installers. Read the docs.

In Conclusion

If you found any of this confusing, you are not alone. To truly understand all of these tools (and even their names) it helps to understand the historical context in which they came to exist. There's a lot of developer and community effort which has gone into the development of these tools and resources. I'm very grateful for it as these tools save me countless hours of my own time every week. Hopefully this little walk down CPAN's memory lane has been helpful to you. In future, if a friend or colleague is confused as to the meaning of any of these terms, you'll be well placed to bore them to tears with the details.

If you'd like to discuss this post, /r/perl might be a good place for that.

meta::hack 3 Wrap Report

Posted on by oalders

As I mentioned in my meta::hack preview post, for the third year running we have had the privilege of being financially sponsored by Booking.com and also working out of the ServerCentral offices in downtown Chicago in order to hack on MetaCPAN. None of this would have been possible without the support of Mark Keating and the Enlightened Perl Organization. Mark has (as usual) worked tirelessly to ensure that sponsor money moves in the right directions so that we are able to fund meta::hack every year. We are extremely grateful for his help.

I'd also like to thank MaxMind (my employer) for supporting me in attending this event.

This year, five of us worked on improving our corner of the Perl ecosystem.

The Attendee List

Getting There

We all had a great time this year. It's always nice to spend time with friends and it's great to see pet projects move ahead. Shawn and I flew in from Toronto on the Wednesday afternoon and had lunch at a German restaurant.

Mickey arrived from Amsterdam a little later in the day. The three of us met up with Doug for drinks and dinner in the evening.

Getting Started

On Thursday morning, Mickey, Shawn and I got up early and got a run in before heading to the office. We ran along the water and through Millenium Park, stopping off at "the bean" to take pictures and have a look around. That was a very nice, if chilly, way to start the day.

Right from day one we had a productive time. We started by discussing what needed to be done vs what we wanted to do and I think we got some combination of these things done. From my personal experience, I think the trick is to have people generally work on the things they find interesting, while also spending some time on the things that just need to get done. I think we found a good balance. Now, rather than give a chronology of the event, I'm going to cover our "greatest hits" for the event.

The Highlights

  • It turns out our CPAN mirrors were not in sync, causing some hard to debug errors. This got fixed.
  • We removed an unused and undocumented endpoint from the api (/search/simple)
  • FastMail came on board as a sponsor. As of this week they are handling our mailboxes. Many thanks to them and to https://metacpan.org/author/RJBS for getting us up and running in a hurry.
  • We are now equipped to delete CPAN authors and releases from MetaCPAN. Previously this was non-trivial, but we can now deal with spammy modules, authors and other issues which require uploads and or data to be removed.
  • The caching on our Travis CI API tests had stopped working, so we fixed this.
  • Our changes endpoint on the API now recognizes more types of Changes files. The means the search site will now more easily be able to help you find Changes files when you need them.
  • There were previously some cases where the API returned a non-unique list of modules in the provides output. This has now been fixed moving forward. If you spot a module which needs to be re-indexed, let us know and we can do this.
  • The search site now includes links to cpancover.com in cases where coverage reports are available. I hope this will bring more visibility to cpancover.com and also be useful to authors and users of CPAN modules. You'll now more easily to be able to find Devel::Cover output for your favourite modules.
  • The /lab/dashboard endpoint of the search site has been broken for a long time. The exception was fixed and regression tests have been added. The content of the page is still mostly broken, but that can be fixed moving forward.
  • A Minion UI has been added for admins, so that we can now view the status of the indexing queue via a web browser
  • More endpoints have been documented. We have done this using OpenAPI. You can view the beginnings of it at fastapi.metacpan.org/static/index.html.
  • We have partially implemented a user search for admins. This will allow us more easily to troubleshoot issues with users logging in or adding PAUSE, github and other identities to their MetaCPAN logins. Also this will soon help us to identify issues with users who are not seeing all of their favourite modules listed.
  • We have partially implemented an admin endpoint for adding CPAN releases for re-indexing. We currently do this at the command line. Having a browser view for this means we can do this without requiring SSH. More importantly, we'll eventually be able to make this available to end users of MetaCPAN so that people can request re-indexing of releases they care about.

That's the bulk of what we got done. I haven't even listed the work that Doug got done on CPANTesters, mostly because I stayed out of his way and I just don't have the details. Doug was a gracious host. In addition to helping make sure that we got fed and got into the office when we needed it, he stayed late with us on those evenings when we just didn't want to leave and was back again early the next morning. We should also thank Joel Berger who also helped to host us during our stay. Working out of the ServerCentral offices for the third year in a row has been really great. We know our way around and we've been treated extremely well. It was a real treat to be allowed back.

Joel was not able to join us onsite on the weekend, but he did join us on screen.

Other Fun Stuff

Aside from getting our work done, we got in a bike ride on Friday morning. It was snowing and quite windy along the water where we rode. I lost most of the feeling in my hands. In future I'll either pack warmer gloves or exercise better judgement. Obviously I can't count on the others to talk me out of bad ideas. 😉

We also got in more runs on Saturday and Sunday morning. So, while we spent a lot of time sitting in the office, putting calories into our bodies, we did get exercise every day of the conference. That really helped me be productive during the day.

Evenings were fun as well. We had drinks on the 96th floor of the Hancock Building, visited the Chicago Institute of Art and Mickey and I went to a Bulls game on Saturday night. There was only one night where everyone else was too tired to go out. I got myself a table for one at the hotel bar. Wipes tears from eyes. (That was actually not a bad experience either.)

In Conclusion

Our Sunday deployment was a bit of a bumpy ride. In retrospect I would have preferred less moving parts to be involved in a single deployment, but it was also interesting to debug the deployment in real time with a few of us troubleshooting via a 60 inch screen.

We left Doug behind in the office, with what was left of his TODO list.

The flight home was uneventful. There were still some deployment issues to sort out on Monday but those eventually got taken care of.

Several of our usual attendees were not able to make it this year and we really missed them. On the whole however, I was quite happy with what we accomplished this time around. Moving forward we'll be able to continue to build on the work done during meta::hack and continue to improve MetaCPAN.

meta::hack is back!

Posted on by oalders

For the third year running, we have the privilege of working out of the ServerCentral offices in downtown Chicago in order to hack on MetaCPAN. This year, five of us will be working on improving our corner of the Perl ecosystem. The physical attendee list is follows:

  • Doug Bell
  • Joel Berger
  • Olaf Alders
  • Mickey Nasriachi
  • Shawn Sorichetti

This, of course, would not be possible without the help of our 2018 sponsors: Booking.com and ServerCentral. You'll notice their logos on the front page of MetaCPAN into 2019 as our way of thanking them for their support.

I'd like briefly to revisit last year's event, which was quite productive and would not have happened without our 2017 sponsors:

In 2017, we made excellent progress on various fronts. Part of this progress came via an onsite visit from Panopta, our monitoring sponsor. They have a Chicago office and were very responsive when I asked them if they'd be willing to send someone down to chat with us about improving our Panopta configuration. So, Shabbir kindly dropped by, giving us in-person advice on how to configure our Panopta monitoring. One of the most helpful tweaks was setting up an IRC integration, so that outages are now broadcast to #metacpan on irc.perl.org.

Here's a photo of Shabbir's visit. (I wasn't able to attend in person last year, but I was able to do so remotely. I'm the one watching from the wall.)

meta::hack v2 in 2017

From left to right:

Doug Bell, Thomas Sibley, Nicolas Rochelemagne, Graham Knop, Leo Lapworth (front), Joel Berger, Brad Lhotsky, Olaf Alders (pictured on monitor), Shabbir Karimi (from Panopta) and Mickey Nasriachi.

I'll post more about our progress over the next few days. We're working on some fixes and something new and shiny too. Wish us luck!

How lazy am I?

Posted on by oalders

Occasionally I find myself running some random Perl script from a Github gist or dealing with some code from a colleague that doesn't have proper dependency management (yet). It's a bit painful to

  • run the script
  • wait for it to die on a failed dependency
  • install the missing dependency
  • re-run the script
  • wait for it to die
  • install the missing dependency
  • rinse
  • lather
  • repeat

Being lazy

I was aware of Acme::Magic::Pony. It already solves this problem, but it uses cpan for the installation. lib::xi also solves this problem, but does this via cpanm. (I was only made aware of lib::xi later). I've lately been using App::cpm to install modules, so I wanted a solution that would also use App::cpm. I came up with something which I've been using for my own needs. I'm now ready for the general public to tell me how bad of an idea it really is. Drum roll please...

I give you lazy.

Let's take an example script, called foo.pl:

If you are missing an installed module, your output might look something like:

Let's try re-invoking this script, but this time with lazy:

perl -Mlazy foo.pl

This will attempt to install any missing dependencies in a place where they are globally available to you. If all goes well you'll be installing dependencies and running your code all in one invocation of your script. Does it work? Let's test it.

It did work and it was pretty painless. lazy noticed that a module (Open::This) was missing, so it installed it, loaded it and then allowed the script successfully to run to completion.

(Note that the above script is giving you the arguments which you could pass to an editor, like vim to open up LWP::UserAgent at the line where sub new is defined. I hope to get to Open::This in a subsequent blog post.)

That's enough information to make you dangerous. At this point you can stop reading and get on with your life. If you'd like to see some more advanced use cases, read on.

Using an arbitrary local::lib

If you'd like to keep the new dependencies in a sandbox, you can do this:

perl -Mlocal::lib=my_sandbox -Mlazy foo.pl

This will install the missing modules to the my_sandbox folder and it will also use this folder to find them. Your install might look something like:

But wait, there's more!

Using a default local::lib

If you're too lazy to specify a local directory, but you still want to use a local lib, lazy will use whatever your first default local::lib install location is:

perl -Mlocal::lib -Mlazy foo.pl

In my case it looks like:

But wait, there's more!

Passing args to cpm

If you want to pass some extra flags to cpm you can do this by passing them to lazy:

perl -Mlazy=-v foo.pl

This switches on cpm's verbose reporting. The first part of the output now looks like:

Now, if you also want to remove the colour from this reporting, that's easy enough:

perl -Mlazy=-v,--no-color foo.pl

Maybe you want to install man pages too:

perl -Mlazy=-v,--no-color,--man-pages foo.pl

Check cpm --help for a more exhaustive list of available options.

Anything you can do via command line switches, you can also do directly in your code.

In Your Editor

This whole thing got me thinking about a vim integration. In my case I've added the following to my .vimrc:

nnoremap <leader>l :!perl -Mlazy -c %:p

This essentially means that in my editor, if I type ,l then vim helpfully takes the name of the file I'm currently working on (Foo.pm) and runs perl -Mlazy -c Foo.pm. Since lazy will still be invoked when run via -c we don't even have to run code to install modules. We can essentially just run a compile check and this sets the module installation chain of events into motion. I can install missing modules without leaving my editor.

Now, this is where the pedants among you speak up and say "but -c does run code in BEGIN and CHECK blocks". See more discussion at https://stackoverflow.com/a/12908487/406224. Essentially code could be run under -c and it's helpful to be aware of this, for security reasons.

In Your Linter

As an https://github.com/w0rp/ale + vim user, I've been using SKAJI's syntax-check-perl plugin for https://github.com/w0rp/ale. Since this linter runs your code via -c, adding a use lazy; to your code will install your CPAN modules on demand, every time your linter runs. So, it's possible for your modules to be installed asynchronously as you happily go about writing your code. Just be sure to add a use lazy; to the code you are writing. This statement must be before any modules which you would like to be auto-installed.

Wrapping Up

You get the idea. I'd be tickled if you decided to give lazy a spin and then headed over to Github to tell me all the things that it doesn't do correctly. 🙂

Perl Toolchain Summit 2018 Wrap-up Report

Posted on by oalders

Perl Toolchain Summit 2018 Wrap-up Report

Getting There

This year I had the pleasure of attending the Perl Toolchain Summit in Oslo, Norway. Because of a conflict in my schedule, I initially didn't think I'd be able to attend. After a lot of mental back and forth, I decided I'd try to make it work. The compromise was that this year I would leave on Sunday morning rather than on Monday. That meant I wasn't able to participate in the last day of the summit, but I'd still have 3 entire days to get things done.

I left Toronto on Tuesday evening and arrived in Frankfurt on Wednesday morning. I got breakfast in Germany and after a two hour layover I was on a short flight to Oslo. From the Oslo airport I took a train into town.

I got into my tiny, tiny room and changed into my running clothes. I spent the next couple of hours running mostly along the water and making lots of stops along the way to take pictures and hang out. That was enough to burn off any excess energy.

In the evening I met up with everyone else at a bar called RØØR. Here it finally became clear to my why Salve had said alcohol would be expensive. (I hadn't really believed him when he originally let us know). I had a great time regardless and it was nice to see a lot of familiar faces and catch up.

Day One

On Thursday morning we had an 8:30 breakfast at the venue and then 9:00 AM standup, where everyone had a chance to introduce themselves and talk about what they intended to work on. Leo and Mickey had a claimed a small room for our group, which meant that for the span of the summit we would all be able to sit around the same table and whiteboard. Some folks dropped by early in the morning to talk about a few items on their wish list. We also made a game plan for the summit and made various work assignments amongst ourselves. These sorts of face to face meetings are what make the summit invaluable. The first morning meeting allowed us to get everyone on the same page and then get to work.

This year we didn't come into the summit with any one grand project to complete. Rather, we spent some time on smaller items, bug fixes, attending to outstanding issues on Github and infrastructure work. Specifically, we're generally working towards having a better disaster recover plan and failover capacity for MetaCPAN. Leo did a lot of work on that, so I'll let him blog about his contributions. For my part, I picked back up on the work that I had started at meta::hack this past November.

At meta::hack I had spent a good chunk of time upgrading our Debian from Wheezy to Stretch. This is an upgrade of two major releases and it turns out to be non-trivial in our case. A large part of the problem is that with our current setup we are on a version of Puppet (3.8.7) which is no longer supported. Since Wheezy has a newer Puppet (4.8.2) by default, I took this as an opportunity to upgrade Puppet. This meant that some of our Puppet config needed updating and that we also need to update our PuppetForge modules (3rd party dependencies). This was complicated enough that I was going to need a lot of help from Leo, so I sat myself beside him in order to make bothering him more convenient for the both of us. It turned out that I bothered him a lot. I shudder to think how long it would have taken me to work through this remotely. I can't say that I enjoyed the work, but it was much more enjoyable to do this face to face and it was a much better use of everyone's time.

Thursday also happened to be Leo's birthday so we went out for drinks and various foods made of pork at a bar called Crowbar. Marcus Ramberg and Batman joined us there and eventually a good chunk of the attendees came along as well.

Day Two

I had gotten the bulk of the upgrade done on Thursday. On Friday I got up early and went for a run. Then, at the venue we continued tying up some loose ends on the Debian/Puppet upgrade and getting Leo involved in testing it out. I should add that part of this upgrade also included an overhaul of how we build our Vagrant boxes. Currently if you want to deploy a Vagrant instance you download a pre-built box which we have uploaded to one of our servers. Moving forward you'll be able to build your own box on demand. It will take slightly longer to get set up, but you'll have the latest and greatest software right away and you won't need to rely on any one of us to upload an updated box for you to download. In the evening we all went out to a restaurant where there were speeches, excellent food and great conversations.

Day Three

On Saturday morning, Mickey and I met up at 6:30 AM for a quick run. At 7:30 AM we met up with Leo and Sawyer and rented some bikes. We cycled along the waterfront by the fortress and the Opera House before making it back just in time for standup. By now we were upgrading Debian and deploying the new Puppet on two of the three machines which Bytemark generously donates to us. This turned out to be non-trivial as the first machine we upgraded would not reboot. After some excellent technical support from Bytemark we were able to get past this as well.

I also added a URL mapping for static files to the MetaCPAN API. This is a first step in self-hosting some API documentation, likely using Swagger in some way. This is partly what's preventing the MacOS Dash application from having more Perl documentation available.

In the evening most of us hung out at a local hacker/maker space. For me, it was a nice way to wrap up the summit.

In General

While all of this was going on, we also had a good debate about sponsorship of our project. We've been getting a lot of interest from potential sponsors over the last year or so and we've mostly been approaching it on a case by case basis. Having everyone in on a group discussion allowed us to set a sponsor policy moving forward that we think will be good for both us and our sponsors. (If your company would like to sponsor MetaCPAN or our next meta::hack conference, please don't hesitate to get in touch with me).

I also fit in the usual code reviews over the 3 days and also tried to catch up on some outstanding Github issues. It turns out I had missed some entirely. There were issues going back as far as January which had not yet been responded to. 🙁

As part of some of the other work going on, there was progress made towards a tighter integration of the CPAN river data in the MetaCPAN API. Also, we now have access to Neil Bowers' CPAN river data generator. There's a plan to have MetaCPAN directly generate this data, rather than having us pull the data from him.

There was also a fair bit of discussion as to where to host the 2018 meta::hack. Our TODO list is long enough that the Perl Toolchain Summit isn't quite enough for us to make the progress that we think we need to make. So, hopefully, we'll be able to get together once more this year to continue to make progress.

Additionally Babs Veloso showed us a proposal for a redesign of the MetaCPAN author pages. It looks great. Now we just need to find someone to implement it. We also spent a fair amount of time on a part of the project which I can't actually tell you about just yet, mostly because the details haven't all been finalized yet. If all goes as expected, we'll announce that in the coming months.

Getting Home

My flight back early on Sunday was uneventful. I spent an hour in Copenhagen and then was back home on Sunday afternoon in time to take the kids to the park. I got into an Eastern Standard Time sleep schedule by Sunday night. I dropped the kids off at school on Monday morning and then flew off to Boston for meetings until the following Friday. Since I telecommute, I rarely need to travel (or even wear shoes), but for some reason I ended up with back to back trips this year. In the end, it all worked out.

Thanks

On the whole this year's PTS was very productive, fun and very well organized. I'd like to thank Salve J. Nilson, Neil Bowers, Stig Palmquist, Philippe Bruhat, Laurent Boivin and anyone else I may have missed. The venue, food, level of organization (and hoodies) were all excellent. It was a smooth experience from start to finish. I feel like I was able to get a lot of good work done and I am particularly pleased that I was able to finish what I had set out to do. Collectively, there was much more work completed on MetaCPAN than what I've described.

I would not have been able to make this trip without the support of MaxMind (my employer). MaxMind has sent me to PTS without hesitation each time that I've been invited and has also been a sponsor of the event for these past two years. Thanks, MaxMind!

Additionally, I'd like to thank all of our sponsors, without whom this would not have been possible:

MaxMind,
NUUG Foundation,
Teknologihuset,
Booking.com,
cPanel,
FastMail,
Elastic,
ZipRecruiter,
MongoDB,
SureVoIP,
Campus Explorer,
Bytemark,
Infinity Interactive,
OpusVL,
Eligo,
Perl Services,
Oetiker+Partner.

WWW::Mechanize Best Practices

Posted on by oalders

fields

Recently at $work we were discussing some of the behaviours of WWW::Mechanize when submitting forms. For instance, when you pass the fields parameter to the submit_form() method, Mechanize might take a very lax approach to submitting your data. Imagine the following form:

Now take the following code:

Mechanize will happily post all of these fields to the form for you, even though the form doesn't contain an input with the field "C". If there is no server side validation which checks for unknown fields, you'll likely get a 2XX status code in your response and all will appear to be well. This can lead to some confusing and hard to debug situations, especially if you've done something as subtle as misspelling an input name. You could be banging your head against the wall for quite some time.

with_fields

You can protect yourself from this scenario via with_fields, which will only submit forms which contain all of the provided fields.

If you try to run this code, Mechanize will die with a message like "There is no form with the requested fields at...". This is already a big improvement. (Note that form_id is optional in this case. If you leave it out then Mechanize will only look for a for which contains all of the fields provided by with_fields. If we provide form_id then Mechanize will want a form which matches the provided id and which also provides all of the required fields).

strict_forms

Can we do any better than this? You bet. We can supply a strict_forms parameter to submit_form. This switches HTML::Form's strict behaviour on. From HTML::Form's docs:

Initialize any form objects with the given strict attribute. If the strict is turned on the methods that change values of the form will croak if you try to set illegal values or modify readonly fields.

That's above and beyond what with_fields brings to the table. Handy stuff. Try to do this wherever possible if you want to make your life easier.

If strict_forms finds a problem, your code will die with something like "No such field 'C' at...".

Notice that the above example uses fields and not with_fields. I would encourage you to use fields with strict_forms whenever possible. The reason is that with_fields will throw an exception before strict_forms is able to perform any validation. So, in many cases you'll end up with the less helpful error message -- the one which doesn't name the offending field(s).

If for some reason, you need with_fields for your form selection, keep in mind that strict_forms can still find additional issues (like trying to set readonly fields). My advice would be to use strict_forms whenever possible and to use with_fields only in cases where fields + strict_forms is not possible.

In summary, TIMTOWDI, but here's the order of preference for incantations of submit_form:

  • fields + strict_forms
  • with_fields + strict_forms
  • with_fields

autocheck

autocheck can save you the overhead of checking status codes for success. You may outgrow it as your needs get more sophisticated, but it's a safe option to start with. Consider the following code:

This code doesn't die or warn, since it assumes you will "do the right thing" by checking status codes etc. Now try this:

If you run the above code you'll get something like "Error GETing foobar.comcom: URL must be absolute at...", which can also save you a lot of heartache.

HTTP::CookieJar::LWP

You are encouraged to install Mozilla::PublicSuffix and use HTTP::CookieJar::LWP as your cookie jar. This allows you to take advantage of HTTP::CookieJar which is more modern than HTTP::Cookies and "adheres as closely as possible to the user-agent rules of RFC 6265". See also https://metacpan.org/pod/HTTP::Cookies#LIMITATIONS

protocols_allowed

How about restricting the protocols your agent might follow? Let's look at protocols_allowed: This option is inherited directly from LWP::UserAgent. It allows you to whitelist the protocols you're willing to allow.

This will prevent you from inadvertently following URLs like file:///etc/passwd

protocols_forbidden

If you don't want to whitelist your protocols, you can blacklist them instead. Unsurprisingly, this option is called protocols_forbidden: This option is also inherited directly from LWP::UserAgent. It allows you to blacklist the protocols you're unwilling to allow.

This will prevent you from inadvertently following URLs like file:///etc/passwd

Creating a Stricter Agent

If we put together all of these together we get something like:

Using these settings in conjunction with strict_forms can help you with debugging, security and also your own sanity.

Making it Official

Much of what has been discussed here is now documented at https://metacpan.org/pod/WWW::Mechanize#BEST-PRACTICES. Edit: It has been pointed out to me that readers of this article may also benefit from my previous UserAgent Debugging Made Easy post.

See Also

If WWW::Mechanize does not fit your use case, see also WWW::Mechanize::Chrome, WWW::Mechanize::Firefox, WWW::Mechanize::Cached, LWPx::UserAgent::Cached and Mojo::UserAgent.

My “Go for Perl Hackers” Cheatsheet

Posted on by oalders

Last year I found myself working on some Go code at $work. When I'm trying to pick up constructs in a new language, I find it helpful to see how I would have done the same things in Perl. This sheet is far from complete, but I think it's already helpful. You can find it at https://github.com/oalders/go-for-perl-hackers. Comments, critique and pull requests are welcome. I've already had some helpful feedback via Twitter which I've incorporated.

As an aside, I did this a long back for Objective-C.

New defaults for Perl Linting in Vim’s Ale Plugin

Posted on by oalders

In my previous post, I talked about using Ale with vim for linting and syntax checking. Since that time, the Ale defaults for Perl have changed. Perl::Critic checks are still on by default, but you will need to enable the syntax and compile checks that are run via the perl binary itself.

The reasoning for the new default is described in the Ale docs. If you want to (re-)enable your Perl checks, you can follow the example in my dot files.