How I Spent my Perl Toolchain Summit 2017

Posted on by oalders

This was my 5th year of being invited to participate in the Perl Tool Chain Summit (formerly Perl QA Hackathon). It was a real pleasure to be invited to a rebranded version of the same helpful event.

Our Sponsors

This event would not have been possible without our sponsors. Let me take a moment to thank:

Overview

For the second year in a row, MetaCPAN was well represented at the event. This is important because it really does allow us to get much more work done. Getting everyone in the same room allows us to make decisions quickly and often deploy new ideas on the same day they’ve been discussed. Just like last year, we got a lot accomplished this year. I’m only going to touch on the work which I was directly involved with. Leo Lapworth, Mickey Nasriachi and Graham Knop both did much more than I’ll cover here.

I’m going to tackle this in roughly chronological order.

Wednesday

I arrived on Wednesday evening, and met up with most of the others at a local restaurant. It was good to see familiar faces and talk about what people had planned for the coming days.

Thursday

Thursday was the first real day of work. I had gotten the 06perms index into MetaCPAN API as part of my preparation for PTS, but it wasn’t yet exposed via the UI. So, on the Thursday I made it my mission to begin the front end work. I also spoke with Paul Johnson and Leo Lapworth about tighter integration of cpancover.com into MetaCPAN. I think this is important because many people probably don’t know about cpancover.com. This site runs coverage tests on all modules at upload time. It’s a great service and something that more people should probably know about.

In the evening we went out as a group to an Ethiopian restaurant. That was a first for me and I really enjoyed it.

Friday

On Friday we went live with the 06perms work from the previous day. Once it was up and running, Neil suggested I write a blog post to get the word out about the new feature.

The module permission views have already allowed people to spot inconsistencies in some module permissions. I had some back and forth with Babs Veloso and Neil about the permissions UI. We’ll be improving the UI moving forward, but it was important to get something out there now rather than perfecting it first.

I took a few minutes to release a new Net::HTTP which consisted of a patch from Shoichi Kaji, who was sitting just a few feet away from me at the time. I did this after a code review from Karen Etheridge, who was sitting in the next room.

I also got in a pull request to change some of the copy on the show more/less links for large Changes files.

Saturday

At meta::hack this past Fall, Joel Berger had worked on moving the MetaCPAN module search from the front end to the API, so that anyone could use it. We quietly merged his changes earlier this year, but the endpoint was lacking in tests and documentation. This was no fault of Joel’s. Leo set about to document this new endpoint and I spent part of my Saturday helping with this. He wanted to make some code changes, but didn’t want to do this without existing unit tests. So, I wrote some passing unit tests for the master branch and he used them in his documentation branch to establish that nothing was being broken by his cleanup work. Now we have better test coverage, some more documentation and better organized code.

While I was working on the tests, I came to the realization that our Travis build times were going to be very problematic. Some of our builds were taking up to 45 minutes or longer. This isn’t a huge issue on normal days where you can push something and come back to it later in the day. However, at a hackathon, where many people are pushing many branches, it becomes a significant constraint. Waiting 30-45 minutes for a passing build before a branch can be merged can be a real problem.

The TLDR is that build times are now in the 5-10 minute range for both the front and back end code. That was a fun journey for me and will be the subject of its own blog post. There’s some information in there which can be helpful to other projects.

As part of my efforts to speed up the build, it was helpful to have Shoichi Kaji sitting at the next table. I had a question about App::cpm and he helped me with a code sample and an explanation immediately. That allowed me to shave 50% off the build time for installing CPAN modules. I also had an issue with a test suite which needed some files to run sequentially and some in parallel. Graham Knop had an answer for me within minutes and Leon Timmermans came over a bit later to help me get the implementation right. This helped me squeeze a lot of extra performance out of the build in a very small amount of developer time.

I also took some time to add some missing meta::hack sponsors to the MetaCPAN sponsors page. I also got a very old pull request merged which fixes a bug that showed user favorites for modules which had been deleted.

Saturday evening a group of us walked into the old part of town. Leo Lapworth, Nicholas Rochelemagne, Joel Berger and I ended up a really fun little restaurant called Chez Sylvie, which offers regional cuisine. Some of us had eaten there three years ago and I had good memories of it. I’m happy to say that it did not disappoint.

Sunday

By Sunday, now that we had faster Travis builds, I took a proper look at the build logs and it became clear to me how noisy our build output was. It was really hard to distinguish between real problems and noise that just needed to be suppressed. I spent a fair bit of time working on that. I’m happy to say that build logs are now extremely succinct. Moving forward it will be much easier to spot warning messages as they appear and deal with them immediately. I also spent time with Leo troubleshooting MetaCPAN search results on the staging machine.

I had a discussion about producing a new, cleaner 02packages file for PAUSE. This would be an additional file which PAUSE produces, which I’m sure will be the subject of a blog post from someone else. As a MetaCPAN developer group, we had additionally had discussion about logistics for meta::hack v2, which we’re hoping to hold later this year. Exact date and location have yet to be determined.

I got on TGV from Lyon to Paris to finish off the day.  While on the train I spent some time partially implementing CPAN river data into the MetaCPAN UI.

Added to all of this, Mickey kept me busy every day reviewing code from his pull requests. He was also a valuable resource in helping me to get my own work finished faster.

I’d also like to send a special thankyou to Travis CI. Their customer support was excellent. I had told them we were at a hackathon and having some build issues and they immediately sprang into action. They played a big part in our weekend being even more productive.

Finally, I’d like to thank my employer for giving me time to attend the summit. To me, it feels like on the job training. I learned a great deal while I was hacking and/or chatting with other attendees. I’ll be able to put this knowledge to use on various open source and $work projects as I move forward.

Postscript

I spent my Sunday evening walking around Paris.  I walked from Notre Dame up to the Eiffel Tower, mostly along the Seine.  I got up early the next morning for a run.

I then caught a plane from CDG to FRA.  At FRA I got into a plane bound for YYZ.  After an apparently successful takeoff from FRA we were informed that the flight behind us had spotted liquid leaking from our aircraft.  The flight crew had then ascertained that one of the hydraulic systems was no longer functional.  So, we had to dump 54 tonnes of fuel before landing back at FRA with loads of emergency vehicles waiting for us on the ground.  We then spent probably a couple of hours in the aircraft while crews assessed the damage.  No air conditioning or meals were available during this time and eventually we all piled into buses that brought us back to the terminal.  Then into another bus that brought us to a fancy airport hotel where there was a warm meal waiting for us.  I got a text that my flight was rescheduled for 18:30 the next day.  For various reasons my 18:30 flight actually left the ground at 22:30.  So, I got to YYZ on Wednesday at 00:30 rather than on Monday at 19:30 as originally planned.

I’m glad we all got home safely.  The way the airline handled it was mostly fine.  It was a bit weak on communication, but it otherwise decent.  My big takeaway is that I’ve learned for next time to pack more stuff that I think I might need into my carry on rather than my checked bag.  🙂

Viewing Your Module Permissions on MetaCPAN

Posted on by oalders

We’re currently at the Perl Toolchain Summit in Lyon, working hard on improving MetaCPAN. One feature which we went live with yesterday is a view on CPAN module permissions. This means that you can now easily see which modules any CPAN author has permission to upload.

If you want to see every module which Neil Bowers has permissions on, you can go to https://metacpan.org/permission/author/NEILB. You can get to this page via the module permissions link on the left sidebar of a MetaCPAN author page.

To see who has permissions to upload a particular module, you can use the module view. This is not yet linked from MetaCPAN pages.

To see who has permissions to upload a particular distribution, you can use the distribution view. You can get to this page via the upload permissions link on the left sidebar of a MetaCPAN release page.

This new feature is helpful in a couple of ways. Firstly, if you’re looking for someone to patch and release a module for you, you can now easily view everyone who has permissions to do this. It’s tempting to believe that the last person who released a module is responsible for it, but the reality is that in many cases there are several people who can upload a new release. This helps shift the burden from one person to multiple people in many cases. In practice, if you want to chase someone to upload a new version of module X, you now can easily find the canonical list of responsible people.

Secondly, if you would like permission to begin uploading a certain module, you can now easily find the module owner. Only the module owner can assign co-maint to you. In the past I’ve made the mistake of contacting the last uploader and asking for co-maint. What I should have done is contact the person who actually is the owner. This can save you some running around trying to contact folks and waiting for replies from the wrong people.

Lastly, you can now audit module permissions. You may notice when looking at permissions that there are authors who should not have co-maint on a module. Or you may notice that authors have co-maint on some modules in a distribution but not on others. Having inconsistent permissions on modules in a release can lead to problems when a distribution is released by an author who has some missing permissions.

So, please do have a look at your permissions and give them a sanity check. If you notice a problem with a module which you don’t own, contact the PAUSE admins at [email protected] and they’ll be happy to work with you to sort out the correct permissions.

Preparing for LWP Hack Night

Posted on by oalders

I’ve had a couple of people ask me how they can prepare for LWP Hack Night, so I thought I’d just give a quick introduction to the set of modules.

I whipped up a graph of the various GitHub repositories to give you an idea of which are the most popular and which have the most open issues. Those stats seem to roughly correspond.

If you want to poke around the repositories on GitHub, that will give you an idea of where you can start.

Now, a lot of these modules have open issues in RT as well, so don’t let the GitHub numbers fool you. You can find the RT bugs for the various libraries here:

If you’ve looked at RT and GitHub, you can see there’s a monster amount of work to be done here, including but not limited to:

  • Establishing which bugs are still bugs
  • Establishing which patches could possibly be applied
  • Adding tests for existing pull requests
  • Rebasing some existing pull requests which have merge conflicts
  • Identifying bugs which are possibly in the wrong queue
  • Performing code review of existing pull requests which look like they are close to a state where they could be merged

How you might like to go about this is entirely up to you. If you have time before the meeting to identify some bugs which you may like to approach or comment on, please feel free to get started now. When we’re at the meeting, we can work out a plan to divide and conquer. There’s more than enough work to go around. We won’t (and can’t) clear this all up in one evening, but the point here is to make incremental improvements and learn something useful in the process.

Please feel free to get in touch with me in advance if you have any questions about this. The best way to do this would be via the Toronto Perl Mongers email list.

Introducing LWP::ConsoleLogger::Everywhere

Posted on by oalders

In an earlier post, I introduced you to LWP::ConsoleLogger. I’ve been using it heavily since then, but one thing I didn’t tackle was how to debug a user agent you can’t easily get it. Some modules don’t provide a public API which allows you to access their user agent. Or, maybe the user agent which you want to debug is so far removed from your code that you can’t easily access its public API. Previously, this was not an easy problem to solve. However, this is no longer the case. simbabque was kind enough to write LWP::ConsoleLogger::Everywhere.

It’s quite simple to use.

use LWP::ConsoleLogger::Everywhere;

Simply add this line to your code and run it. Any objects of the LWP::UserAgent family should now dump extensive logging information to your terminal. It can get a bit fancier than that, but this is really all you need to know in order to get started debugging 3rd party LWP::UserAgent-based HTTP requests.

meta::hack Wrap-up Report

Posted on by oalders

meta::hack v1

Earlier this month (Thu, Nov 16 – Sun, Nov 20) I had the pleasure of meeting up with 7 other Perl hackers at ServerCentral’s downtown Chicago offices, in order to hack on MetaCPAN. Before I get started, I’d like to thank our sponsors.

This hackathon wouldn’t have been possible without the overwhelming support of our sponsors. Our platinum sponsors were Booking.com and cPanel. Our gold sponsors were Elastic, FastMail, and Perl Careers. Our silver sponsors were ActiveState, Perl Services, ServerCentral and Advance Systems. Our bronze sponsors were Vienna.pm, Easyname, and the Enlightened Perl Organisation (EPO). Please take a moment to thank them for helping our Perl community.

For the past 2.5 years, we’ve been working off and on at porting MetaCPAN from Elasticsearch 0.20.2 to 1.x and (eventually) 2.x. There were enough breaking changes between the versions to make this a non-trivial task. We had made very good progress over the past two QA hackathons, but the job was just too big to finish in the hours that we had available.

After the QA Hackathon in Rugby, I spoke to Neil Bowers about how we might go about doing some fundraising. Neil was so kind as to offer to help. His offer to help soon evolved into him taking on all of the work (thanks Neil)! Neil worked his magic and got the event fully funded. I know there was a lot of work invovled, but he made it look easy. Mark Keating and the Enlightened Perl Organization kindly took on the financial side of things, invoicing and accepting payment from sponsors. Without EPO and Neil, this event never would have taken place. (Please do take a moment to thank them).

While this was going on, we began searching for a venue. Joel Berger offered to host us at ServerCentral in Chicago and we immediately took him up on the offer. After that it was just a matter of folks booking plane tickets and getting approval from employers for the time off.

The final list of invitees was:

  • Brad Lhotsky (San Francisco)
  • Doug Bell (Chicago)
  • Graham Knop (Baltimore)
  • Joel Berger (Chicago)
  • Leo Lapworth (London)
  • Mickey Nasriachi (Amsterdam)
  • Olaf Alders (Toronto)
  • Thomas Sibley (Seattle)

The event was invitation only. We did this in order to maximize the amount of work we’d be able to finish at the event. [Insert reference to “The Mythical Man Month”]. Everyone who participated was already up to speed on the internals of the project or has an area of expertise which we needed in order to complete our goal of launching fully with v1 of the API. Because everyone already had a working VM and working knowledge of the project, we were able to tackle the problems at hand right from the first morning.

As far as living space goes, we initially had looked at renting hotel rooms, but the cost would have made it almost prohibitive to meet in Chicago. After doing some research, we booked two apartments (each with 3 bedrooms) on the same floor of a condo building in the Lincoln Park area of Chicago. We booked the accommodations via booking.com of course. 🙂 I think we were happy with the housing. Everyone had their own room and we had big enough living rooms for all of us to meet up mornings and some evenings. At the end of the day the rental was a fraction of the price of a Chicago hotel. I’ve also made a mental note not to be the last one to arrive in town. Apparently it also means you get the smallest room.

Each day we took the subway downtown to ServerCentral. We had a dedicated boardroom in the office with a large TV that we could use for sharing presentations, IRC chat or error logs. ServerCentral also sponsored lunch each day of the event. Extra monitors were also available for those who wanted them. (Lots of Roost laptop stands were to be seen. Also lots of people who couldn’t figure out how to open them after having collapsed them for the first time in forever).

After settling in at the office we’d discuss our plans for the day and map out goals for that day. We had breakout discussions where appropriate but the time spent not writing code was minimal. Generally, as a group, we worked well into the evenings. We didn’t get the full Chicago experience, but we got a lot done. We did make it to the Chicago Christkindlmarkt, which was a few blocks from the office and we went out for a breakfast and a dinner as well. Minimal downtime, but the breaks we had were lots of fun.

Day one was spent removing anything which was blocking the API upgrade. Wishlist items were ignored and as a group we worked really well. Lots of pull requests were created, reviewed and merged.

By day two of the hackathon we flipped the switch and went live with the new API. We could have waited a bit longer, but we opted to make the change earlier so that we could troubleshoot any issues as a group and watch the error logs in real time. There were no showstopping bugs and the transition was actually pretty smooth.

Day three was spent squashing some of the bugs which came up after the upgrade. We also started to tackle some wishlist items.

Day four was a slightly shorter day. We wrapped around 4 PM. Some of us went to check out “the Bean” before flying out while Leo and I headed right for our respective airports.

This list is by no means exhaustive, but over this long weekend we:

  • moved ++ data to v1 of the API.
  • moved https://metacpan.org to v1 of the API.
  • implemented load balancing via Fastly, our CDN sponsor.
  • reduced noise in the logs by squashing bugs which generated warnings or exceptions.
  • updated our API documentation as well as the metacpan-examples GitHub repository from v0 to v1.
  • published an upgrade document which explains to how upgrade your query syntax and configuration for v1.
  • moved http://explorer.metacpan.org to v1 of the API.
  • began work on streaming logs to Elasticsearch.
  • began moving the query logic that metacpan.org uses over to the API so that other clients can use this same logic.
  • began porting author queries from metacpan.org to the API as well.
  • added a meta::hack event page along with sponsor info to metacpan.org.
  • continued work on adding a /permission endpoint which will provide access to the data in 06perms.txt.
  • added more tests for the /download_url endpoint which translates module names into download URL. Specifically this is meant to be used by cpanm.
  • added snapshotting of Elasticsearch indices in v1 so that we can easily restore from backup.

/permission is something I spent a fair bit of my time working on over the last two days. Having 06perms.txt data in the API will mean that we can display a list of all authors who have maint on a module on metacpan.org. This will make it easier to track down authors who can release a module, particularly for those who aren’t familiar with the way PAUSE works. I think this branch is probably about 1.5 years old, so I was happy to get the time to try to finish it off. I didn’t quite get there, but that’s okay. It was a wishlist item and it’s actually quite close to being released.

Also of note is the fact that we’ve now officially deprecated the v0 API. There is a 6 month runway to move clients over to v1 and v0 will be taken offline on or after June 1, 2017.

Since https://metacpan.org now uses v1 of the API, results for v0 are no longer available. If you have a client which uses v0 of the API, please feel free to reach out to us with any concerns you may have about making the switch.

If you rely on updated ++ data, you’ll need to switch to v1 now, as ++ data in v0 is no longer being updated. The indexer is, however, still running on v0, so it will still find and index new CPAN uploads. v0 development is officially closed. Any v0 bugs (barring catastrophic issues) will likely not be addressed. v0 has been around for just over 6 years now. It has served us well, but it’s time to let it go. [Insert musical scene with a talking snowman, an ice queen and her loyal sister.]

UserAgent Debugging Made Easy

Posted on by oalders

Earlier today I saw a recent blog post from Gabor Szabo. In it, he shows a very concise way to handle Basic Authentication using LWP::UserAgent. Now, what if you had a problem running the script? How might you go about debugging it? You could add a bunch of print statements. Maybe dump the request and the response objects. That’s entirely valid, but I want to show you a slightly simpler way of going about it, using LWP::ConsoleLogger::Easy.

Gabor’s original script looks like this:

use strict;
use warnings;
use v5.10;
use LWP::UserAgent;
use HTTP::Request::Common;
 
my $ua = LWP::UserAgent->new();
my $request = GET 'https://pause.perl.org/pause/authenquery';
 
$request->authorization_basic('szabgab', '*******');
 
my $response = $ua->request($request);
say $response->as_string();

Let’s run it to see what the output looks like.

olaf$ perl gabor.pl
HTTP/1.0 401 Unauthorized
Connection: close
Date: Thu, 29 Sep 2016 02:07:34 GMT
WWW-Authenticate: Basic realm="PAUSE"
Content-Length: 22
Content-Type: text/plain
Client-Date: Thu, 29 Sep 2016 02:07:34 GMT
Client-Peer: 207.171.7.119:443
Client-Response-Num: 1
Client-SSL-Cert-Issuer: /C=US/O=GeoTrust, Inc./CN=RapidSSL CA
Client-SSL-Cert-Subject: /serialNumber=K6R2GP4nh37grllJm9PZlQm0SS-oSNZ4/C=DE/O=pause.perl.org/OU=GT38611495/OU=See www.rapidssl.com/resources/cps (c)11/OU=Domain Control Validated - RapidSSL(R)/CN=pause.perl.org
Client-SSL-Cipher: AES128-GCM-SHA256
Client-SSL-Socket-Class: IO::Socket::SSL

Authorization required

Here’s the debugging version. Note the important changes are on lines 4 and 9.

use strict;
use warnings;
use v5.10;
use LWP::ConsoleLogger::Easy qw( debug_ua );
use LWP::UserAgent;
use HTTP::Request::Common;

my $ua      = LWP::UserAgent->new();
debug_ua( $ua );
my $request = GET 'https://pause.perl.org/pause/authenquery';

$request->authorization_basic( 'szabgab', '*******' );

my $response = $ua->request($request);

The output we get is:

 olaf$ perl basic-authentication.pl
GET https://pause.perl.org/pause/authenquery

.----------------+----------------------------.
| Request Header | Value                      |
+----------------+----------------------------+
| Authorization  | Basic c3phYmdhYjoqKioqKioq |
| User-Agent     | libwww-perl/6.15           |
'----------------+----------------------------'

==> 401 Unauthorized

.-------------------------+-----------------------------------------------------------------------------------------.
| Response Header         | Value                                                                                   |
+-------------------------+-----------------------------------------------------------------------------------------+
| Client-Date             | Thu, 29 Sep 2016 01:54:23 GMT                                                           |
| Client-Peer             | 207.171.7.119:443                                                                       |
| Client-Response-Num     | 1                                                                                       |
| Client-SSL-Cert-Issuer  | /C=US/O=GeoTrust, Inc./CN=RapidSSL CA                                                   |
| Client-SSL-Cert-Subject | /serialNumber=K6R2GP4nh37grllJm9PZlQm0SS-oSNZ4/C=DE/O=pause.perl.org/OU=GT38611495/OU=- |
|                         | See www.rapidssl.com/resources/cps (c)11/OU=Domain Control Validated - RapidSSL(R)/CN=- |
|                         | pause.perl.org                                                                          |
| Client-SSL-Cipher       | AES128-GCM-SHA256                                                                       |
| Client-SSL-Socket-Class | IO::Socket::SSL                                                                         |
| Connection              | close                                                                                   |
| Content-Length          | 22                                                                                      |
| Content-Type            | text/plain                                                                              |
| Date                    | Thu, 29 Sep 2016 01:54:23 GMT                                                           |
| WWW-Authenticate        | Basic realm="PAUSE"                                                                     |
'-------------------------+-----------------------------------------------------------------------------------------'

.------------------------.
| Content                |
+------------------------+
| Authorization required |
'------------------------'

.------------------------.
| Text                   |
+------------------------+
| Authorization required |
'------------------------'

You can see that the debugging version is just one line longer. I added 2 lines and removed a print statement. It prints out a whole pile of (nicely?) formatted information. Let’s try running it with valid credentials. (Brace yourself, there’s going to be a lot of output.)

olaf$ LWPCL_REDACT_HEADERS='Authorization' perl basic-authentication.pl
GET https://pause.perl.org/pause/authenquery

.----------------+------------------.
| Request Header | Value            |
+----------------+------------------+
| Authorization  | [REDACTED]       |
| User-Agent     | libwww-perl/6.15 |
'----------------+------------------'

==> 200 OK

.-------------------------+-----------------------------------------------------------------------------------------.
| Response Header         | Value                                                                                   |
+-------------------------+-----------------------------------------------------------------------------------------+
| Cache-Control           | no-cache                                                                                |
| Client-Date             | Thu, 29 Sep 2016 02:06:02 GMT                                                           |
| Client-Peer             | 207.171.7.119:443                                                                       |
| Client-Response-Num     | 1                                                                                       |
| Client-SSL-Cert-Issuer  | /C=US/O=GeoTrust, Inc./CN=RapidSSL CA                                                   |
| Client-SSL-Cert-Subject | /serialNumber=K6R2GP4nh37grllJm9PZlQm0SS-oSNZ4/C=DE/O=pause.perl.org/OU=GT38611495/OU=- |
|                         | See www.rapidssl.com/resources/cps (c)11/OU=Domain Control Validated - RapidSSL(R)/CN=- |
|                         | pause.perl.org                                                                          |
| Client-SSL-Cipher       | AES128-GCM-SHA256                                                                       |
| Client-SSL-Socket-Class | IO::Socket::SSL                                                                         |
| Connection              | close                                                                                   |
| Content-Length          | 11251                                                                                   |
| Content-Type            | text/html; charset=utf-8                                                                |
| Date                    | Thu, 29 Sep 2016 02:06:02 GMT                                                           |
| Last-Modified           | Thu, 29 Sep 2016 02:06:02 GMT                                                           |
| Link                    | ; rel="shortcut icon"; type="image/jpeg", - |
|                         | ; rel="stylesheet"; title="pause"; type="text/css"                                      |
| Pragma                  | no-cache                                                                                |
| Title                   | PAUSE: menu                                                                             |
| Vary                    | accept-encoding                                                                         |
'-------------------------+-----------------------------------------------------------------------------------------'

.-------------------------------------------------------------------------------------------------------------------.
| Content                                                                                                           |
+-------------------------------------------------------------------------------------------------------------------+
| PAUSE: menu                                                                            |
|                                     |
|                                     |
| 
PAUSE Logo
The [Perl programming] Authors Upload                                  |
|   Server
OALDERS <[email protected]>
encrypted session
     |
| 

   |
|                                                                                        |
|                                                                     |
|                                                                                          |
|                                                                                    |
|                                                                                |
|                                                                                     |
|      |
|                                                                                    |
|                                                                                         |
|                                                                                |
|                                                                                        |
|                                                                  |
|                                                                                      |
|                                                                                   |
|                                                                        |
|                                                                                         |
|                                                                                      |
| - |
|                                                                                 |
|                                                                      |
|                                                                                      |
|                                                                                    |
|                                                                                   |
| 
Public menu
Request PAUSE account
About PAUSE
On The Naming of Modu- |
| les
PAUSE News
PAUSE History
Imprint/Impressum
List of pumpkins
User menu
Files
Upload a file to CPAN
Show my files
Repair a Pending Upload
Delete Files
Permissions
View Permissions
Change Permissions
Utils
Force Reindexing
Reset Version
Tail Daemon Logfile
Account
Edit Account Info
Change Password
About Logging Out
                                                                                                          |
| 		                                                                                      |
| 
Hi Olaf Alders,
please choose an action from the menu.
                          |
| 
The usermenu to the left shows all menus available to                                                          |
|     you, the table below shows descriptions for all menues available                                              |
|     to anybody on PAUSE.
                                                                                      |
| 
                                        |
|                                                                                            |
|  |
|                                                                                            |
|                                                                                             |
|                                                                                            |
|                                                                                                    |
|                                                                                            |
|                                                                         |
|                                                                                            |
|             |
|                                                                                            |
|                       |
|                                                                                            |
|                        |
|                                                                                            |
|  |
|                                                                                            |
|                                                                                               |
|                                                                                            |
|                |
|                                                                                            |
|                                                                           |
|                                                                                            |
|                                                                            |
|                                                                                            |
|                                                                                                   |
|                                                                                            |
|                                                       |
|                                                                                            |
|                                        |
|                                                                                            |
|                                                    |
|                                                                                            |
|                           |
|                                                                                            |
|                                                                   |
|                                                                                            |
|                                                                                            |
|                                                                                            |
|                             |
|                                                                                            |
|                                                                                        |
|                                                                                            |
|                                                                                |
|                                                                                            |
|                                                                                      |
|                                                                                            |
|                                                                                       |
|                                                                                            |
|                                                                               |
|                                                                                            |
|                                                                                               |
|                                                                                            |
|                                                                 |
|                                                                                            |
|                                                                                                                |
|                                                                                            |
|                                                                                                    |
|                                                                                            |
|                                                                                                          |
|                                                                                            |
|                                                                                      |
|                                                                                            |
|                                         |
| 
ActionGroupDescription
Request PAUSE accountpublicApply for a PAUSE account.
Forgot Password?publicA passwordmailer that sends you a             |
| 			password that enables you to set a new                                                                         |
| 			password.
About PAUSEpublicSame as modules/04pause.html on any CPAN s- |
| erver
On The Naming of ModulespublicA couple of suggestions- |
|  that hopefully get you on track
PAUSE NewspublicWhat's going on on PAUSE
PAUSE HistorypublicOld News
Imprint/ImpressumpublicN/A
List of pumpkinspublicA list, also available as YAML
Upload a file to CPANuserThis is the heart of the Upload        |
| 			 Server, the page most heavily used on                                                                     |
| 			 PAUSE.
Show my filesuserfind . -ls resemblance
Repair a Pending UploaduserWhen an upload you requested hangs    |
| 			   for some reason, you can go here and edit                                                                   |
| 			   the file to be uploaded.
Delete FilesuserSchedule files for deletion.                  |
| 			      There is a delay until the deletion                                                                      |
| 			      really happens. Until then you can also                                                                  |
| 			      undelete files here.
View PermissionsuserWhose uploads of what are being indexed on- |
|  PAUSE
Change PermissionsuserEnable other users to upload a           |
|                              module for any of your namespaces, manage                                            |
|                              your own permissions.
Force ReindexinguserTell the indexer to index a file again         |
|                          (e.g. after a change in the perms table)
Reset VersionuserOverrule the record of the current          |
|                            version number of a module that the indexer                                            |
|                            uses and set it to 'undef'
Tail Daemon LogfileuserN/A
Edit Account InfouserEdit your user name, your email             |
| 			   addresses (both public and secret one),                                                                     |
| 			   change the URL of your homepage.
Change PassworduserChange your password any time             |
| 				you want.
About Logging OutuserN/A
Select Mailinglist/ActionmlreprRepresentatives of mailing |
| 					  lists have their special                                                                                   |
| 					  menu here.
Show Mailinglist RepsmlreprAdmins and the representatives th- |
| emselves                                                                                                          |
|                                       can lookup who is                                                           |
| 				      elected to be representative of                                                                         |
| 				      a mailing list.
Add a User or MailinglistadminAdmins can add users or             |
| 				  mailinglists.
Look up the forward email addressadminAdmins can look whe- |
| re email should go
Manage a registration request (alpha)adminshow/reject - |
| open registration requests
Edit a MailinglistadminAdmins and mailing list                     |
| 				 representatives can change the name,                                                                         |
| 				 address and description of a mailing                                                                         |
| 				 list.
Select User/ActionadminAdmins can access PAUSE as-if           |
| 				     they were somebody else. Here                                                                            |
| 				     they select a user/action pair.
Post a messageadminPost a message to a specific user.
Show/Delete MsgsadminDelete your messages from the message b- |
| oard.
Index users with digrams (BROKEN)adminBatch-index all users.<- |
| /td>
Show bad xhtml outputadminMonitor bad xhtml output stored fro- |
| m previous sessions
coredumpadminN/A
                                                                                                          |
| 
                                                                                                |
| 
                                                                                                |
| 
                                                               |
|                                                                                                               |
|                                                                                                    |
|                         |
|            |
|                                                                                                              |
|                                                                                                               |
|                                                                                             |
|                                                                                                              |
|                                                                                                              |
|                                                                                                              |
|      
 
Rev: 1071.02
To validate, download page first.

                                                                                                       |
|         Valid CSS!                                                         |
|                                                                                                               |
|                                                                |
|          Valid XHTML 1.0!                                                       |
|                                                                                                               |
|        
                                                                                                     |
|                                                                                                     |
'-------------------------------------------------------------------------------------------------------------------'

.-------------------------------------------------------------------------------------------------------------------.
| Text                                                                                                              |
+-------------------------------------------------------------------------------------------------------------------+
| PAUSE: menu The [Perl programming] Authors Upload ServerOALDERS <[email protected]>encrypted sessio- |
| n Public menuRequest PAUSE account About PAUSE On The Naming of Modules PAUSE News PAUSE History Imprint/Impress- |
| um List of pumpkins User menuFiles Upload a file to CPAN Show my files Repair a Pending Upload Delete Files Perm- |
| issions View Permissions Change Permissions Utils Force Reindexing Reset Version Tail Daemon Logfile Account Edi- |
| t Account Info Change Password About Logging Out   Hi Olaf Alders,please choose an action from the menu. Th- |
| e usermenu to the left shows all menus available to you, the table below shows descriptions for all menues avail- |
| able to anybody on PAUSE. ActionGroupDescription Request PAUSE accountpublicApply for a PAUSE account. Forgot Pa- |
| ssword?publicA passwordmailer that sends you a password that enables you to set a new password. About PAUSEpubli- |
| cSame as modules/04pause.html on any CPAN server On The Naming of ModulespublicA couple of suggestions that hope- |
| fully get you on track PAUSE NewspublicWhat's going on on PAUSE PAUSE HistorypublicOld News Imprint/Impressumpub- |
| licN/A List of pumpkinspublicA list, also available as YAML Upload a file to CPANuserThis is the heart of the Up- |
| load Server, the page most heavily used on PAUSE. Show my filesuserfind . -ls resemblance Repair a Pending Uploa- |
| duserWhen an upload you requested hangs for some reason, you can go here and edit the file to be uploaded. Delet- |
| e FilesuserSchedule files for deletion. There is a delay until the deletion really happens. Until then you can a- |
| lso undelete files here. View PermissionsuserWhose uploads of what are being indexed on PAUSE Change Permissions- |
| userEnable other users to upload a module for any of your namespaces, manage your own permissions. Force Reindex- |
| inguserTell the indexer to index a file again (e.g. after a change in the perms table) Reset VersionuserOverrule- |
|  the record of the current version number of a module that the indexer uses and set it to 'undef' Tail Daemon Lo- |
| gfileuserN/A Edit Account InfouserEdit your user name, your email addresses (both public and secret one), change- |
|  the URL of your homepage. Change PassworduserChange your password any time you want. About Logging OutuserN/A S- |
| elect Mailinglist/ActionmlreprRepresentatives of mailing lists have their special menu here. Show Mailinglist Re- |
| psmlreprAdmins and the representatives themselves can lookup who is elected to be representative of a mailing li- |
| st. Add a User or MailinglistadminAdmins can add users or mailinglists. Look up the forward email addressadminAd- |
| mins can look where email should go Manage a registration request (alpha)adminshow/reject open registration requ- |
| ests Edit a MailinglistadminAdmins and mailing list representatives can change the name, address and description- |
|  of a mailing list. Select User/ActionadminAdmins can access PAUSE as-if they were somebody else. Here they sele- |
| ct a user/action pair. Post a messageadminPost a message to a specific user. Show/Delete MsgsadminDelete your me- |
| ssages from the message board. Index users with digrams (BROKEN)adminBatch-index all users. Show bad xhtml outpu- |
| tadminMonitor bad xhtml output stored from previous sessions coredumpadminN/A   Rev: 1071.02 To validate, d- |
| ownload page first.                                                                                               |
'-------------------------------------------------------------------------------------------------------------------'

You can see that I ran the script with LWPCL_REDACT_HEADERS='Authorization'. That’s a handy flag to use if you want to copy/paste an example when asking for help publicly. It replaced the Authorization header value with [REDACTED]. That’s maybe not a big deal here, but there are cases where it’s more important. See also LWP_REDACT_PARAMS.

Let’s make it prettier. We’ll do this by installing HTML::FormatText::Lynx.

Let’s run it again. I’ll only show you the changed part. Instead of just displaying the text with the HTML stripped away, we get something nicer to look at.

.------------------------------------------------------------------------------------------------------------------------------------------------------.
| Text                                                                                                                                                 |
+------------------------------------------------------------------------------------------------------------------------------------------------------+
| [1]PAUSE Logo                                                                                                                                        |
|                                                                                                                                                      |
| The [Perl programming] Authors Upload Server                                                                                                         |
|                                                                                                                                                      |
|    OALDERS                                                                                                                 |
|    encrypted session                                                                                                                                 |
|                                                                                                                                                      |
|    Public menu                                                                                                                                       |
|    [2]Request PAUSE account                                                                                                                          |
|    [3]About PAUSE                                                                                                                                    |
|    [4]On The Naming of Modules                                                                                                                       |
|    [5]PAUSE News                                                                                                                                     |
|    [6]PAUSE History                                                                                                                                  |
|    [7]Imprint/Impressum                                                                                                                              |
|    [8]List of pumpkins                                                                                                                               |
|    User menu                                                                                                                                         |
|    Files                                                                                                                                             |
|    [9]Upload a file to CPAN                                                                                                                          |
|    [10]Show my files                                                                                                                                 |
|    [11]Repair a Pending Upload                                                                                                                       |
|    [12]Delete Files                                                                                                                                  |
|    Permissions                                                                                                                                       |
|    [13]View Permissions                                                                                                                              |
|    [14]Change Permissions                                                                                                                            |
|    Utils                                                                                                                                             |
|    [15]Force Reindexing                                                                                                                              |
|    [16]Reset Version                                                                                                                                 |
|    [17]Tail Daemon Logfile                                                                                                                           |
|    Account                                                                                                                                           |
|    [18]Edit Account Info                                                                                                                             |
|    [19]Change Password                                                                                                                               |
|    [20]About Logging Out                                                                                                                             |
|                                                                                                                                                      |
|                                                                                                                                                      |
| Hi Olaf Alders,                                                                                                                                      |
| please choose an action from the menu.                                                                                                               |
|                                                                                                                                                      |
|    The usermenu to the left shows all menus available to you, the table                                                                              |
|    below shows descriptions for all menues available to anybody on PAUSE.                                                                            |
|    Action Group Description                                                                                                                          |
|    Request PAUSE account public Apply for a PAUSE account.                                                                                           |
|    Forgot Password? public A passwordmailer that sends you a password that                                                                           |
|    enables you to set a new password.                                                                                                                |
|    About PAUSE public Same as modules/04pause.html on any CPAN server                                                                                |
|    On The Naming of Modules public A couple of suggestions that hopefully                                                                            |
|    get you on track                                                                                                                                  |
|    PAUSE News public What's going on on PAUSE                                                                                                        |
|    PAUSE History public Old News                                                                                                                     |
|    Imprint/Impressum public N/A                                                                                                                      |
|    List of pumpkins public A list, also available as YAML                                                                                            |
|    Upload a file to CPAN user This is the heart of the Upload Server, the                                                                            |
|    page most heavily used on PAUSE.                                                                                                                  |
|    Show my files user find . -ls resemblance                                                                                                         |
|    Repair a Pending Upload user When an upload you requested hangs for                                                                               |
|    some reason, you can go here and edit the file to be uploaded.                                                                                    |
|    Delete Files user Schedule files for deletion. There is a delay until                                                                             |
|    the deletion really happens. Until then you can also undelete files                                                                               |
|    here.                                                                                                                                             |
|    View Permissions user Whose uploads of what are being indexed on PAUSE                                                                            |
|    Change Permissions user Enable other users to upload a module for any                                                                             |
|    of your namespaces, manage your own permissions.                                                                                                  |
|    Force Reindexing user Tell the indexer to index a file again (e.g.                                                                                |
|    after a change in the perms table)                                                                                                                |
|    Reset Version user Overrule the record of the current version number of                                                                           |
|    a module that the indexer uses and set it to 'undef'                                                                                              |
|    Tail Daemon Logfile user N/A                                                                                                                      |
|    Edit Account Info user Edit your user name, your email addresses (both                                                                            |
|    public and secret one), change the URL of your homepage.                                                                                          |
|    Change Password user Change your password any time you want.                                                                                      |
|    About Logging Out user N/A                                                                                                                        |
|    Select Mailinglist/Action mlrepr Representatives of mailing lists have                                                                            |
|    their special menu here.                                                                                                                          |
|    Show Mailinglist Reps mlrepr Admins and the representatives themselves                                                                            |
|    can lookup who is elected to be representative of a mailing list.                                                                                 |
|    Add a User or Mailinglist admin Admins can add users or mailinglists.                                                                             |
|    Look up the forward email address admin Admins can look where email                                                                               |
|    should go                                                                                                                                         |
|    Manage a registration request (alpha) admin show/reject open                                                                                      |
|    registration requests                                                                                                                             |
|    Edit a Mailinglist admin Admins and mailing list representatives can                                                                              |
|    change the name, address and description of a mailing list.                                                                                       |
|    Select User/Action admin Admins can access PAUSE as-if they were                                                                                  |
|    somebody else. Here they select a user/action pair.                                                                                               |
|    Post a message admin Post a message to a specific user.                                                                                           |
|    Show/Delete Msgs admin Delete your messages from the message board.                                                                               |
|    Index users with digrams (BROKEN) admin Batch-index all users.                                                                                    |
|    Show bad xhtml output admin Monitor bad xhtml output stored from                                                                                  |
|    previous sessions                                                                                                                                 |
|    coredump admin N/A                                                                                                                                |
|      __________________________________________________________________                                                                              |
|                                                                                                                                                      |
|                                                                                                                                                      |
|    Rev: 1071.02                                                                                                                                      |
|                                                                                                                                                      |
|                                          To validate, download page first.                                                                           |
|                                                                                                                                                      |
|      [21]Valid CSS! [22]Valid XHTML 1.0!                                                                                                             |
|                                                                                                                                                      |
| References                                                                                                                                           |
|                                                                                                                                                      |
|    1. https://pause.perl.org/pause/authenquery                                                                                                       |
|    2. https://pause.perl.org/pause/authenquery?ACTION=request_id                                                                                     |
|    3. https://pause.perl.org/pause/authenquery?ACTION=pause_04about                                                                                  |
|    4. https://pause.perl.org/pause/authenquery?ACTION=pause_namingmodules                                                                            |
|    5. https://pause.perl.org/pause/authenquery?ACTION=pause_05news                                                                                   |
|    6. https://pause.perl.org/pause/authenquery?ACTION=pause_06history                                                                                |
|    7. https://pause.perl.org/pause/authenquery?ACTION=pause_04imprint                                                                                |
|    8. https://pause.perl.org/pause/authenquery?ACTION=who_pumpkin                                                                                    |
|    9. https://pause.perl.org/pause/authenquery?ACTION=add_uri                                                                                        |
|   10. https://pause.perl.org/pause/authenquery?ACTION=show_files                                                                                     |
|   11. https://pause.perl.org/pause/authenquery?ACTION=edit_uris                                                                                      |
|   12. https://pause.perl.org/pause/authenquery?ACTION=delete_files                                                                                   |
|   13. https://pause.perl.org/pause/authenquery?ACTION=peek_perms                                                                                     |
|   14. https://pause.perl.org/pause/authenquery?ACTION=share_perms                                                                                    |
|   15. https://pause.perl.org/pause/authenquery?ACTION=reindex                                                                                        |
|   16. https://pause.perl.org/pause/authenquery?ACTION=reset_version                                                                                  |
|   17. https://pause.perl.org/pause/authenquery?ACTION=tail_logfile                                                                                   |
|   18. https://pause.perl.org/pause/authenquery?ACTION=edit_cred                                                                                      |
|   19. https://pause.perl.org/pause/authenquery?ACTION=change_passwd                                                                                  |
|   20. https://pause.perl.org/pause/authenquery?ACTION=pause_logout                                                                                   |
|   21. http://jigsaw.w3.org/css-validator/                                                                                                            |
|   22. http://validator.w3.org/file-upload.html                                                                                                       |
'------------------------------------------------------------------------------------------------------------------------------------------------------'

Now, we can also turn down the verbosity of the script by passing a flag to debug_ua(). Any integer from 0-8 will do the trick. Let’s try 6.

use strict;
use warnings;
use v5.10;
use LWP::ConsoleLogger::Easy qw( debug_ua );
use LWP::UserAgent;
use HTTP::Request::Common;

my $ua      = LWP::UserAgent->new();
debug_ua( $ua, 6 );
my $request = GET 'https://pause.perl.org/pause/authenquery';

$request->authorization_basic( 'oalders', 'seekrit' );

my $response = $ua->request($request);

Let’s see what we get:

 olaf$ LWPCL_REDACT_HEADERS='Authorization' perl basic-authentication.pl
GET https://pause.perl.org/pause/authenquery

.----------------+------------------.
| Request Header | Value            |
+----------------+------------------+
| Authorization  | [REDACTED]       |
| User-Agent     | libwww-perl/6.15 |
'----------------+------------------'

==> 200 OK

.-------------------------+----------------------------------------------------------------------------------------------------------------------------.
| Response Header         | Value                                                                                                                      |
+-------------------------+----------------------------------------------------------------------------------------------------------------------------+
| Cache-Control           | no-cache                                                                                                                   |
| Client-Date             | Thu, 29 Sep 2016 02:13:51 GMT                                                                                              |
| Client-Peer             | 207.171.7.119:443                                                                                                          |
| Client-Response-Num     | 1                                                                                                                          |
| Client-SSL-Cert-Issuer  | /C=US/O=GeoTrust, Inc./CN=RapidSSL CA                                                                                      |
| Client-SSL-Cert-Subject | /serialNumber=K6R2GP4nh37grllJm9PZlQm0SS-oSNZ4/C=DE/O=pause.perl.org/OU=GT38611495/OU=See www.rapidssl.com/resources/cps - |
|                         | (c)11/OU=Domain Control Validated - RapidSSL(R)/CN=pause.perl.org                                                          |
| Client-SSL-Cipher       | AES128-GCM-SHA256                                                                                                          |
| Client-SSL-Socket-Class | IO::Socket::SSL                                                                                                            |
| Connection              | close                                                                                                                      |
| Content-Length          | 11251                                                                                                                      |
| Content-Type            | text/html; charset=utf-8                                                                                                   |
| Date                    | Thu, 29 Sep 2016 02:13:51 GMT                                                                                              |
| Last-Modified           | Thu, 29 Sep 2016 02:13:51 GMT                                                                                              |
| Link                    | ; rel="shortcut icon"; type="image/jpeg", ; rel="stylesheet"; title="pause"; - |
|                         | type="text/css"                                                                                                            |
| Pragma                  | no-cache                                                                                                                   |
| Title                   | PAUSE: menu                                                                                                                |
| Vary                    | accept-encoding                                                                                                            |
'-------------------------+----------------------------------------------------------------------------------------------------------------------------'

That’s far easier to read now.

This is just a very basic example of what you can do with LWP::ConsoleLogger::Easy. There’s a lot more you can do with it and it’s all laid out for you in the documentation. It really shines when you have a user agent which is going through multiple links or if you’re debugging someone else’s API calls. Have fun with it. It beats inserting arbitrary print statements and it could save you from pulling a lot of your own hair out someday.

Make libwww-perl Great Again ™

Posted on by oalders

You may have noticed that WWW::Mechanize has seen some releases over the last couple of months. No big, breaking changes, but bugs have been fixed and enhancements have been shipped. This module is part of the libwww-perl ecosystem and also a part of the libwww-perl GitHub organization, to which I now also belong. I started pestering people to get involved because these modules, although quite important in the CPAN scheme of things, aren’t really on a regular release cycle. I don’t have the backstory on everything and this is not a complaint about anybody who has commit bits, maint or co-maint. It’s just an observation that a lot of modules on CPAN depend on the modules in this organization. The issue queues are slowly growing and pull requests are going unmerged.

I think there’s a fairly simple solution to all of this and my hope is that we can crowdsource enough mindshare to get this done. (I’m hoping that previous sentence is fully buzzword compliant).

Now, I don’t have a lot of hours of spare time to devote to this stuff in any given week, but this doesn’t all fall to me anyway. What I’d like to see is more eyeballs on the code. If you’d like to get involved or you have an interest in seeing things move along with any of these modules, please go through any outstanding pull requests and issues. Even comments such as “LGTM” (looks good to me) are very helpful. If enough people who know what they’re doing stamp a “LGTM” on a pull request, then that signals that this code is less risky to merge. If people can look into open issues and identify what is or is not a bug and what is or is not RFC-compliant, then that can speed up the issue review cycle as well.

If you’d like to join the libwww-perl org, then that would be great as well. Probably a good first step for that would be to get involved with reviewing open pull requests and issues or even contributing some code.

Here’s a quick summary of the repositories which are currently in the org:

olaf$ github-mergevelocity --url libwww-perl/WWW-Mechanize --url libwww-perl/libwww-perl --url libwww-perl/URI --url libwww-perl/Net-HTTP --url libwww-perl/HTTP-Message --url libwww-perl/LWP-Protocol-https --url libwww-perl/www-mechanize-cached
.-------------+----------------------+----------+-----+----------+---------------+----------+---------------+----------+-----------------.
| user        | repo                 | velocity | PRs | merged   | merge days    | closed   | close days    | open     | open days       |
+-------------+----------------------+----------+-----+----------+---------------+----------+---------------+----------+-----------------+
| libwww-perl | www-mechanize-cached | 157      | 10  | 80% (8)  | 6/PR (45)     | 0        | 0             | 20% (2)  | 1/PR (2)        |
| libwww-perl | LWP-Protocol-https   | -159     | 23  | 30% (7)  | 34/PR (235)   | 35% (8)  | 129/PR (1034) | 35% (8)  | 644/PR (5155)   |
| libwww-perl | libwww-perl          | -200     | 66  | 33% (22) | 40/PR (874)   | 36% (24) | 165/PR (3962) | 30% (20) | 801/PR (16012)  |
| libwww-perl | URI                  | -207     | 25  | 24% (6)  | 11/PR (64)    | 52% (13) | 234/PR (3041) | 24% (6)  | 823/PR (4940)   |
| libwww-perl | Net-HTTP             | -234     | 15  | 27% (4)  | 127/PR (509)  | 40% (6)  | 244/PR (1462) | 33% (5)  | 585/PR (2925)   |
| libwww-perl | HTTP-Message         | -347     | 28  | 29% (8)  | 101/PR (810)  | 18% (5)  | 447/PR (2236) | 54% (15) | 604/PR (9053)   |
| libwww-perl | WWW-Mechanize        | -360     | 34  | 47% (16) | 135/PR (2159) | 26% (9)  | 391/PR (3516) | 26% (9)  | 1140/PR (10256) |
'-------------+----------------------+----------+-----+----------+---------------+----------+---------------+----------+-----------------'

velocity indicates how likely a pull request is likely to get merged. You can see that WWW::Mechanize is the worst offender of the bunch, despite my minimal cleanup attempts. You can mostly ignore WWW::Mechanize::Cached for these purposes as that’s a module I’ve been actively maintaining for a lot of years.

However, you can see that the libwww-perl (LWP::UserAgent) repo, for instance takes about 874 days per pull request before that pull is merged. It takes an average of 165 days before a PR is closed and the remaining open pulls have been open for 801 days each. If you’re looking at over 2 years before the average pull request is merged, you can see how this probably isn’t encouraging people to get involved.

For my part, I’ve added a Travis CI config to all of the repositories and I’ve also converted WWW::Mechanize to use Dist::Zilla. Not all of the repositories are in a passing state, but at least now we have a baseline for passing and failing tests.

Now, I don’t have co-maint on most of a lot of the remaining modules, but I’m willing to pester people who do. People can also help by releasing TRIAL distributions so that CPAN testers can smoke the dist before we pester someone to release a module.

So, that’s my plea for today. Please feel free to pitch in and help clean this up. If you rely on these modules at your $work, please find a way to donate a few hours here and there to the upkeep of these modules.

For those of you who are bound to say “what about Mojo::UserAgent or module X”, I have two responses:

1) TIMTOWDI
2) It’s easier to maintain these modules than to update and re-release all of the CPAN which currently use them

Fortunately, I don’t know of any really terrible bugs which have gone unfixed, but I think if these modules do see active development and releases, then any terrible bugs will be easier to patch and ship if and when they do rear their ugly heads.

Edit: I neglected to mention that there is #lwp on irc.perl.org for libwww-perl discussion.

Announcing meta::hack

Posted on by oalders

Every so often, someone asks if they can donate money to MetaCPAN. I usually direct them to CPAN Testers, since (due to our generous hosting sponsors) we’ve generally not had a need for money. You can probably see where I’m going with this. Times have changed. We’re no longer turning financial sponsors away.

Back at the QA Hackathon in Rugby, we had a great group of hackers together and we got a lot of work done. However, as we worked together, it became clear that the size of our job meant that we wouldn’t be able to finish everything we had set out to do over that four day period. There are times when there’s no replacement for getting everyone in the same room together.


P4230367.jpg

The first dedicated MetaCPAN hackathon will be held at the offices of ServerCentral
in Chicago, from November 17th through 20th. The primary goal for this hackathon is to complete MetaCPAN’s transition to Elasticsearch version 2. This will enable the live service to run on a cluster of machines, greatly improving reliability and performance. The hackathon will also give the core team a chance to plan work for the coming 18 months.

The meta::hack event is a hackathon where we’re bringing together key developers to work on the MetaCPAN search engine and API. This will give core team members time to work together to complete the transition to Elasticsearch version 2, and time to discuss gnarly issues and plan the roadmap beyond the v1 upgrade.

MetaCPAN is now one of the key tools in a Perl developer’s toolbox, so supporting this event is a great way to support the Perl community and raise your company’s profile at the same time. This hackathon is by invitation only. It’s a core group of MetaCPAN hackers. We are keeping the group small in order to maintain focus on the v1 API and maximize the productivity of the group.

Why sponsor the MetaCPAN Hackathon?

 

• If your company uses Perl in any way, then your developers almost certainly use MetaCPAN to find CPAN modules, and they probably use other tools that are built on the MetaCPAN API.
• The MetaCPAN upgrade will improve the search engine and the API for all Perl developers. As a critical tool, we need it to be always available, and fast. This upgrade is a key step in that direction.
• This is a good way to establish your company as a friend of Perl, for example if you’re hiring.

Participants

 

There will be 8 people taking part, including me. Everyone taking part is an experienced senior-level software engineer, and most of them have already spent a lot of time working on MetaCPAN. As noted above, this is an invitational event with a very specific focus.

What is meta::hack?

 

MetaCPAN was created in late 2010. Version 0 of the MetaCPAN API was built on a very early version of Elasticsearch. For the first 5 years, most of the work on MetaCPAN focussed on improving the data coverage, and the web interface. In that time Elasticsearch has moved on, and we’re now well behind.

The work to upgrade Elasticsearch began in May of 2014. It continued in early Feb of 2015. Later, at the 2015 QA Hackathon in Berlin, Clinton Gormley (who works for Elastic) and I worked on moving MetaCPAN to Elasticsearch version 2. This work was continued at the 2016 QA Hackathon in Rugby, and as a result we now have a beta version in live usage.

The primary goal of meta::hack is to complete the port to Elasticsearch version 2, so the public API and search engine can be switched over. There are a number of benefits:

• Switching from a single server to a cluster of 3 servers, giving a more reliable service and improved performance.
• Once we decommission the old service, we’ll be able to set up a second cluster of 3 machines in a second data centre, for further improvements.
• We’ll be able to take advantage of new Elasticsearch features, like search suggesters.
• We’ll be able to use a new endpoint that has been developed specifically to speed up cpanminus lookups. Cpanminus is probably the most widely used CPAN client these days, so improving this will benefit a large percentage of the community.
• If and when search.cpan.org is decommissioned, we’ll be able to handle the extra traffic that will bring with it, and we’ll also have the redundancy to do this safely.
• We’ll be able to shift focus back to bug fixes and new MetaCPAN features.

Becoming a Sponsor

 

Neil Bowers has kindly taken on the task of shepherding the sponsorship process.  (He also wrote the sponsorship prospectus from which I cribbed most of this post.) Please contact Neil or contact me for a copy of the meta::hack sponsorship prospectus.  It contains most of the information listed above as well as the various available sponsorship levels which are available.  Thank you for your help in making this event happen.  We’re looking forward to getting the key people together in one room again and making this already useful tool even better.

Getting to Travis and GitHub Pages Quickly

Posted on by oalders

Disclaimer: I’m sure this functionality exists elsewhere, but this was a fun little thing for me to work on. Also, you’ll need a minimum of git 2.7 for this to work.

Often, when I’m working locally I like to bounce right over to a GitHub repository url to check something. I ended up writing a bit of code to make this easier. While I was at it, I decided it would be nice to have the same thing for Travis URLs. So, I’ve released this as part of Git::Helpers.

When you’re inside a Git repository, you can use gh-open to open a browser window with the GitHub URL of your repository. gh-open also accepts an origin name as an argument, so 

gh-open upstream

would open a tab in your default browser containing your upstream’s URL, assuming you have an origin by that name. Don’t specify a remote name and it will assume origin: 

gh-open

It doesn’t currently care which branch you’re on, but patches welcome (in the kindest sense of the expression).

If you want to check your Travis page for the repository then travis-open will do the same kind of thing. It also accepts an origin name, just as gh-open does: 

travis-open upstream

or defaults to origin if you don’t: 

travis-open

Don’t Forget about URI::Heuristic

Posted on by oalders

Imagine you’ve got some user input that is supposed to be a valid URL, but it’s user input, so you can’t be sure of anything. It’s not very consistent data, so you at least make sure to prepend a default scheme to it. It’s a fairly common case. Sometimes I see it solved this way:

my $url = 'example.com';
$url = 'http://' . $url unless $url =~ m{http://}i;

This converts example.com to http://example.com, but it can be error prone. For instance, what if I forgot to make the regex case insensitive? Actually, I’ve already made a mistake. Did you spot it? In my haste I’ve neglected to deal with https URLs. Not good. URI::Heuristic can help here.

use URI::Heuristic qw(uf_uristr);
my $url = 'example.com';
$url    = uf_uristr( $url );

This does exactly the same thing as the example above, but I’ve left the logic of checking for an existing scheme to the URI::Heuristic module. If you like this approach, but you’d rather get a URI object back then try this:

use URI::Heuristic qw(uf_uri);
my $url = 'example.com';
$url    = uf_uristr( $url );
say $url->as_string;

Caveats

use URI::Heuristic qw(uf_uri);
my $url = uf_uri('/etc/passwd');      # file:/etc/passwd

Are we sure this is what we want? Checking the scheme is helpful and even if we weren’t using this module, we’d probably want to do this anyway.

use List::AllUtils qw( any );
use URI::Heuristic qw(uf_uri);

my $url = uf_uri('/etc/passwd');
unless ( $url->scheme && any { $url->scheme eq $_ } ('http', 'https') ) {
    die 'unsupported scheme: ' . $url->scheme;
}

That’s it! This module has been around for almost 18 years now, but it still solves some of today’s problems.