UserAgent Debugging Made Easy

Posted on September 29, 2016March 23, 2017 by oalders

Earlier today I saw a recent blog post from Gabor Szabo. In it, he shows a very concise way to handle Basic Authentication using LWP::UserAgent. Now, what if you had a problem running the script? How might you go about debugging it? You could add a bunch of print statements. Maybe dump the request and the response objects. That’s entirely valid, but I want to show you a slightly simpler way of going about it, using LWP::ConsoleLogger::Easy.

Gabor’s original script looks like this:

use strict;
use warnings;
use v5.10;
use LWP::UserAgent;
use HTTP::Request::Common;
 
my $ua = LWP::UserAgent->new();
my $request = GET 'https://pause.perl.org/pause/authenquery';
 
$request->authorization_basic('szabgab', '*******');
 
my $response = $ua->request($request);
say $response->as_string();

Let’s run it to see what the output looks like.

olaf$ perl gabor.pl
HTTP/1.0 401 Unauthorized
Connection: close
Date: Thu, 29 Sep 2016 02:07:34 GMT
WWW-Authenticate: Basic realm="PAUSE"
Content-Length: 22
Content-Type: text/plain
Client-Date: Thu, 29 Sep 2016 02:07:34 GMT
Client-Peer: 207.171.7.119:443
Client-Response-Num: 1
Client-SSL-Cert-Issuer: /C=US/O=GeoTrust, Inc./CN=RapidSSL CA
Client-SSL-Cert-Subject: /serialNumber=K6R2GP4nh37grllJm9PZlQm0SS-oSNZ4/C=DE/O=pause.perl.org/OU=GT38611495/OU=See www.rapidssl.com/resources/cps (c)11/OU=Domain Control Validated - RapidSSL(R)/CN=pause.perl.org
Client-SSL-Cipher: AES128-GCM-SHA256
Client-SSL-Socket-Class: IO::Socket::SSL

Authorization required

Here’s the debugging version. Note the important changes are on lines 4 and 9.

use strict;
use warnings;
use v5.10;
use LWP::ConsoleLogger::Easy qw( debug_ua );
use LWP::UserAgent;
use HTTP::Request::Common;

my $ua      = LWP::UserAgent->new();
debug_ua( $ua );
my $request = GET 'https://pause.perl.org/pause/authenquery';

$request->authorization_basic( 'szabgab', '*******' );

my $response = $ua->request($request);

The output we get is:

 olaf$ perl basic-authentication.pl
GET https://pause.perl.org/pause/authenquery

.----------------+----------------------------.
| Request Header | Value                      |
+----------------+----------------------------+
| Authorization  | Basic c3phYmdhYjoqKioqKioq |
| User-Agent     | libwww-perl/6.15           |
'----------------+----------------------------'

==> 401 Unauthorized

.-------------------------+-----------------------------------------------------------------------------------------.
| Response Header         | Value                                                                                   |
+-------------------------+-----------------------------------------------------------------------------------------+
| Client-Date             | Thu, 29 Sep 2016 01:54:23 GMT                                                           |
| Client-Peer             | 207.171.7.119:443                                                                       |
| Client-Response-Num     | 1                                                                                       |
| Client-SSL-Cert-Issuer  | /C=US/O=GeoTrust, Inc./CN=RapidSSL CA                                                   |
| Client-SSL-Cert-Subject | /serialNumber=K6R2GP4nh37grllJm9PZlQm0SS-oSNZ4/C=DE/O=pause.perl.org/OU=GT38611495/OU=- |
|                         | See www.rapidssl.com/resources/cps (c)11/OU=Domain Control Validated - RapidSSL(R)/CN=- |
|                         | pause.perl.org                                                                          |
| Client-SSL-Cipher       | AES128-GCM-SHA256                                                                       |
| Client-SSL-Socket-Class | IO::Socket::SSL                                                                         |
| Connection              | close                                                                                   |
| Content-Length          | 22                                                                                      |
| Content-Type            | text/plain                                                                              |
| Date                    | Thu, 29 Sep 2016 01:54:23 GMT                                                           |
| WWW-Authenticate        | Basic realm="PAUSE"                                                                     |
'-------------------------+-----------------------------------------------------------------------------------------'

.------------------------.
| Content                |
+------------------------+
| Authorization required |
'------------------------'

.------------------------.
| Text                   |
+------------------------+
| Authorization required |
'------------------------'

You can see that the debugging version is just one line longer. I added 2 lines and removed a print statement. It prints out a whole pile of (nicely?) formatted information. Let’s try running it with valid credentials. (Brace yourself, there’s going to be a lot of output.)

olaf$ LWPCL_REDACT_HEADERS='Authorization' perl basic-authentication.pl
GET https://pause.perl.org/pause/authenquery

.----------------+------------------.
| Request Header | Value            |
+----------------+------------------+
| Authorization  | [REDACTED]       |
| User-Agent     | libwww-perl/6.15 |
'----------------+------------------'

==> 200 OK

.-------------------------+-----------------------------------------------------------------------------------------.
| Response Header         | Value                                                                                   |
+-------------------------+-----------------------------------------------------------------------------------------+
| Cache-Control           | no-cache                                                                                |
| Client-Date             | Thu, 29 Sep 2016 02:06:02 GMT                                                           |
| Client-Peer             | 207.171.7.119:443                                                                       |
| Client-Response-Num     | 1                                                                                       |
| Client-SSL-Cert-Issuer  | /C=US/O=GeoTrust, Inc./CN=RapidSSL CA                                                   |
| Client-SSL-Cert-Subject | /serialNumber=K6R2GP4nh37grllJm9PZlQm0SS-oSNZ4/C=DE/O=pause.perl.org/OU=GT38611495/OU=- |
|                         | See www.rapidssl.com/resources/cps (c)11/OU=Domain Control Validated - RapidSSL(R)/CN=- |
|                         | pause.perl.org                                                                          |
| Client-SSL-Cipher       | AES128-GCM-SHA256                                                                       |
| Client-SSL-Socket-Class | IO::Socket::SSL                                                                         |
| Connection              | close                                                                                   |
| Content-Length          | 11251                                                                                   |
| Content-Type            | text/html; charset=utf-8                                                                |
| Date                    | Thu, 29 Sep 2016 02:06:02 GMT                                                           |
| Last-Modified           | Thu, 29 Sep 2016 02:06:02 GMT                                                           |
| Link                    | ; rel="shortcut icon"; type="image/jpeg", - |
|                         | ; rel="stylesheet"; title="pause"; type="text/css"                                      |
| Pragma                  | no-cache                                                                                |
| Title                   | PAUSE: menu                                                                             |
| Vary                    | accept-encoding                                                                         |
'-------------------------+-----------------------------------------------------------------------------------------'

.-------------------------------------------------------------------------------------------------------------------.
| Content                                                                                                           |
+-------------------------------------------------------------------------------------------------------------------+
| PAUSE: menu                                                                            |
|                                     |
|                                     |
| The [Perl programming] Authors Upload                                  |
|   Server OALDERS <[email protected]>
encrypted session     |
| 

   |
|                                                                                        |
|                                                                     |
|                                                                                          |
|                                                                                    |
|                                                                                |
|                                                                                     |
|      |
|                                                                                    |
|                                                                                         |
|                                                                                |
|                                                                                        |
|                                                                  |
|                                                                                      |
|                                                                                   |
|                                                                        |
|                                                                                         |
|                                                                                      |
| - |
|                                                                                 |
|                                                                      |
|                                                                                      |
|                                                                                    |
|                                                                                   |
| Public menu
Request PAUSE account 
About PAUSE 
On The Naming of Modu- |
| les 
PAUSE News 
PAUSE History
Imprint/Impressum 
List of pumpkins 
User menu
Files
Upload a file to CPAN
Show my files 
Repair a Pending Upload 
Delete Files 
Permissions
View Permissions 
Change Permissions
Utils
Force Reindexing 
Reset Version 
Tail Daemon Logfile 
Account
Edit Account Info 
Change Password
About Logging Out                                                                                                          |
|                                                                                        |
| Hi Olaf Alders,
please choose an action from the menu.                          |
| The usermenu to the left shows all menus available to                                                          |
|     you, the table below shows descriptions for all menues available                                              |
|     to anybody on PAUSE.                                                                                      |
|                                         |
|                                                                                            |
|  |
|                                                                                            |
|                                                                                             |
|                                                                                            |
|                                                                                                    |
|                                                                                            |
|                                                                         |
|                                                                                            |
|             |
|                                                                                            |
|                       |
|                                                                                            |
|                        |
|                                                                                            |
|  |
|                                                                                            |
|                                                                                               |
|                                                                                            |
|                |
|                                                                                            |
|                                                                           |
|                                                                                            |
|                                                                            |
|                                                                                            |
|                                                                                                   |
|                                                                                            |
|                                                       |
|                                                                                            |
|                                        |
|                                                                                            |
|                                                    |
|                                                                                            |
|                           |
|                                                                                            |
|                                                                   |
|                                                                                            |
|                                                                                            |
|                                                                                            |
|                             |
|                                                                                            |
|                                                                                        |
|                                                                                            |
|                                                                                |
|                                                                                            |
|                                                                                      |
|                                                                                            |
|                                                                                       |
|                                                                                            |
|                                                                               |
|                                                                                            |
|                                                                                               |
|                                                                                            |
|                                                                 |
|                                                                                            |
|                                                                                                                |
|                                                                                            |
|                                                                                                    |
|                                                                                            |
|                                                                                                          |
|                                                                                            |
|                                                                                      |
|                                                                                            |
|                                         |
| Action Group Description
Request PAUSE account public Apply for a PAUSE account.
Forgot Password? public A passwordmailer that sends you a             |
| 			password that enables you to set a new                                                                         |
| 			password.
About PAUSE public Same as modules/04pause.html on any CPAN s- |
| erver
On The Naming of Modules public A couple of suggestions- |
|  that hopefully get you on track
PAUSE News public What's going on on PAUSE
PAUSE History public Old News
Imprint/Impressum public N/A
List of pumpkins public A list, also available as YAML
Upload a file to CPAN user This is the heart of the Upload        |
| 			 Server, the page most heavily used on                                                                     |
| 			 PAUSE.
Show my files user find . -ls resemblance
Repair a Pending Upload user When an upload you requested hangs    |
| 			   for some reason, you can go here and edit                                                                   |
| 			   the file to be uploaded.
Delete Files user Schedule files for deletion.                  |
| 			      There is a delay until the deletion                                                                      |
| 			      really happens. Until then you can also                                                                  |
| 			      undelete files here.
View Permissions user Whose uploads of what are being indexed on- |
|  PAUSE
Change Permissions user Enable other users to upload a           |
|                              module for any of your namespaces, manage                                            |
|                              your own permissions.
Force Reindexing user Tell the indexer to index a file again         |
|                          (e.g. after a change in the perms table)
Reset Version user Overrule the record of the current          |
|                            version number of a module that the indexer                                            |
|                            uses and set it to 'undef'
Tail Daemon Logfile user N/A
Edit Account Info user Edit your user name, your email             |
| 			   addresses (both public and secret one),                                                                     |
| 			   change the URL of your homepage.
Change Password user Change your password any time             |
| 				you want.
About Logging Out user N/A
Select Mailinglist/Action mlrepr Representatives of mailing |
| 					  lists have their special                                                                                   |
| 					  menu here.
Show Mailinglist Reps mlrepr Admins and the representatives th- |
| emselves                                                                                                          |
|                                       can lookup who is                                                           |
| 				      elected to be representative of                                                                         |
| 				      a mailing list.
Add a User or Mailinglist admin Admins can add users or             |
| 				  mailinglists.
Look up the forward email address admin Admins can look whe- |
| re email should go
Manage a registration request (alpha) admin show/reject - |
| open registration requests
Edit a Mailinglist admin Admins and mailing list                     |
| 				 representatives can change the name,                                                                         |
| 				 address and description of a mailing                                                                         |
| 				 list.
Select User/Action admin Admins can access PAUSE as-if           |
| 				     they were somebody else. Here                                                                            |
| 				     they select a user/action pair.
Post a message admin Post a message to a specific user. 
Show/Delete Msgs admin Delete your messages from the message b- |
| oard.
Index users with digrams (BROKEN) admin Batch-index all users.<- |
| /td>
Show bad xhtml output admin Monitor bad xhtml output stored fro- |
| m previous sessions
coredump admin N/A                                                                                                          |
|                                                                                                 |
|                                                                                                 |
| 
                                                               |
|                                                                                                               |
|                                                                                                    |
|                         |
|            |
|                                                                                                              |
|                                                                                                               |
|                                                                                             |
|                                                                                                              |
|                                                                                                              |
|                                                                                                              |
|        Rev: 1071.02 To validate, download page first.

                                                                                                       |
|                                                                  |
|                                                                                                               |
|                                                                |
|                                                                 |
|                                                                                                               |
|                                                                                                             |
|                                                                                                     |
'-------------------------------------------------------------------------------------------------------------------'

.-------------------------------------------------------------------------------------------------------------------.
| Text                                                                                                              |
+-------------------------------------------------------------------------------------------------------------------+
| PAUSE: menu The [Perl programming] Authors Upload ServerOALDERS <[email protected]>encrypted sessio- |
| n Public menuRequest PAUSE account About PAUSE On The Naming of Modules PAUSE News PAUSE History Imprint/Impress- |
| um List of pumpkins User menuFiles Upload a file to CPAN Show my files Repair a Pending Upload Delete Files Perm- |
| issions View Permissions Change Permissions Utils Force Reindexing Reset Version Tail Daemon Logfile Account Edi- |
| t Account Info Change Password About Logging Out   Hi Olaf Alders,please choose an action from the menu. Th- |
| e usermenu to the left shows all menus available to you, the table below shows descriptions for all menues avail- |
| able to anybody on PAUSE. ActionGroupDescription Request PAUSE accountpublicApply for a PAUSE account. Forgot Pa- |
| ssword?publicA passwordmailer that sends you a password that enables you to set a new password. About PAUSEpubli- |
| cSame as modules/04pause.html on any CPAN server On The Naming of ModulespublicA couple of suggestions that hope- |
| fully get you on track PAUSE NewspublicWhat's going on on PAUSE PAUSE HistorypublicOld News Imprint/Impressumpub- |
| licN/A List of pumpkinspublicA list, also available as YAML Upload a file to CPANuserThis is the heart of the Up- |
| load Server, the page most heavily used on PAUSE. Show my filesuserfind . -ls resemblance Repair a Pending Uploa- |
| duserWhen an upload you requested hangs for some reason, you can go here and edit the file to be uploaded. Delet- |
| e FilesuserSchedule files for deletion. There is a delay until the deletion really happens. Until then you can a- |
| lso undelete files here. View PermissionsuserWhose uploads of what are being indexed on PAUSE Change Permissions- |
| userEnable other users to upload a module for any of your namespaces, manage your own permissions. Force Reindex- |
| inguserTell the indexer to index a file again (e.g. after a change in the perms table) Reset VersionuserOverrule- |
|  the record of the current version number of a module that the indexer uses and set it to 'undef' Tail Daemon Lo- |
| gfileuserN/A Edit Account InfouserEdit your user name, your email addresses (both public and secret one), change- |
|  the URL of your homepage. Change PassworduserChange your password any time you want. About Logging OutuserN/A S- |
| elect Mailinglist/ActionmlreprRepresentatives of mailing lists have their special menu here. Show Mailinglist Re- |
| psmlreprAdmins and the representatives themselves can lookup who is elected to be representative of a mailing li- |
| st. Add a User or MailinglistadminAdmins can add users or mailinglists. Look up the forward email addressadminAd- |
| mins can look where email should go Manage a registration request (alpha)adminshow/reject open registration requ- |
| ests Edit a MailinglistadminAdmins and mailing list representatives can change the name, address and description- |
|  of a mailing list. Select User/ActionadminAdmins can access PAUSE as-if they were somebody else. Here they sele- |
| ct a user/action pair. Post a messageadminPost a message to a specific user. Show/Delete MsgsadminDelete your me- |
| ssages from the message board. Index users with digrams (BROKEN)adminBatch-index all users. Show bad xhtml outpu- |
| tadminMonitor bad xhtml output stored from previous sessions coredumpadminN/A   Rev: 1071.02 To validate, d- |
| ownload page first.                                                                                               |
'-------------------------------------------------------------------------------------------------------------------'

You can see that I ran the script with LWPCL_REDACT_HEADERS='Authorization'. That’s a handy flag to use if you want to copy/paste an example when asking for help publicly. It replaced the Authorization header value with [REDACTED]. That’s maybe not a big deal here, but there are cases where it’s more important. See also LWP_REDACT_PARAMS.

Let’s make it prettier. We’ll do this by installing HTML::FormatText::Lynx.

Let’s run it again. I’ll only show you the changed part. Instead of just displaying the text with the HTML stripped away, we get something nicer to look at.

.------------------------------------------------------------------------------------------------------------------------------------------------------.
| Text                                                                                                                                                 |
+------------------------------------------------------------------------------------------------------------------------------------------------------+
| [1]PAUSE Logo                                                                                                                                        |
|                                                                                                                                                      |
| The [Perl programming] Authors Upload Server                                                                                                         |
|                                                                                                                                                      |
|    OALDERS                                                                                                                 |
|    encrypted session                                                                                                                                 |
|                                                                                                                                                      |
|    Public menu                                                                                                                                       |
|    [2]Request PAUSE account                                                                                                                          |
|    [3]About PAUSE                                                                                                                                    |
|    [4]On The Naming of Modules                                                                                                                       |
|    [5]PAUSE News                                                                                                                                     |
|    [6]PAUSE History                                                                                                                                  |
|    [7]Imprint/Impressum                                                                                                                              |
|    [8]List of pumpkins                                                                                                                               |
|    User menu                                                                                                                                         |
|    Files                                                                                                                                             |
|    [9]Upload a file to CPAN                                                                                                                          |
|    [10]Show my files                                                                                                                                 |
|    [11]Repair a Pending Upload                                                                                                                       |
|    [12]Delete Files                                                                                                                                  |
|    Permissions                                                                                                                                       |
|    [13]View Permissions                                                                                                                              |
|    [14]Change Permissions                                                                                                                            |
|    Utils                                                                                                                                             |
|    [15]Force Reindexing                                                                                                                              |
|    [16]Reset Version                                                                                                                                 |
|    [17]Tail Daemon Logfile                                                                                                                           |
|    Account                                                                                                                                           |
|    [18]Edit Account Info                                                                                                                             |
|    [19]Change Password                                                                                                                               |
|    [20]About Logging Out                                                                                                                             |
|                                                                                                                                                      |
|                                                                                                                                                      |
| Hi Olaf Alders,                                                                                                                                      |
| please choose an action from the menu.                                                                                                               |
|                                                                                                                                                      |
|    The usermenu to the left shows all menus available to you, the table                                                                              |
|    below shows descriptions for all menues available to anybody on PAUSE.                                                                            |
|    Action Group Description                                                                                                                          |
|    Request PAUSE account public Apply for a PAUSE account.                                                                                           |
|    Forgot Password? public A passwordmailer that sends you a password that                                                                           |
|    enables you to set a new password.                                                                                                                |
|    About PAUSE public Same as modules/04pause.html on any CPAN server                                                                                |
|    On The Naming of Modules public A couple of suggestions that hopefully                                                                            |
|    get you on track                                                                                                                                  |
|    PAUSE News public What's going on on PAUSE                                                                                                        |
|    PAUSE History public Old News                                                                                                                     |
|    Imprint/Impressum public N/A                                                                                                                      |
|    List of pumpkins public A list, also available as YAML                                                                                            |
|    Upload a file to CPAN user This is the heart of the Upload Server, the                                                                            |
|    page most heavily used on PAUSE.                                                                                                                  |
|    Show my files user find . -ls resemblance                                                                                                         |
|    Repair a Pending Upload user When an upload you requested hangs for                                                                               |
|    some reason, you can go here and edit the file to be uploaded.                                                                                    |
|    Delete Files user Schedule files for deletion. There is a delay until                                                                             |
|    the deletion really happens. Until then you can also undelete files                                                                               |
|    here.                                                                                                                                             |
|    View Permissions user Whose uploads of what are being indexed on PAUSE                                                                            |
|    Change Permissions user Enable other users to upload a module for any                                                                             |
|    of your namespaces, manage your own permissions.                                                                                                  |
|    Force Reindexing user Tell the indexer to index a file again (e.g.                                                                                |
|    after a change in the perms table)                                                                                                                |
|    Reset Version user Overrule the record of the current version number of                                                                           |
|    a module that the indexer uses and set it to 'undef'                                                                                              |
|    Tail Daemon Logfile user N/A                                                                                                                      |
|    Edit Account Info user Edit your user name, your email addresses (both                                                                            |
|    public and secret one), change the URL of your homepage.                                                                                          |
|    Change Password user Change your password any time you want.                                                                                      |
|    About Logging Out user N/A                                                                                                                        |
|    Select Mailinglist/Action mlrepr Representatives of mailing lists have                                                                            |
|    their special menu here.                                                                                                                          |
|    Show Mailinglist Reps mlrepr Admins and the representatives themselves                                                                            |
|    can lookup who is elected to be representative of a mailing list.                                                                                 |
|    Add a User or Mailinglist admin Admins can add users or mailinglists.                                                                             |
|    Look up the forward email address admin Admins can look where email                                                                               |
|    should go                                                                                                                                         |
|    Manage a registration request (alpha) admin show/reject open                                                                                      |
|    registration requests                                                                                                                             |
|    Edit a Mailinglist admin Admins and mailing list representatives can                                                                              |
|    change the name, address and description of a mailing list.                                                                                       |
|    Select User/Action admin Admins can access PAUSE as-if they were                                                                                  |
|    somebody else. Here they select a user/action pair.                                                                                               |
|    Post a message admin Post a message to a specific user.                                                                                           |
|    Show/Delete Msgs admin Delete your messages from the message board.                                                                               |
|    Index users with digrams (BROKEN) admin Batch-index all users.                                                                                    |
|    Show bad xhtml output admin Monitor bad xhtml output stored from                                                                                  |
|    previous sessions                                                                                                                                 |
|    coredump admin N/A                                                                                                                                |
|      __________________________________________________________________                                                                              |
|                                                                                                                                                      |
|                                                                                                                                                      |
|    Rev: 1071.02                                                                                                                                      |
|                                                                                                                                                      |
|                                          To validate, download page first.                                                                           |
|                                                                                                                                                      |
|      [21]Valid CSS! [22]Valid XHTML 1.0!                                                                                                             |
|                                                                                                                                                      |
| References                                                                                                                                           |
|                                                                                                                                                      |
|    1. https://pause.perl.org/pause/authenquery                                                                                                       |
|    2. https://pause.perl.org/pause/authenquery?ACTION=request_id                                                                                     |
|    3. https://pause.perl.org/pause/authenquery?ACTION=pause_04about                                                                                  |
|    4. https://pause.perl.org/pause/authenquery?ACTION=pause_namingmodules                                                                            |
|    5. https://pause.perl.org/pause/authenquery?ACTION=pause_05news                                                                                   |
|    6. https://pause.perl.org/pause/authenquery?ACTION=pause_06history                                                                                |
|    7. https://pause.perl.org/pause/authenquery?ACTION=pause_04imprint                                                                                |
|    8. https://pause.perl.org/pause/authenquery?ACTION=who_pumpkin                                                                                    |
|    9. https://pause.perl.org/pause/authenquery?ACTION=add_uri                                                                                        |
|   10. https://pause.perl.org/pause/authenquery?ACTION=show_files                                                                                     |
|   11. https://pause.perl.org/pause/authenquery?ACTION=edit_uris                                                                                      |
|   12. https://pause.perl.org/pause/authenquery?ACTION=delete_files                                                                                   |
|   13. https://pause.perl.org/pause/authenquery?ACTION=peek_perms                                                                                     |
|   14. https://pause.perl.org/pause/authenquery?ACTION=share_perms                                                                                    |
|   15. https://pause.perl.org/pause/authenquery?ACTION=reindex                                                                                        |
|   16. https://pause.perl.org/pause/authenquery?ACTION=reset_version                                                                                  |
|   17. https://pause.perl.org/pause/authenquery?ACTION=tail_logfile                                                                                   |
|   18. https://pause.perl.org/pause/authenquery?ACTION=edit_cred                                                                                      |
|   19. https://pause.perl.org/pause/authenquery?ACTION=change_passwd                                                                                  |
|   20. https://pause.perl.org/pause/authenquery?ACTION=pause_logout                                                                                   |
|   21. http://jigsaw.w3.org/css-validator/                                                                                                            |
|   22. http://validator.w3.org/file-upload.html                                                                                                       |
'------------------------------------------------------------------------------------------------------------------------------------------------------'

Now, we can also turn down the verbosity of the script by passing a flag to debug_ua(). Any integer from 0-8 will do the trick. Let’s try 6.

use strict;
use warnings;
use v5.10;
use LWP::ConsoleLogger::Easy qw( debug_ua );
use LWP::UserAgent;
use HTTP::Request::Common;

my $ua      = LWP::UserAgent->new();
debug_ua( $ua, 6 );
my $request = GET 'https://pause.perl.org/pause/authenquery';

$request->authorization_basic( 'oalders', 'seekrit' );

my $response = $ua->request($request);

Let’s see what we get:

 olaf$ LWPCL_REDACT_HEADERS='Authorization' perl basic-authentication.pl
GET https://pause.perl.org/pause/authenquery

.----------------+------------------.
| Request Header | Value            |
+----------------+------------------+
| Authorization  | [REDACTED]       |
| User-Agent     | libwww-perl/6.15 |
'----------------+------------------'

==> 200 OK

.-------------------------+----------------------------------------------------------------------------------------------------------------------------.
| Response Header         | Value                                                                                                                      |
+-------------------------+----------------------------------------------------------------------------------------------------------------------------+
| Cache-Control           | no-cache                                                                                                                   |
| Client-Date             | Thu, 29 Sep 2016 02:13:51 GMT                                                                                              |
| Client-Peer             | 207.171.7.119:443                                                                                                          |
| Client-Response-Num     | 1                                                                                                                          |
| Client-SSL-Cert-Issuer  | /C=US/O=GeoTrust, Inc./CN=RapidSSL CA                                                                                      |
| Client-SSL-Cert-Subject | /serialNumber=K6R2GP4nh37grllJm9PZlQm0SS-oSNZ4/C=DE/O=pause.perl.org/OU=GT38611495/OU=See www.rapidssl.com/resources/cps - |
|                         | (c)11/OU=Domain Control Validated - RapidSSL(R)/CN=pause.perl.org                                                          |
| Client-SSL-Cipher       | AES128-GCM-SHA256                                                                                                          |
| Client-SSL-Socket-Class | IO::Socket::SSL                                                                                                            |
| Connection              | close                                                                                                                      |
| Content-Length          | 11251                                                                                                                      |
| Content-Type            | text/html; charset=utf-8                                                                                                   |
| Date                    | Thu, 29 Sep 2016 02:13:51 GMT                                                                                              |
| Last-Modified           | Thu, 29 Sep 2016 02:13:51 GMT                                                                                              |
| Link                    | ; rel="shortcut icon"; type="image/jpeg", ; rel="stylesheet"; title="pause"; - |
|                         | type="text/css"                                                                                                            |
| Pragma                  | no-cache                                                                                                                   |
| Title                   | PAUSE: menu                                                                                                                |
| Vary                    | accept-encoding                                                                                                            |
'-------------------------+----------------------------------------------------------------------------------------------------------------------------'

That’s far easier to read now.

This is just a very basic example of what you can do with LWP::ConsoleLogger::Easy. There’s a lot more you can do with it and it’s all laid out for you in the documentation. It really shines when you have a user agent which is going through multiple links or if you’re debugging someone else’s API calls. Have fun with it. It beats inserting arbitrary print statements and it could save you from pulling a lot of your own hair out someday.

Make libwww-perl Great Again ™

Posted on September 26, 2016September 26, 2016 by oalders

You may have noticed that WWW::Mechanize has seen some releases over the last couple of months. No big, breaking changes, but bugs have been fixed and enhancements have been shipped. This module is part of the libwww-perl ecosystem and also a part of the libwww-perl GitHub organization, to which I now also belong. I started pestering people to get involved because these modules, although quite important in the CPAN scheme of things, aren’t really on a regular release cycle. I don’t have the backstory on everything and this is not a complaint about anybody who has commit bits, maint or co-maint. It’s just an observation that a lot of modules on CPAN depend on the modules in this organization. The issue queues are slowly growing and pull requests are going unmerged.

I think there’s a fairly simple solution to all of this and my hope is that we can crowdsource enough mindshare to get this done. (I’m hoping that previous sentence is fully buzzword compliant).

Now, I don’t have a lot of hours of spare time to devote to this stuff in any given week, but this doesn’t all fall to me anyway. What I’d like to see is more eyeballs on the code. If you’d like to get involved or you have an interest in seeing things move along with any of these modules, please go through any outstanding pull requests and issues. Even comments such as “LGTM” (looks good to me) are very helpful. If enough people who know what they’re doing stamp a “LGTM” on a pull request, then that signals that this code is less risky to merge. If people can look into open issues and identify what is or is not a bug and what is or is not RFC-compliant, then that can speed up the issue review cycle as well.

If you’d like to join the libwww-perl org, then that would be great as well. Probably a good first step for that would be to get involved with reviewing open pull requests and issues or even contributing some code.

Here’s a quick summary of the repositories which are currently in the org:

olaf$ github-mergevelocity --url libwww-perl/WWW-Mechanize --url libwww-perl/libwww-perl --url libwww-perl/URI --url libwww-perl/Net-HTTP --url libwww-perl/HTTP-Message --url libwww-perl/LWP-Protocol-https --url libwww-perl/www-mechanize-cached
.-------------+----------------------+----------+-----+----------+---------------+----------+---------------+----------+-----------------.
| user        | repo                 | velocity | PRs | merged   | merge days    | closed   | close days    | open     | open days       |
+-------------+----------------------+----------+-----+----------+---------------+----------+---------------+----------+-----------------+
| libwww-perl | www-mechanize-cached | 157      | 10  | 80% (8)  | 6/PR (45)     | 0        | 0             | 20% (2)  | 1/PR (2)        |
| libwww-perl | LWP-Protocol-https   | -159     | 23  | 30% (7)  | 34/PR (235)   | 35% (8)  | 129/PR (1034) | 35% (8)  | 644/PR (5155)   |
| libwww-perl | libwww-perl          | -200     | 66  | 33% (22) | 40/PR (874)   | 36% (24) | 165/PR (3962) | 30% (20) | 801/PR (16012)  |
| libwww-perl | URI                  | -207     | 25  | 24% (6)  | 11/PR (64)    | 52% (13) | 234/PR (3041) | 24% (6)  | 823/PR (4940)   |
| libwww-perl | Net-HTTP             | -234     | 15  | 27% (4)  | 127/PR (509)  | 40% (6)  | 244/PR (1462) | 33% (5)  | 585/PR (2925)   |
| libwww-perl | HTTP-Message         | -347     | 28  | 29% (8)  | 101/PR (810)  | 18% (5)  | 447/PR (2236) | 54% (15) | 604/PR (9053)   |
| libwww-perl | WWW-Mechanize        | -360     | 34  | 47% (16) | 135/PR (2159) | 26% (9)  | 391/PR (3516) | 26% (9)  | 1140/PR (10256) |
'-------------+----------------------+----------+-----+----------+---------------+----------+---------------+----------+-----------------'

velocity indicates how likely a pull request is likely to get merged. You can see that WWW::Mechanize is the worst offender of the bunch, despite my minimal cleanup attempts. You can mostly ignore WWW::Mechanize::Cached for these purposes as that’s a module I’ve been actively maintaining for a lot of years.

However, you can see that the libwww-perl (LWP::UserAgent) repo, for instance takes about 874 days per pull request before that pull is merged. It takes an average of 165 days before a PR is closed and the remaining open pulls have been open for 801 days each. If you’re looking at over 2 years before the average pull request is merged, you can see how this probably isn’t encouraging people to get involved.

For my part, I’ve added a Travis CI config to all of the repositories and I’ve also converted WWW::Mechanize to use Dist::Zilla. Not all of the repositories are in a passing state, but at least now we have a baseline for passing and failing tests.

Now, I don’t have co-maint on most of a lot of the remaining modules, but I’m willing to pester people who do. People can also help by releasing TRIAL distributions so that CPAN testers can smoke the dist before we pester someone to release a module.

So, that’s my plea for today. Please feel free to pitch in and help clean this up. If you rely on these modules at your $work, please find a way to donate a few hours here and there to the upkeep of these modules.

For those of you who are bound to say “what about Mojo::UserAgent or module X”, I have two responses:

1) TIMTOWDI
2) It’s easier to maintain these modules than to update and re-release all of the CPAN which currently use them

Fortunately, I don’t know of any really terrible bugs which have gone unfixed, but I think if these modules do see active development and releases, then any terrible bugs will be easier to patch and ship if and when they do rear their ugly heads.

Edit: I neglected to mention that there is #lwp on irc.perl.org for libwww-perl discussion.

Announcing meta::hack

Posted on September 17, 2016September 19, 2016 by oalders

Every so often, someone asks if they can donate money to MetaCPAN. I usually direct them to CPAN Testers, since (due to our generous hosting sponsors) we’ve generally not had a need for money. You can probably see where I’m going with this. Times have changed. We’re no longer turning financial sponsors away.

Back at the QA Hackathon in Rugby, we had a great group of hackers together and we got a lot of work done. However, as we worked together, it became clear that the size of our job meant that we wouldn’t be able to finish everything we had set out to do over that four day period. There are times when there’s no replacement for getting everyone in the same room together.

The first dedicated MetaCPAN hackathon will be held at the offices of ServerCentral
in Chicago, from November 17th through 20th. The primary goal for this hackathon is to complete MetaCPAN’s transition to Elasticsearch version 2. This will enable the live service to run on a cluster of machines, greatly improving reliability and performance. The hackathon will also give the core team a chance to plan work for the coming 18 months.

The meta::hack event is a hackathon where we’re bringing together key developers to work on the MetaCPAN search engine and API. This will give core team members time to work together to complete the transition to Elasticsearch version 2, and time to discuss gnarly issues and plan the roadmap beyond the v1 upgrade.

MetaCPAN is now one of the key tools in a Perl developer’s toolbox, so supporting this event is a great way to support the Perl community and raise your company’s profile at the same time. This hackathon is by invitation only. It’s a core group of MetaCPAN hackers. We are keeping the group small in order to maintain focus on the v1 API and maximize the productivity of the group.

Why sponsor the MetaCPAN Hackathon?

• If your company uses Perl in any way, then your developers almost certainly use MetaCPAN to find CPAN modules, and they probably use other tools that are built on the MetaCPAN API.
• The MetaCPAN upgrade will improve the search engine and the API for all Perl developers. As a critical tool, we need it to be always available, and fast. This upgrade is a key step in that direction.
• This is a good way to establish your company as a friend of Perl, for example if you’re hiring.

Participants

There will be 8 people taking part, including me. Everyone taking part is an experienced senior-level software engineer, and most of them have already spent a lot of time working on MetaCPAN. As noted above, this is an invitational event with a very specific focus.

What is meta::hack?

MetaCPAN was created in late 2010. Version 0 of the MetaCPAN API was built on a very early version of Elasticsearch. For the first 5 years, most of the work on MetaCPAN focussed on improving the data coverage, and the web interface. In that time Elasticsearch has moved on, and we’re now well behind.

The work to upgrade Elasticsearch began in May of 2014. It continued in early Feb of 2015. Later, at the 2015 QA Hackathon in Berlin, Clinton Gormley (who works for Elastic) and I worked on moving MetaCPAN to Elasticsearch version 2. This work was continued at the 2016 QA Hackathon in Rugby, and as a result we now have a beta version in live usage.

The primary goal of meta::hack is to complete the port to Elasticsearch version 2, so the public API and search engine can be switched over. There are a number of benefits:

• Switching from a single server to a cluster of 3 servers, giving a more reliable service and improved performance.
• Once we decommission the old service, we’ll be able to set up a second cluster of 3 machines in a second data centre, for further improvements.
• We’ll be able to take advantage of new Elasticsearch features, like search suggesters.
• We’ll be able to use a new endpoint that has been developed specifically to speed up cpanminus lookups. Cpanminus is probably the most widely used CPAN client these days, so improving this will benefit a large percentage of the community.
• If and when search.cpan.org is decommissioned, we’ll be able to handle the extra traffic that will bring with it, and we’ll also have the redundancy to do this safely.
• We’ll be able to shift focus back to bug fixes and new MetaCPAN features.

Becoming a Sponsor

Neil Bowers has kindly taken on the task of shepherding the sponsorship process. (He also wrote the sponsorship prospectus from which I cribbed most of this post.) Please contact Neil or contact me for a copy of the meta::hack sponsorship prospectus. It contains most of the information listed above as well as the various available sponsorship levels which are available. Thank you for your help in making this event happen. We’re looking forward to getting the key people together in one room again and making this already useful tool even better.

	Rev: 1071.02	To validate, download page first.
	\| \| \| \|	\| \| \| \| \| \| \| \|

The [Perl programming] Authors Upload | | Server

Hi Olaf Alders,please choose an action from the menu.

Why sponsor the MetaCPAN Hackathon?

Participants

What is meta::hack?

Becoming a Sponsor

Hi Olaf Alders,
please choose an action from the menu.