Friday, September 06, 2013

Some details on perl's `my` internals

Recently I've been asked why using my when you unwrap arguments is slower than unwrap them into variables outside the function:

    sub foo { my $bar = shift; ... } # slower
    # vs.
    my $bar;
    sub foo { $bar = shift; ... } # faster

Also, on other hand the following shows opposite behaviour:

    sub foo { my ($bar, $baz) = (@_); ... } # faster
    # vs.
    my ($bar, $baz);
    sub foo { ($bar, $baz) = (@_); ... } # slower

Short answer

A function has its scope and my declares variable with lexical scope, so when scope ends perl has to take additional action to close the scope.

In second case the same rule applies and second code may win, but list assignment without my on left hand side (LHS) and @_ on right hand side (RHS) has to deal with possibility that both left and right sides share the same variable, like ($foo, $bar) = ($bar, $foo). So it has to copy arguments on RHS first and this penaly is larger than the win.

Long answer and how to draw conclusions on your own

Let's look at the following code

    my $bar;
    sub foo { $bar = shift; ... } # faster

It has problem of not being recursive ready. You can not call foo from inside foo as you use "global" variable to store argument. More correct code would be:

    our $bar;
    sub foo { local $bar = shift; ... }

It sure gets slower than any of two variants.

How to see difference

You can see difference without looking into perl source code, just look at optree of your code with B::Concise module:

    $ perl -MO=Concise,foo -E 'my $aa; sub foo { $aa = shift; undef }'
    ...
    4        <2> sassign vKS/2 ->5
    2           <0> shift s* ->3
    3           <0> padsv[$aa:FAKE:] sRM* ->4
    ...

I left out other lines to highlight assignment code. Compare above to:

    $ perl -MO=Concise,foo -E 'sub foo { my $aa = shift; undef}'
    ...
    4        <2> sassign vKS/2 ->5
    2           <0> shift s* ->3
    3           <0> padsv[$aa:47,48] sRM*/LVINTRO ->4
    ...

As you can see primary difference is in LVINTRO flag on padsv operation. pad* operations fetch a variable declared with 'my' from special lists for such variables (called PADs or PADLISTs). It's way harder to figure out what LVINTRO means without looking into source code, but in most cases it means that operation should be "localized". Whatever localization means for the operation.

How to find it in the perl code

Let's look at the code of padsv. You can find code by looking for pp_padsv in pp_*.c files:

    $ ack pp_padsv -A 30 pp_*.c
    pp_hot.c
    392:PP(pp_padsv)
    393-{
    ...
    406-    if (op->op_flags & OPf_MOD) {
    407-        if (op->op_private & OPpLVAL_INTRO)
    408-            if (!(op->op_private & OPpPAD_STATE))
    409-                save_clearsv(padentry);
    ...

Once again highlighted relevant code that says: if padsv is in lvalue context, localizing and it's not state declarator then save (not safe) our target variable from the PAD to be cleared when scope ends.

Excercise for readers

Now you have all the tools to figure out differences in other mentioned situations, here commands to get you there quickier, so you don't have excuse to skip:

    perl -MO=Concise,foo -E 'sub foo { local $aa = shift; undef}'

    perl -MO=Concise,foo -E 'my ($aa, $bb); sub foo { ($aa, $bb) = (@_); undef }'

    perl -MO=Concise,foo -E 'sub foo { my ($aa, $bb) = (@_); undef }'

    perl -MO=Concise,foo -E 'my ($aa, $bb); sub foo { ($aa, $bb) = (shift, shift); undef }'

A comment with explanation of list assignment would be nice :)

Thursday, February 21, 2013

My play-perl and perl5 core quests

If you don't know about play-perl then check this out. I've decided how I'm going to use this site.

Participating in Perl5 core development

I would love to participate more in perl5 core development. From time to time I read perl5-porters mailing list, sometimes participate in discussions. However, nobody likes people who says "A" and then disappears for a week or more. The only thing I can do is code tiny things on my own schedule. I don't want to run an idea by p5p list and block somebody from implementing it. Often I don't know if an idea is implementable or not, how big would impact, how hard it would be to maintain thing... So there is a big gap between idea and contacting p5p list for review and assessment.

My play perl

Here comes play-perl to the scene. I'm going to push my ideas as quests with link to this blog post and some common wording in the comment. The goals are:

  • sharing ideas very early
  • discussion at early stage
  • assessing importance with likes
  • hooking in contributors

You are free to take over a quest

At the moment play-perl is not very multiplayer, but there are plans. For now to coordinate collaboration on a quest you can do the following besides usual discussion:

  • ask for status update
  • ask for more details
  • claim a quest for a day - a week

Let's see where it would take us. Looking forward to experience.

Thursday, April 19, 2012

Compiling perl for debugging with gdb

If you ever looked inside of perl source then you know code uses tons of C preprocessor macroses. GDB has a nice feature to expand them for you. However, you have to compile program to contain information about macroses and it took me a while to figure it out how to do it.

Short answer

    ./Configure -des -Dprefix=/Users/ruz/perl/blead -Dusedevel \
        -Doptimize='-O3 -g3' -DEBUGGING \
        -Dusemymalloc -Dusethreads -Dinc_version_list=none

Explanation

I find INSTALL very misleading, so filed a bug report and may be it will be explained in docs or Configure script will be improved.

Anyway, here is explanation. -DEBUGGING=-g3 doesn't work, it always puts -g flag. It's not clear from the doc that -Doptimize is different thing and you can put -g3 there. There is more to this, but to be short if -DDEBUGGING is set and -g* is in -Doptimize then things are not adjusted and you get what you want. However, setting -Doptimize overrides default optimization flags and you have to repeat them.

See also

Dumping perl data structures in GDB

Monday, April 16, 2012

Improving development environment on MacOS Snow Leopard

Spent weekend improving terminal, bash and vim configs and learning some new tricks.

vim config

I was using astrail's dotvim solution. It worked good, but braids are terrible. So I was playing with git submodules to see if they can do better. They can not, but during investigation I've spotted vundle.

It was good surprise to see that astrails switched to vundles. Now, dotvim project is much better. Clone it. Edit vundels.vim to get plugins you need from github, vim-scripts or any other git repo. Run make install. In five you'll get everything installed.

shell

I don't use apple's bash, but one from gentoo prefix, but it was configured in terminal app to start after open. It works for primary user, but sudo su - still was colorless, without bash completion.

To change shell in MacOS you have to use chsh command and also full path to the shell script must be /etc/shells.

bash history

I'm used to one liners up to 500 characters long :). Always have 5-15 shell sessions running for months. My goal was to make history longer, shared acros sessions and less bloated with often commands.

Found helpful discussion on StackOverflow, but decided to stick to variant when history file is updated after every command, but session's is not refreshed from the file:

    export HISTSIZE=10000
    export HISTFILESIZE=10000
    export HISTCONTROL="ignorespace:erasedups"
    export PROMPT_COMMAND="history -a; $PROMPT_COMMAND"
    export HISTIGNORE="cd *:ls *:mplayer *"

I'm going to use history -r when I need some random command.

Colors in Terminal.app

While playing with vim config I've noticed that solarized theme doesn't look like on public screenshots. Terminal app on Snow Leopard supports only 8 colors. 256 colors only supported on Lion. I've switched to iTerm2 application and set TERM to xterm-256color.

Beside colors it has full screen mode and other nice features I'm going to use.

Friday, April 13, 2012

Fixing broken macports after lib upgrade

Many package maintaing systems upgrade a lib and leave existing applications and libs with broken dependecy on lib that doesn't exist anymore.

Macports is not an exclusion. Recent upgrade of libpng12 to libpng14 result in many broken ports. When you try to install/upgrade/run some application you get:

    dyld: Library not loaded: /opt/local/lib/libpng12.0.dylib

This was the last drop. I'm tired of looking for ports I should rebuild, so I wrote a script to find broken ports.

Usage

It's a good headstart to fix broken ports. Here is what I did to fix my system:

    ./port-broken-libs /opt/local/lib/*.dylib

Above command lists ports which have broken dependcy. Then you can pipe it to port upgrade:

    ./port-broken-libs /opt/local/lib/*.dylib \
    | xargs sudo port -pnfuc upgrade --force

A little about port's command line options:

Don't stop upgrade if some port fails. This may happen for variouse reasons.

Required part. If you omit this option then upgrade of port X may try to upgrade port Y first and that may fail.

Tells to ignore previouse attempts to build a port.

It's up to you to decide if you want to clean things after and unistall replaced ports.

Tells to force rebuild even if there is no newer version available.

It took me a few rounds to rebuild most ports. Still a few ports were broken and packager couldn't fix them, but list was much smaller. I just uninstalled them with force applied and installed once again.

After fixing all files in /opt/local/lib dir. I fixed /opt/local/{bin,sbin} as well.

That's it. Enjoy.

Thursday, April 05, 2012

Lovely week connecting LDAP client on RHEL to Windows AD over TLS/SSL

A client contacted us to enable SSL connection in Request Tracker's integration with LDAP. It was epic battle.

We had

First of all, Windows Server 2008 R2 with Active Directory, but it has an "add-on" installed that hardens security. On other side RHEL 6, Apache running RT with mod_perl and serving it via HTTPS.

Problem

Everything works just fine as long as you don't use secure connection to LDAP. If start_tls is enabled or you try to connect via ldaps:// then AD drops connection after first packet and client throws "connection reset by peer" error.

Investigation

From perl modules we moved down to `openssl s_client` command and got the same results. AD admins failed to provide any helpful information, so we captured ldp.exe session with wireshark. No surprise microsoft client worked just fine. Comparison of handshake packets ldp and s_client send showed that MS's LDAP client announce EDHCE group of cipher suits, openssl doesn't. Openssl project implemented this family of ciphers based on elliptic curves, but for whatever reason RHEL ships openssl without them.

Solution

So we compiled the latest openssl and installed it in its own location. Compiled Net::SSLeay against new openssl and installed it into custom location. Client was using Apache with mod_perl, but we had to switch to FastCGI. This is required as apache uses openssl library for HTTPS and we don't want perl and apache to load different versions of the library into the same process. mod_fcgid is not available in RHEL, but you can get it from EPEL repository. Simple patch to RT to put special library paths into @INC.

Hope it would help somebody.

Tuesday, April 03, 2012

Performance regression in perl with precompiled regexps

Some time ago I wrote about preparing regexpes earlier for cases where you can not use /o modifier. For example you have:

    my $re = qr{\d+};

And often you need to match:

    $str =~ /^$re$/;

You can not use /o modifier if $re may change. In the blog post I've described precompiling and showed that it works as fast as regexp with /o modifier:

    my $re = qr{...};
    my $re_prepared = /^$re$/;
    $str =~ $re_prepared;

Guess my surprize when I executed the same script with perl 5.14.1. Any method is at least 2 times faster than prepared, even ... =~ /^$re$/!

Results for 5.8.9:

    $ perl5.8.9 dynamic_regexp_with_o_effect.pl
                         Rate      no_o         o no_o_pre no_o_pre_braces    static
    no_o            1581903/s        --      -10%     -25%            -32%      -50%
    o               1764430/s       12%        --     -16%            -24%      -44%
    no_o_pre        2104366/s       33%       19%       --             -9%      -34%
    no_o_pre_braces 2323549/s       47%       32%      10%              --      -27%
    static          3174754/s      101%       80%      51%             37%        --

Results for 5.10.0 and newer:

    $ perl5.10.0 dynamic_regexp_with_o_effect.pl
                         Rate no_o_pre no_o_pre_braces      no_o         o    static
    no_o_pre         873813/s       --             -0%      -52%      -68%      -75%
    no_o_pre_braces  877714/s       0%              --      -52%      -68%      -75%
    no_o            1816840/s     108%            107%        --      -34%      -49%
    o               2759410/s     216%            214%       52%        --      -22%
    static          3542486/s     305%            304%       95%       28%        --